Promtail Character encoding of log files

Grafana Loki
1

We are trying to setup Loki and Promtail to collect logs for some legacy system, however we are running into issues regarding the Character encoding of the log files. The log files are encoded as ISO-8859-1 however Promtail reads the files as UTF-8 causing some characters to be grabled.

Is there any workaround for this Issue?

2

no workarounds that I know of I’m afraid.

I did look at what it would take to add this and I think it’s something we could fit in over the next couple weeks.

3

Hi Ed
Thank you for the response. Should I create or is there already an issue i github where I can follow the progress on this?

4

Was this ever done? I have the same problems with trying to send logs from a legacy windows application into loki with promtail. When there are special characters (i.e scandinavian characters) grafana error out with invalid utf-8 rune. So need to figure out some fix for this.

5

Loki PR2604 seems to have fixed this issue by adding encoding: utf-16 parameter to promtail which we expect it to fix our issues with Windows logs encoded in utf-16.

scrape_configs:
- job_name: system_convert
  encoding: utf-16
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/utf16.log

For some reason which I don’t understand, it was not merged by owen-d. The bot later closed it. Seemingly vyzigoldigold put a lot of efforts in Loki PR2604 and I don’t understand why it was dismissed so easily. I gave it a try, but personally I’m not able to build GitHub - vyzigold/loki: Like Prometheus, but for logs. and now we are stuck with some utf-16 encoded Windows logs that neither promtail nor loki can handle properly.

@ewelch could you please reconsider reopening Loki PR2604 and merging (a revised version of) it?