Garbled log lines - Unable to parse IIS logs using Promtail in Windows server 2016

[kirannhegde]

Hello there,

Here are my environment detals:

Loki: v2.8.4 - Running on Azure AKS. Installed using the Bitnami helm chart for Grafana Loki
Promtail: v2.8.2 - Running on Window server 2016 English

Promtail configuration

server:
  http_listen_port: 5000
  grpc_listen_port: 0

positions:
  filename: D:/promtail/promtail/positions.yaml

clients:
  - url: http://loki.pmto.fake.com/loki/api/v1/push

scrape_configs:
- job_name: IIS-W3SVC1
  encoding: UTF-8
  
  static_configs:
  - targets:
      - localhost
    labels:
      job: IIS-W3SVC1
      __path__: C:/inetpub/logs/LogFiles/W3SVC1/*
  pipeline_stages:
  - match:
      selector: '{job="IIS-W3SVC1"}'
      stages:
        - drop:
            expression: "(?i)(\\W|^)(#Version|#Software:|#Date:|#Fields:)(\\W|$)"
  - match:
      selector: '{job="IIS-W3SVC1"} != "#Fields"'
      stages:
        - regex:
            expression: "^(?s)(?P<Date>\\S+?) (?P<Time>\\S+?) (?P<Server_IP_Address>\\S+?) (?P<CS_Method>\\S+?) (?P<CS_Uri_Stem>\\S+?) (?P<CS_Uri_Query>\\S+?) (?P<Server_Port>\\S+?) (?P<Client_User_Name>\\S+?) (?P<Client_IP_Address>\\S+?) (?P<CS_User_Agent>\\S+?) (?P<CS_Referer>\\S+?) (?P<HTTP_Status>\\S+?) (?P<SC_Substatus>\\S+?) (?P<Win32_Status>\\S+?) (?P<Time_Taken>.*)$"       
        - template:
            source: Time
            template: '{{ .Date }}T{{ .Time }}Z'
        - timestamp:
            source: Time
            format: RFC3339
        - labels:
            Server_IP_Address:
        - labeldrop:
            - filename
            
  - match:
      selector: '{job = "IIS-W3SVC1"}'
      stages:
        - drop:
            older_than: 168h

Description of the issue
The Promtail executable is being run on a Windows server 2016 on the command line and is pointing to the Loki running on Azure AKS.
When i search for the ingested log lines being shipped into Loki with the labels as chosen in the below image, i see garbled output:

I have inspected the IIS log file using Notepad++ and the encoding of the log file states that it is utf-8.

I have also observed the following:
-When i search by the label named: Server_IP_Address, i see absolutely no output.
-When i enable live log line trailing to see the log lines being ingested into Loki, i see that the correct log line is being ingested.

I am at loss to understand what could possibly be wrong. At the outset, it seems like an encoding issue. However, i have already set the encoding to utf-8 in Promtail and still continue to see this issue.

Any help would be very much appreciated.

Regards,
Kiran Hegde