Grafana Agent Windows logs help

Grafana Loki
, ,
1

Hello,

I’ve been working through an implementation of Grafana Loki for a home logging setup (testing out, was using Graylog previously) and was trying to get my windows system logs to loki but have been having issues. Currently my setup includes pfSense and NAS systems sending to rsyslog on the loki host which then goes to promtail, then loki. After some research I saw there is a Grafana Agent that is supposed to allow for this to work (sending windows logs to loki) but I have been unable to get this to work. I’ve also been looking into sending the logs to rsyslog which then would get fed into promtail like the other systems are currently doing.

When I try to send to syslog with fluentbit I can see the logs but with no job/label that I’ve included in the config, only see them in the “loki” host label which is not what I want, would love to be able to separate them into a separate job or host. When I try the grafana agent route I’m not able to see any of the windows logs.

If someone could confirm this is actually possible that would be great, and if so…any pointers would be greatly appreciated.

Here is an example of the Grafana agent agent-config.yaml file:

server:
  log_level: debug
logs:
  positions_directory: "C:\\Program Files\\Grafana Agent"
  configs:
    - name: windows
      clients:
        - url: http://10.10.30.9:3100/loki/api/v1/push
      scrape_configs:
        - job_name: windows
          static_configs:
            - targets:
                - localhost
          labels:
            job: windows-logs

Thanks.

2

I think you are missing __path__ in your log configuration if you are trying to scrape logs locally on your windows box with Grafana Agent. See full example configuration here: Create a configuration file | Grafana Agent documentation.

If you could share a bit of your overall infrastructure that would be helpful too. Generally I like to keep the logging pipeline as simple as possible. If you have the option to install Grafana Agent on each of your Windows box or appliances, I’d definitely do that, and then send the logs directly to Loki.

3

Hi,

I’ve got the internet into my house into a pfSense appliance firewall, then a SynologyNAS, a raspberryPi running pi-hole, a small virtualization system running XCP-NG with various VMs (the loki host being one of them while testing it out), and two Windows systems. The plan is to install the agent on the windows systems but was testing it out on one first to ensure I could get it working first, then mirror its setup on the other system.

That very well could be it, I’ll give that a shot. Yes, trying to scrape the windows logs and send to loki and be able to use a label for them

4

That seems to have been the issue. I can see now the job created from it now using this config:

server:
  log_level: debug
logs:
  positions_directory: "C:\\Program Files\\Grafana Agent"
  configs:
    - name: windows
      clients:
        - url: http://10.10.30.9:3100/loki/api/v1/push
      scrape_configs:
        - job_name: windows
          static_configs:
            - targets:
                - localhost
              labels:
                job: windows
                __path__: C:\WINDOWS\system32\config\*

But there is a new issue now when I query against that job…the logs show up but messed up, assuming due to encoding (but just an assumption):

image
image1795×266 78.9 KB

5

Yeah, I vaguely remember having a discussion about this a while ago. Try setting the encoding to utf16. Also take a look at this thread, the example configuration may be useful to you there: Need help specifying log path for MSSQL log files - #19 by erobichaux

6

added encoding: utf-16 to the config but now when querying against that job it takes forever to load anything - after clicking the blue “Run Query” button, it just stays in the red “Cancel” state and nothing loads. I was getting this problem before adding that line as well but it seemed to have resolved itself but is happening again now…I’m just full of problems over here…

7

  1. It may be helpful to try and open one of the log files and confirm the text encoding used.

  2. There is a loki.echo components that you can use to output logs to console instead of a Loki instance, which may help in terms of troubleshooging (see loki.echo | Grafana Agent documentation). This is the flow mode documentation, I am not sure what the equivalent is in the static mode.

  3. If you have a small log file that are safe to share, please feel free to send it to me. I’ll be happy to poke at it.

8

1 - I opened one of the windows event viewer files in Notepad++ and the Encoding looks like its listed as ANSI (though even in notepad++, lots of strange characters…)

3 - I dont feel too comfortable sharing as its the windows event logs so think they may contain some sensitive information so I’d rather not unfortunately

9

Update:
Got it to work and it correctly pulls in the data with no weird encoding with the following agent-config.yaml:

server:
  log_level: debug
logs:
  positions_directory: "C:\\Program Files\\Grafana Agent"
  configs:
    - name: windows
      clients:
        - url: http://10.10.30.9:3100/loki/api/v1/push
      scrape_configs:
      - job_name: windows
        windows_events:
          use_incoming_timestamp: false
          bookmark_path: "./bookmark.xml"
          eventlog_name: "Application"
          xpath_query: '*'
          labels:
            job: windows
        relabel_configs:
          - source_labels: ['computer']
            target_label: 'host'

So now, just working through finding a way to grab all of the eventlog names (Application, Security, System, and Setup).

Update 2:
Got all of them showing up successfully with the following agent-config.yaml:

server:
  log_level: debug
logs:
  positions_directory: "C:\\Program Files\\Grafana Agent"
  configs:
    - name: windowsApplication
      clients:
        - url: http://10.10.30.9:3100/loki/api/v1/push
      scrape_configs:
      - job_name: windowsApplication
        windows_events:
          use_incoming_timestamp: false
          bookmark_path: "./bookmark.xml"
          eventlog_name: "Application"
          xpath_query: '*'
          labels:
            job: windowsApplication
        relabel_configs:
          - source_labels: ['computer']
            target_label: 'host'
      - job_name: windowsSecurity
        windows_events:
          use_incoming_timestamp: false
          bookmark_path: "./bookmark.xml"
          eventlog_name: "Security"
          xpath_query: '*'
          labels:
            job: windowsSecurity
        relabel_configs:
          - source_labels: ['computer']
            target_label: 'host'
      - job_name: windowsSystem
        windows_events:
          use_incoming_timestamp: false
          bookmark_path: "./bookmark.xml"
          eventlog_name: "System"
          xpath_query: '*'
          labels:
            job: windowsSystem
        relabel_configs:
          - source_labels: ['computer']
            target_label: 'host'
      - job_name: windowsSetup
        windows_events:
          use_incoming_timestamp: false
          bookmark_path: "./bookmark.xml"
          eventlog_name: "Setup"
          xpath_query: '*'
          labels:
            job: windowsSetup
        relabel_configs:
          - source_labels: ['computer']
            target_label: 'host'
1 Like