Trying to get Loki to watch a log file using Grafana Agent

Grafana Loki
1

I’m trying to create a proof-of-concept setup to evaluate Grafana as a possible solution for a visibility platform. I have a Grafana Cloud account and I can successfully gather data from InfluxDB using a Private Data Connector (PDC) on a local Windows VM. For next step, I’m trying to monitor logs with Loki. I have Loki running on the same local VM that InfluxDB is on. I’m pretty sure that it’s running because when I enter this in a browser on the VM:

http://localhost:3100/loki/api/v1/status/buildinfo

I get the following response:

{
    "version": "2.9.2",
    "revision": "a17308db6",
    "branch": "HEAD",
    "buildUser": "root@eee92863de73",
    "buildDate": "2023-10-16T14:20:36Z",
    "goVersion": ""
}

I’m trying to get Loki to watch a log file using a Grafana Agent, but I’m stuck. Here is my agent-config.yaml file:

server:
  log_level: info
metrics:
  wal_directory: C:\ProgramData\grafana-agent-wal
  global:
    scrape_interval: 1m
  configs:
    - name: integrations
integrations:
  windows_exporter:
    enabled: true
logs:
  positions_directory: C:\loki_configs\
  configs:
  - name: test1
    # clients:
    #   url: http://localhost:3100/loki/api/v1/push
    scrape_configs:
      - job_name: Lokitest
        static_configs:
          - targets: [localhost]
            labels:
              job: Lokitest
              __path__: C:\Users\daveb\Desktop\*.log

The agent service can start with the “clients” section commented out as shown above, but when I uncomment those lines, the service fails to start, I can’t find any helpful log messages in the Windows log, just that the service took an error.

Obviously, I’m new to all of this, so I can’t tell if something obviously wrong is staring me in the face. I created the above config based on the documentation as well as some example configs online. Any help would be greatly appreciated.

Thanks for any help!

2

You mentioned that you are using grafana cloud, yes? If so, your client section needs to be configured to send to Grafana Cloud.

3

Tony, thanks for the reply. Just to confirm, my current (intended) data flow is as follows:

logfile → Grafana agent → Loki → PDC → Grafana_cloud

The first four items are all on a single Windows VM in my network. Only the last connection from the PDC to the Grafana cloud goes outside of my network. Also, my Grafana cloud instance does successfully connect to Loki. I’m trying to complete the last step which is to load data into Loki using the Grafana agent. Note that I chose the Grafana agent rather than Promtail based on a recommendation in the documentation for my use case.

So, having said all of that, do I still need to add some configuration setting to the yaml file that tells the agent about Grafana cloud? I was expecting that it didn’t need to know about that, but obviously I’m just learning how to set this up.

Assuming that you are correct, can you give me an idea of what the configuration setting might look like?

Thanks again.
Dave

4

I don’t know how PDC works, so I’ll refrain from commenting on that.

I’d first make sure your loki instance is actually working. There are API commands that you can use to test.

5

Thanks Tony. I did verify that Loki is working by entering the following URL in the local browser on the VM where Loki is installed:

http://localhost:3100/loki/api/v1/status/buildinfo

I get the following response:

{
“version”: “2.9.2”,
“revision”: “a17308db6”,
“branch”: “HEAD”,
“buildUser”: “root@eee92863de73”,
“buildDate”: “2023-10-16T14:20:36Z”,
“goVersion”: “”
}

Also, my PDC instance is working properly because Grafana can communicate with Loki over the PDC.

6

What is the error you get from grafana agent when it stops?

7

I’m somewhat blind on that. The Grafana Agent is installed as a Windows service. If I have the “clients” section commented out in the config file, the service starts successfully. When I uncomment those lines, it fails and just shows the following message on a popup:

“Windows could not start the Grafana Agent service on Local Computer.

Error 1067: The process terminated unexpectedly.”

The Windows Event Log just shows an Error entry regarding the Grafana agent with no additional information.

I don’t know if there is a separate Grafana Agent log file somewhere, but I couldn’t locate one.

8

Ugh, I found my issue (at least as far as getting the service to run). I missed the dash before the URL under the clients section. It needs to look like this:

logs:
  positions_directory: C:\loki_configs\
  configs:
  - name: test1
    clients:
      - url: http://localhost:3100/loki/api/v1/push
    scrape_configs:

Tony, thanks for working with me on this.

9

I am not sure how to troubleshoot further. I would recommend two more things to check:

  1. Use an actual push API for Loki to push some arbitrary log. You can find an example in the API section of Loki documentation. Just to make sure push is actually working.

  2. I am not familiar with Windows, but there has to be a way to look for service logs. If you can’t find them, there might be a way to manually start the grafana agent from command line. Unfortunately the extent of my knowledge with Windows is rather limited.