Add labels to logs

Grafana Alloy
1

Hello everyone!

When setting up a new server, I realized that the option to use grafana-agent is no longer available. Therefore, I set up alloy to send my server log files to loki in Grafana Cloud.

Everything is working fine and I can process the logs in Grafana Cloud. However, I am missing one feature that made it easier to handle the logs.

With grafana-agent, I was able to set tags for job and service_name. With alloy, I currently cannot find a solution and am looking for help.

I tried to use the loki.process as a pipeline step:

loki.process "add_labels" {
  forward_to = [loki.write.grafana_cloud_loki.receiver]

	stage.labels {
		values = {
		  job  = "my_job_name",
		  service_name = "my_job_name"
		}
	}
}

However, when restarting alloy, I get the following error message:

Failed to start Vendor-agnostic OpenTelemetry Collector distribution with programmable pipelines.

Any idea how I can add tags that I can then later use in Grafana as a selector statement (e.g., {job=“my_job_name”})?

Looking forward to your responses!

2

You are missing a , after the service_name line. Try this:

loki.process "add_labels" {
  forward_to = [loki.write.grafana_cloud_loki.receiver]

  stage.labels {
    values = {
      job  = "my_job_name",
      service_name = "my_job_name",
    }
  }
}
3

Thank you very much for your message, Tony! I tried adding the , after the service_name line. Unfortunately, I still get the same error after restarting the alloy service.

Here is the full output of the status command:

× alloy.service - Vendor-agnostic OpenTelemetry Collector distribution with programmable pipelines
     Loaded: loaded (/lib/systemd/system/alloy.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2024-07-08 19:40:41 CEST; 28s ago
       Docs: https://grafana.com/docs/alloy
    Process: 53294 ExecStart=/usr/bin/alloy run $CUSTOM_ARGS --storage.path=/var/lib/alloy/data $CONFIG_FILE (code=exited, status=1/FAILURE)
   Main PID: 53294 (code=exited, status=1/FAILURE)

Jul 08 19:40:41 SERVER systemd[1]: alloy.service: Scheduled restart job, restart counter is at 5.
Jul 08 19:40:41 SERVER systemd[1]: Stopped Vendor-agnostic OpenTelemetry Collector distribution with programmable pipelines.
Jul 08 19:40:41 SERVER systemd[1]: alloy.service: Start request repeated too quickly.
Jul 08 19:40:41 SERVER systemd[1]: alloy.service: Failed with result 'exit-code'.
Jul 08 19:40:41 SERVER systemd[1]: Failed to start Vendor-agnostic OpenTelemetry Collector distribution with programmable pipelines.
4
  1. Can you share your entire Alloy configuration?
  2. Check journal log, it usually tells you if there is a configuration error.
5

Thank you very much! Your hint with checking the journal provided the solution. I accidentally forgot to add the receiver attribute as part of forward_to.

Here is the whole info in case someone else has a similar problem. For me, it is solved now.

Config file:

local.file_match "local_files" {
    path_targets = [{"__path__" = "/home/user/myapp/logs/*.log*"}]
    sync_period = "60s"
}
loki.source.file "log_scrape" {
   targets    = local.file_match.local_files.targets
   //forward_to = [loki.write.grafana_cloud_loki.receiver]
   forward_to = [loki.process.add_labels]
   tail_from_end = true
}
loki.process "add_labels" {
  forward_to = [loki.write.grafana_cloud_loki.receiver]

  stage.labels {
    values = {
      job  = "myjobname",
      service_name = "myservicename",
    }
  }
}
loki.write "grafana_cloud_loki" {
  endpoint {
    url = "XXX"

    basic_auth {
      username = "XXX"
      password = "XXX"
    }
  }
}

Journal with command: sudo journalctl -u alloy.service

Error: /etc/alloy/config.alloy:58:18: loki.process.add_labels should be capsule, got object
57 |    //forward_to = [loki.write.grafana_cloud_loki.receiver]
58 |    forward_to = [loki.process.add_labels]
    |                  ^^^^^^^^^^^^^^^^^^^^^^^
 59 |    tail_from_end = true

Thanks again for your help! :slight_smile: