Tell apart systemd journal logs from machines

Grafana Alloy
, ,
1

Hi,
I’ve been trying out Alloy for a little by now and I’ve set up a local Loki instance which receives Systemd Journal logs from multiple machines with Alloy, here’s the Alloy config of the machines:

loki.relabel "journal" {
  forward_to = []
  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label  = "unit"
  }
}

loki.source.journal "journalctl"  {
  forward_to    = [loki.write.main_loki.receiver]
  relabel_rules = loki.relabel.journal.rules
  labels        = {
    component = "loki.source.journal",
    instance = constants.hostname,
  }
}

loki.write "main_loki" {
   endpoint {
      url = "http://loki:3100/loki/api/v1/push"
   }
}

It works fine but here’s my question, how do I tell apart from which machine the logs are from? Right now I’m adding an instance label to the logs with the hostname of the machine sending the logs, but that leaves the responsibility of identification to the Alloy client.
I wanted to ask if there’s some way to identify the origin machines from the Loki instance or an Alloy proxy.
I know i could make an Alloy proxy with a different listener for each machine but i don’t think that scales too well.

Thank you in advance.

2

Hi @adam53r - I’m also new and have been learning over the last few days. You seperate in grafana by filtering based on ‘instance’ - it appears in grafana (and prometheus) instance is the word they use for hostname.

3

Thank you for responding @TheRyanMonty, but I don’t think or know that’s the answer, that may be true when using Prometheus to extract metrics from a server (with node_exporter for example), but I’m talking about Alloy, by default and to my understanding It doesn’t export any labels or info about the log’s origin, see the attached picture below that shows the labels being received from my Loki instance with the following single Alloy instance with this config:

loki.relabel "journal" {
  forward_to = []
  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label  = "unit"
  }
}

loki.source.journal "journalctl"  {
  forward_to    = [loki.write.main_loki.receiver]
  relabel_rules = loki.relabel.journal.rules
}

loki.write "main_loki" {
   endpoint {
      url = "http://msi-server.arm53.xyz:3100/loki/api/v1/push"
   }
}

Labels exported from Alloy
Labels exported from Alloy1871×498 56.2 KB

I hope there’s something I’m missing because I can’t really see how else you’re supposed to do it.
Thanks again for your response :smile:

4

Hi @adam53r - I haven’t rolled out prometheus yet and am running journal and syslog via alloy to loki using grafana as the frontend myself. I believe the instance line from your initial post is assigning the hostname to instance and is what I’m running myself:

Here’s what my grafana looks like isolating to the host:

image
image1510×972 96.9 KB

5

Yes exactly, my question then is: Do I have to rely on the Alloy “Client” to identify itself (with the instance line on my first post) or is there a way for the “Server” (maybe an Alloy proxy) to identify the Alloy clients by itself, maybe with a generated id or something like that.
Thank you.

6

My understanding is that the data to identify anything within Grafana as a label comes from the Alloy client (or whatever a person is using to send data to Loki), but I would absolutely defer to someone more knowledgeable, as I’m on day 3 of learning.

7

Thank you very much for your help @TheRyanMonty

1 Like