Querying latest log per unique set of labels

Grafana Loki
,
1

I write a log to Loki every time the state of a systemd service changes. I would like to use these logs to build a Grafana dashboard that shows the current (latest) state of every service.

I can use a range query like this to retrieve all log entries within a certain time period:

{service_name="systemd-opentelemetry-monitor"}
  | line_format `{{.host_name}}/{{.systemd_unit_name}}: {{.systemd_unit_active_state}}`

Now the problem is filtering those log entries to only include the most recent line per unique (host_name,systemd_unit_name) label set.

I want to write something like this, but the latest aggregation function doesn’t exist:

latest( {service_name="systemd-opentelemetry-monitor"}
  | line_format `{{.host_name}}/{{.systemd_unit_name}}: {{.systemd_unit_active_state}}`
) by (host_name,systemd_unit_name)

I experimented with Grafana data transformations but also couldn’t find a way to do this. I also tried using last_over_time, but it would need to unwrap the systemd_unit_active_state field, and string fields are not supported. Perhaps I could use the label_format expression to convert the status to a numeric value and then back again, but this is rather verbose and clunky.

Is it possible to solve this neatly?

2

The only latest function I can find is last_over_time, which only works for metrics query. I would recommend you to change your log format (if possible), to include both a status string and some sort of ID (you’d have to make this up and keep it consistent in your code).

Example, let’s say running = 0 and failed = 1 (you can add other status too, and even a catch-all status like unknown = 99 for example), you’d change your log to something like this (and I generally recommend JSON because it’s standard and structured, but this is up to you):

{"host_name":"host1", "systemd_unit_name":"sshd","systemd_unit_state":"running","systemd_unit_state":0}
{"host_name":"host1", "systemd_unit_name":"httpd","systemd_unit_state":"failed","systemd_unit_state_id":1}

Then, you can query the latest state like so:

last_over_time(
  {SELECTOR} | json | unwrap systemd_unit_state_id
  [6h]
)
1 Like
3

I should also point out that if you have a metrics platform this is probably better done with something like node-exporter :slight_smile:

4

Thanks for the help.

I found eventually that this was a good fit for the state timeline view in Grafana, which nicely sidesteps the need to get latest values (and I was able to make it work with a few transformations).