Time difference between 2 loglines

Grafana Loki
, ,
1

Hello, using loki with logQL. I am trying to get the difference in time between consecutive log lines. My goal is to get the total time that the state machine was in a certain state (attributes_target). I cannot seem to do operations on the timestamps of the logs
Here is a example of the logs that show to state transitions from source to target state.

1739968397584	{"body":"State transition","severity":"INFO","attributes":{"NAME":"StateMachineHandler","source":"ReconnectingConnectedModem","target":"WaitingForFirstFrame"},"resources":{"service.name":"tablet","service.version":"0.0.1"},"instrumentation_scope":{"name":"tablet::state_machine::tablet_statemachine_handler"}}
1739968396771	{"body":"State transition","severity":"INFO","attributes":{"NAME":"StateMachineHandler","source":"Interrupted","target":"ReconnectingConnectedModem"},"resources":{"service.name":"tablet","service.version":"0.0.1"},"instrumentation_scope":{"name":"tablet::state_machine::tablet_statemachine_handler"}}
1739968392724	{"body":"State transition","severity":"INFO","attributes":{"NAME":"StateMachineHandler","source":"Playing","target":"Interrupted"},"resources":{"service.name":"tablet","service.version":"0.0.1"},"instrumentation_scope":{"name":"tablet::state_machine::tablet_statemachine_handler"}}
2

I don’t think this is possible by using Loki alone. Logs are processed line by line, it’s hard to do calculate across. You might be able to do it in Grafana, I am not entirely sure, but I think it’s worth a try:

  1. Create three queries, one for each abbributes_target, and use line_format to output the timestamp as nanoseconds.
  2. In Grafana dashboard, use transformation to join all three metrics into one.
  3. Use another transformation to do calculation.

It would be great if your source can determine the duration of state for you (for example, perhaps have another log line that says something to the effect of exiting a state with duration).

3

Thanks for the answer! In my example there are only 3 log lines, but normally have have to get the time difference between all the logs lines (>1000).

  • which transformation would you use operate between the different cumulative timestamps and always show the difference (in a new column)
  • how do you transfer the timestamp to unix nanoseconds? I converted it to seconds with
| line_format " {{ __timestamp__ | unixEpoch}}"