How to get time lost between messages?

I have a java application that prints periodically GC logs in the format:

2024-03-26T16:30:04.797-0400:: Application time  1.12345 seconds
2024-03-26T16:30:05.797-0400:: Application time  0.91234 seconds

The application time is a little bit lower to the time that has passed since the last message was printed. For example, above, the second message prints exactly one second after the first, but the application time is lower than 1 second. The lines do not come in any constant interval, in fact the message come in the logs very very randomly.

I want to capture and draw how much lower the application time is over the actual spent time between messages, to see how much application time is lost.

How do I subtract the time from the last message to the extracted value from the message? Currently I have:

sum by(job) (sum_over_time(
  {job=~"$JOB"}
  | regexp `Application time: (?P<apptime>\d+\.\d+)`
  | unwrap apptime
  [$__interval]
))

However, that only shows values equal to the values. I need to substract the time that has passed since the last message. I could take the timestamp of the last message within each [$__interval] and remember it and substract it from the next [$__interval] chunk, but I do not know how to do it. How do i “query” the last message before [$__interval] is applied?

Thanks.

1 Like