Log metric query day is shifted when displayed in grafana

Grafana Loki
, , ,
1

Hello!

I am making a dashboard that counts log messages and groups them by type and day.
I managed to get the correct logs and visualize them with this query:

sum by (MessageTemplate)(count_over_time({...} | json | MessageTemplate="foo" or MessageTemplate="bar" [$__auto] )) or vector(0)

image

image
image1194×597 14.4 KB

HOWEVER… The days for that the logs are grouped under are shifted one day forward…

Here is a portion of the logs that renders the above dashboard, the ones I have marked should in my mind end up on the 15th of march, but in the panel they end up on the 16th.

image

Does this have to do with how the step (which is configured to [1d]) is configured? And how should it be configured otherwise?
Or is it something with how I represent the

The extra hour that gets added has to do with the timezone of my browser. The issue is the same when I’m running the dashboard in UTC.

Anyone that knows if it is fixable or what I am doing wrong?
I would like to avoid the ugly hack of subtracting 24 hours of each grouping in the result…

2

Just because your logs have Timestamp in them doesn’t mean Loki automatically reads them. When injecting logs into Loki the timestamp is the time of injection, unless you configure your log pipeline to adjust timestamp.

You should be able to find the real timestamp of your logs, which wouldn’t be part of your logs.

3

Hi @tonyswumac

I understand that, and I am using the scrapetime of the logs, not the “TimeStamp” that is written on the log. I could have been clearer on that.

The issue is still the same though, the grouping is off by an entire day.

4

  1. Can you show a screenshot of timestamp on your logs?

  2. What are the results you see from API call?

5

@tonyswumac

image
image751×537 91.7 KB

image
image1276×760 40.3 KB

6

Your logs look good, your table view also looks good, that leaves the most likely problem to be your graph. Try creating a brand new visualization, slap your query on it and nothing else, and see what it looks like.

If that looks good, great. If not, double check your API call produced by Grafana and ran it by hand and compare results.

I’d also double check your interval properties. Also try not setting step. And check what transformation you have.