Hi,
I’ve identified a behaviour in the dashboards I’ve created I don’t know if it’s the correct one.
I’ve made a dashboard in Grafana obtaining the result from Loki logs and when I select the filter time in the bar from Grafana, it doesn’t make the graph correctly.
It seems as if doesn’t “paint” the last sampling.
For example; If I select in the time filter 30 minutes, the graph is the following.
The last 7 minutes are empty.
I’ve made different tests and it doesn’t “paint” the data correctly, Or I’m not understanding the behavior.
If I reduce the time range of the date, the time when the graphic doesn’t appear increases.
The same example but for the last 15 minutes:
Is this correct?
Can you share your Loki configuration?
Also double check and make sure there are actually logs going into Loki with timestamp during those 7 minutes. We have some log pipelines with heavy caching and processing that delays for a minute or two. But 7 minutes is probably some sort of configuration issue on Loki, but still good to double check.
I confirm in Loki exist the data. If I see the data in a log format I see the data exists in Loki, that’s why I don’t understand this behavior
I see the data in log format in Grafana. (in the option Explore) This issue is when I try to see the data in a graphic way
From your screenshot it looks like you are highlighting the timestamp “string” present in your logs, not the actual timestamp. If you expand on the log message it’ll tell show you the real timestamp, what do those look like?
If I expand the data:
It still continues with the same behavior and the data exist
I still don’t think that’s your log’s timestamp in Loki.
In version 9 Grafana used to show you timestamp in nanoseconds as a field when you expand the log. I don’t see that anymore. But from our own logs it looks like the first part of message is the timestamp (at least for us).
- Also, have you tried to perhaps visualize it in a timeseries graph and see if it exhibits the same behavior?
- You can query against Loki with an API, which will return actual timestamps for each log. Alternatively you can also look at query inspector in Grafana, which should have the API call and the actual text return.
Thanks a lot for your explanations, you’re right.
The timestamp is according to the data Grafana shows. I made the test at first and the data in the log and the time were the same, that’s why I’ve hidden the time in the Grafana log and I confused the timestamp with the logline as you said.
