When I am querying the logs from loki at grafana explore, there is a mismatch in timestamp in log content and the timestamp which grafana is showing up which is enabled by toggling the Time option.
Which component associates this associated timestamp with the log, is it loki, log agent or something else.
@tonyswumac can you suggest me on this?
Loki uses the timestamp when the logs are sent to Loki. That doesn’t always match to whatever timestamp that shows in the log string. If you want Loki timestamp to match the timestamp shown in your logs, then you need to parse for timestamp from your logs before injecting to Loki.
1 Like
Thanks for this info @tonyswumac . So, if agent doesnt associate any time stamp to log line then loki itself associates the timestamp when it was ingested. Is that correct. Can you help me with any doc or something supporting the same?
I did a testing to check for the same with a promtail agent. There was no timestamp stage in my promtail config. I stopped the promtail agent at 1:00 and waited for 2 mins. I again started the promtail agent at 1:02 and found that logs generated within the range of 1:00 and 1:02 had their respective timestamps between 1:00 and 1:02. But I was expecting timestamp shown in grafana would be greater than at least 1:02:00.
Actually I am facing an issue with usecase where logstash agent is used to send logs to loki. And there is an ambiguous time mismatch and and also the logs are not showing up in correct order as compared to the source.
If you want timestamp to line up then you have to parse for it. There is no other way around it.
Also I should clarify, “injection time” can be different depending on situations. If you use an API then yes it would be inject time, but if you use a log agent then it could be the time when the log agent picks up the logs and not the time it actually makes it into Loki.