In our observability setup we have this flow:
- Containerized application outputs its logs to local .log files
- The local .log files are being collected by Fluentd and put to a shared /var/log/containers file
- Another log collector Fluent-Bit takes logs from /var/log/containers file and sends them to Elasticsearch
Initially we had an issue where there was a mismatch between timestamp values - the “@timestamp” value was populated by the Fluent-Bit’s time, which would usually differ by several hundred milliseconds from the original app timestamp’s value.
We implemented changes in Fluentd’s and Fluent-Bit’s configs so that the “@timestamp” variable would is now populated with whatever value is found in the original log entry.
However, when using Grafana’s Explore, if the “Time” toggle is on, then the “@timestamp” value is still somehow different from the original value:
If we turn off “Time” toggle, then the timestamps match OK:
Can i get an explanation on what this “Time” toggle in Grafana’s Explore is about? Where does its timestamp value come from? Is its value equals to the log’s receive/read time by Grafana itself by any chance?