I am ingesting log files to Loki via Fluentbit, but I found the log lines with same timestamp were not showing in order as they are in the original log files:
I’ll need to run some test tomorrow to see if increment_duplicate_timestamp works or not. But couple of things you could try:
If you expand your logline in the explore view you should be able to see the timestamp in ns. Compare that to all your logs and see if it’s increasing. If not then either increment_duplicate_timestamp didn’t work for you or it’s not configured for some reason.
As you probably read in the config page, increment_duplicate_timestamp is not perfect. But it looks like your logs are from the same container so it probably shouldn’t be an issue. I’ll test tomorrow and let you know what I find.
Try an API call, and see if the timestamp returned is being incremented.
I did some testing on the configuration increment_duplicate_timestamp, and I can’t seem to make it work either. I did three tests, all three tests were done with monolithic Loki deployment and with API call.
First test: Loki 2.9.8, increment_duplicate_timestamp set to false.
Second test: Loki 2.9.8, increment_duplicate_timestamp set to true.
Third test: Loki 3.0.0., increment_duplicate_timestamp set to true.
In all three tests the timestamp remains the same for all the logs. I purposely set the timestamp to be 0000 on the last 4 digits (so should be same millisecond), and I did not see any change on timestamp regardless of what increment_duplicate_timestamp was set to.
I will probably look at the code and see what I can find when i have some time, but hopefully someone more knowledgable can chime in.
I should also point out though, that I did not see the problem you are experiencing, meaning when I did my test I sent in 10 log lines for each test, from number 1 to 10. And when querying they did show up in order in Grafana and API call, it’s just that you can’t reverse or change the order from the explore UI because they all had the timestamps, but nevertheless they still showed up in order.
I ran my tests using just API calls. If you are using fluent bit I’d recommend double checking and see if you are using multiple threads. If you are using multiple threads then your logs could arrive out-of-order if you are sending logs to Loki with more than one threads.
BTW what do you mean exactly by “reversed ordered”?
If you send the following lines of logs with the same timestamp:
line1
line2
line3
In Grafana you’ll always see
line3
line2
line1
Is this what you are seeing? If so this is normal, because grafana by default always shows the latest logs first, and because your logs have the same timestamp you can’t reverse the order.
Order - Display results in descending or ascending time order. The default is Descending, showing the newest logs first. Set to Ascending to show the oldest log lines first.
So it is by design, isn’t it?
I am afraid end user might be confused or even misguided by the reversed order from line3 to line1, as the real order is from line1 to line3 in the original logs.
This is what increment_duplicate_timestamp is supposed to fix, but it seems to not work. I don’t know if there is a way to change the default display order from Grafana, I did a quick search and couldn’t find a configuration for it, but I am also not an expert on Grafana.
I think your best bet is to educate users accordingly for now, and consider adding nanoseconds to your logs if your app supports it. Short of that, we’d have to see if a developer can pick up your github issue.
I don’t use LogCLI myself, so never actually noticed it. I usually use API call if I need to poke at Loki outside of Grafana.
