Grafana UI Table: transformation "sort by: time" removes milliseconds from timestamp

  • What Grafana version and what operating system are you using?
    Grafana v9.3.6 on windows 10 (amd64) with Loki and Promtail version 2.7.3

  • What are you trying to achieve?
    When sorting a table in Grafana UI table view by the timestamp of the log the timestamp only shows YYYY-MM-DD hh:mm:ss. If I remove the sort by transformation the timestamp again shows milliseconds in addition to the above timestamp YYYY-MM-DD hh:mm:ss.SSS. This is what I want. However, the timestamps are sorted correctly according to the milliseconds even though only seconds are shown. Essentially, when I sort by timestamp with milliseconds it is because I want to see the timestamp with milliseconds and not have Grafana remove them for me.

  • How are you trying to achieve it?
    Load a single log file of the following configuration in Loki+Promtail into Grafana:

2023-02-17 12:34:56.789 [type1] Message something
2023-02-17 12:34:57.123 [type4] Message something else
2023-02-17 12:34:57.124 [type2] Message something else again

In Promtail the timestamp is extracted and set to the log timestamp so that Grafana/Loki correctly recognises this. The “types” as seen in the brackets above are few and therefore I use a label for those. Grafana sees these as well and I am easily able to sort by type.

  • What happened?
    The transformation “sort by” in table view removes the milliseconds of the timestamp when sorting by time. This is inconvenient as I am interested in the milliseconds. It sorts correctly with milliseconds though, so I assume it is only a “how-is-it-shown-in-Grafana” issue.

  • What did you expect to happen?
    I expected the transformation “sort by” to sort my data by timestamp and not format the timestamp simultaneously (removing milliseconds).

  • Can you copy/paste the configuration(s) that you are having problems with?
    This is the data unsorted in the table view (editing mode):


    My three transformations is the disabled sort by, extract labels and organize fields. The behaviour doesn’t change based on whether the others are enabled or not. As you can see the timestamp under time matches the timestamp on the log line, however, the time is not sorted.
    In the time column on the left you can see the timestamp with milliseconds. Sorry, I have to hide the message content because it might be confidential.


The three transformations are the same as above. The sort by is now enabled. You can see on the line that the time is correctly sorted by milliseconds however, in the time column the milliseconds are now not shown.

I don’t think it is a configuration issue but perhaps rather a GUI issue.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    No errors (related to the issue at least haha)

  • Did you follow any online instructions? If so, what is the URL?
    I followed the following instructions:
    Grafana loki timestamp configuration (has no issues in my configuration) timestamp | Grafana Loki documentation
    Transform data documentation (sort of doesn’t mention the availability of milliseconds so I assume it is not possible?) Transform data | Grafana documentation

I realise that I would be able to sort by time and keep the line as is so that the millisecond information is kept just not in the time column, but my wish is to have the time, type and message in separate columns in order to sort in the table view for specific messages, timestamps and types. It is difficult to select/sort specific timestamps with milliseconds when the milliseconds are not visible in the sorting.

I hope you understand my issue. I am very sorry for my poor English. Thank you in advance for any help (perhaps there are workarounds?) or any comments on this issue. Have a good day!

1 Like

[SOLUTION]

Or, more like a workaround that does not solve the issue but gets the result I want.

After working a bit on the scrape configs in my Promtail configuration file to simplify labels and lower cardinality, I encountered that Grafana simply stopped showing milliseconds at all times no matter what I did with the table or time series queries, options, transformations and so on. Therefore I regoogled the issue with this new angle and found this Github pull request:

Dankeder suggests writing in the Grafana ini to change the time resolution, however, Grafana suggests using the sample.ini to make a custom.ini instead:

So, this solution solved my problem. Now, all date-time formats are always configured as YYYY-MM-DD hh:mm:ss.SSSS. It is one digit too many but it defaults to 0, which is just fine.

A little guide, in case anyone sees this and needs the same solution:

  • Open the */grafana/conf folder. Mine is located in '/Program Files/GrafanaLabs/
  • Edit the sample.ini in Notepad. Locate the full_date setting.
  • Remove the semicolon ; and add .SSS after hh:mm:ss
  • Save the file as custom.ini. If windows will not allow you to do this because you do not have permission, you need to go back to the /conf/ folder, right click it, click on properties, then security, then edit, then find your user in the list, click it and tick below allow full control. Then you can save in the folder.
  • Restart grafana in windows services (find by searching windows for ‘Services’, scroll down until you find Grafana and press restart on the left).

I restarted Loki and Promtail as well, but it all loaded immediately and worked right away.