How to have timestamp in miliseconds in Loki

Grafana Loki
1

Hi,
We are using grafana and loki in kubernetes. I have promtail to send the logs, and in our application we have all json logs with the timestamp of the format:
2023-09-05T21:52:19.989Z

In promtail conf we have stage:

  - timestamp:
      source: timestamp
      format: RFC3339

But logs in grafana explore are still in seconds,
2023-09-05 21:52:19
There are no milliseconds, but i need those as some logs come at the same second, so the ordering is not always correct.
Am i missing something, do i need to make some changes in grafana or different format in the timestamp stage?
Thank you

2

Hello,

You can use a transformation to convert the type of the Time to string following this format YYYY-MM-DD hh:mm:ss.SSS

3

Hello,
I am not sure I follow. I tried the format RFC3339Nano and it’s not working. Is that done as a stage? And then I add -timestamp stage?

RIght now I have this:

config:
lokiAddress: http://loki:3100/loki/api/v1/push
snippets:
pipelineStages:
- cri: {}
- json:
expressions:
app:
- match:
selector: ‘{app=“rico”}’
stages:
- json:
expressions:
timestamp: timestamp
message: message
level: level
- match:
selector: ‘{app=“vault”}’
stages:
- json:
expressions:
timestamp: timestamp
message: message
level: level
mdc:
process_id: processID
- json:
source: mdc
expressions:
span_id: spanId
trace_id: traceId
- template:
source: level
template: “{{ ToLower .Value }}”
- template:
source: trace_id
template: ‘{{if .Value }} {{- .Value -}} {{else}} {{- “-” -}} {{end}}’
- template:
source: span_id
template: ‘{{if .Value }} {{- .Value -}} {{else}} {{- “-” -}} {{end}}’
- template:
source: output
template: “Trace[{{ .trace_id }}] Span[{{ .span_id }}] | {{ .message }}”
- labeldrop:
- filename
- container
- job
- node_name
- stream
- labels:
level:
- timestamp:
source: timestamp
format: RFC3339Nano
- output:
source: output

4

Your time string is neither RFC3339 nor RFC3339Nano. You can see the full list here: time package - time - Go Packages

You can write your own custom format with string such as "2006-01-02T15:04:05.000Z". It also might be a good idea and just write a simple go script to test your time string with the parser.

5

Hey Tony,
thank you for the feedback. I think i even tried this option as well at some point and it didn’t work. Let me try again with the below stage:

  • timestamp:
    source: timestamp
    format: “2006-01-02T15:04:05.000Z”

And again, I don’t have milliseconds in my timestamp that appear is grafana explore section.
I am not really a developer or have knowledge of Go :frowning: thanks for suggestion

6

Did you get an answer? I am facing the same problem on my project, if you find the solution please share here.

7

Yes, the problem i faced was grafana version. I had little bit older kube-prometheus-stack helm version deployed. When i updated that one, and Grafana was updated to version i think something 9.5 or something. So now i am on version 10+ and miliseconds are there

8

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.