How does the `timestamp` configuration in Loki work?

I am attempting to configure Pomtail to send historical logs to Loki. The logs are in JSON format, but I can also get them in CSV or XML. I can change pretty much anything I need to about the logs. This is a code coverage report, so it’s just key-value pairs including a timestamp of when the report was run.

I’ve tried supplying the timestamp in both UNIX and RFC3339 formats. For the UNIX timestamp I supplied it as both a string and an integer. I’ve attempted to use the following formats:

  • unix
  • “unix”
  • Unix
  • “Unix”
  • 1562708916
  • “1562708916”

I thought maybe there was some problem parsing Unix timestamps so I switched to RFC3339 format but I’m still not getting anywhere.

My current job config looks like this:

- job_name: loc
  pipeline_stages:
  - json:
      expressions:
        timestamp: "time"
        loc: "loc"
        cloc: "cloc"
  - labels:
      timestamp:
      loc:
      cloc:
  - timestamp:
      source: timestamp
      format: "DATE_RFC3339"
  static_configs:
  - targets:
      - localhost
    labels:
      job: hosted
      __path__: /logs/loc.log

The (simplified) JSON schema looks something like this:

{
  "loc": 1099687,
  "cloc": 119713,
  "time": "2019-02-14T00:00:00+00:00"
}

At first I thought that I was having problems because the timestamp was being loaded under “detected fields” instead of “log labels” in the Grafana interface, but after messing with it for several hours, I finally got timestamp to show up under “log labels” by adding the - labels pipeline stage configuration, but the timestamp configuration still does not seem to do anything.

Googling this problem yields a lot of different weird edge cases including using double quoted strings for some fields. Is there any example of a working implementation of using a timestamp from a JSON payload instead of using the timestamp that Grafana automatically attaches to the log?

1 Like

Is there any update on this topic? I face the same issue.

1 Like

I have the exact same issue, I have some logs with their own timestamp. I’ve managed to extract them as timestamp labels, but grafana still always shows the time as the time I got the logs, so that huge file as one timestamp. I still haven’t figured out how to use those timestamp labels to get a timeseries of the logs. Did you manage to figure this out? Or can anyone help?

Why is it so hard to swap out Loki’s automatic timestamps for log timestamps?

For me the timestamp extracts just fine, but now all the other fields in the log are gone :sob: .

        pipeline_stages:
          - cri: {}
          - match:
              selector: '{app="supply-deltas-sync", container="geth-supply-deltas"}'
              stages:
              - json: 
                  expressions:
                    level: lvl
                    output: msg
                    timestamp: t
              - labels:
                  level:
                  message: output
                  other_timestamp: timestamp
              - timestamp:
                  source: timestamp
                  format: RFC3339Nano
              - output:
                  source: output

I finally got it working! Thanks to everyone in this thread, you got me to the one yard line.

In our case, the date format was a little off. I had to figure out the Go date formats - and it’s quite literal, the mythical ‘2006-01-02 15:04:05 -0700’

    pipeline_stages:
      - multiline:
            firstline: '^\[\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s[+-]\d{4}\]\s'
            max_lines: 512
            max_wait_time: 3s
      - regex:
            expression: '\[(?P<time>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s[+-]\d{4})\]\s\[(?P<procid>\d*)\]\s\[(?P<severity>.*)\]'
      - labels:
            # these have to be parsed from the regex, otehrwise list
            # static labels in 'labels' above
            time:
            severity:
            procid:

      - timestamp:
            # [2020-08-21 18:37:05 +0000] [3366] [INFO] Booting worker with pid: 3366
            source: time
            # this is a weird Go format. See:
            # https://grafana.com/docs/loki/latest/clients/promtail/stages/timestamp/
            format: '2006-01-02 15:04:05 -0700'

This allowed us to import some historical logs. I found several articles that said historical logs can’t be parsed, but for me the solution was:

  1. Startup Promtail
  2. Shutdown Promtail
  3. Edit the ‘positions’ file
  4. Change any offsets for logs you want historical information on to “0” and save.
  5. Restart

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.