Can't parse custom timestamp

Grafana Loki
1

My timestamps are in the format 2006-01-02T15:04:05.000000-07:00. When I try it with the following config, it doesn’t work:

scrape_configs:
  pipeline_stages:
    - timestamp:
        source: time
        format: 2006-01-02T15:04:05.000000-07:00

I suspect it’s because -07:00 is not listed in the Promtail timestamp stage documentation, so I’m trying to use a template stage to remove the colon from the timezone and parse the timestamp in the format 2006-01-02T15:04:05.000000-0700, which should be accepted. This is what my config now looks like:

scrape_configs:
  pipeline_stages:
    - json:
        expressions:
          time: time
    - template:
        source: time
        template: '{{ regexReplaceAll "(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{6}-\d{2}):(\d{2})" .Value "${1}${2}" }}'
    - timestamp:
        source: time
        format: 2006-01-02T15:04:05.000000-0700

I’m pretty sure everything should be correct, but Promtail exits with this error:

level=error ts=2023-09-15T13:38:29.601015608Z caller=main.go:170 msg=“error creating promtail” error=“failed to make file target manager: invalid template stage config: template: pipeline_template:1: invalid syntax”

2

Can you post a couple of samples of your original log and your actual promtail configuration?

3

Sure!

Log line excerpts:

{"time":"2023-09-18T06:56:15.068572+00:00","seve":"INFO","mesg":"Logged in user: root (::1)","data":null}
{"time":"2023-09-18T07:02:08.141130+00:00","seve":"INFO","mesg":"Api request handled. Runtime(0.060374975204468), Status(200)","data":null}

Promtail Config:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: sys3
  static_configs:
  - targets:
      - localhost
    labels:
      job: backend_logs
      node: Node_1
      __path__: /var/www/html/project/backend/logs/*.log
  pipeline_stages:
    - json:
        expressions:
          time: time
    - template:
        source: time
        template: '{{ regexReplaceAll "(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{6}-\d{2}):(\d{2})" .Value "${1}${2}" }}'
    - timestamp:
        source: time
        format: 2006-01-02T15:04:05.000000-0700

I just realized that I have to use [-+] in the regex to account for both negative and positive timezone offsets, but that doesn’t solve the error. It says that something is wrong with the template syntax, but I just don’t see what’s supposed to be invalid.

4

Unless I am missing something, it doesn’t seem to be necessary to break your timestamp string. Here is what I am using for testing (copied your two lines of logs and changed time to 0000, +0700, and -0700):

{"time":"2023-09-18T06:56:15.068572+00:00","seve":"INFO","mesg":"Logged in user: root (::1)","data":null}
{"time":"2023-09-18T07:02:08.141130+00:00","seve":"INFO","mesg":"Api request handled. Runtime(0.060374975204468), Status(200)","data":null}
{"time":"2023-09-18T06:56:15.068572+07:00","seve":"INFO","mesg":"Logged in user: root (::1)","data":null}
{"time":"2023-09-18T07:02:08.141130+07:00","seve":"INFO","mesg":"Api request handled. Runtime(0.060374975204468), Status(200)","data":null}
{"time":"2023-09-18T06:56:15.068572-07:00","seve":"INFO","mesg":"Logged in user: root (::1)","data":null}
{"time":"2023-09-18T07:02:08.141130-07:00","seve":"INFO","mesg":"Api request handled. Runtime(0.060374975204468), Status(200)","data":null}

Promtail configuration:

    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    positions:
      filename: /tmp/positions.yaml

    clients:
      - url: http://loki:3100/loki/api/v1/push

    scrape_configs:
    - job_name: sys3
      static_configs:
      - targets:
          - localhost
        labels:
          job: backend_logs
          __path__: /tmp/test.log
      pipeline_stages:
        - json:
            expressions:
              time: time
        #- template:
        #    source: time
        #    template: '{{ regexReplaceAll "(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{6}-\d{2}):(\d{2})" .Value "${1}${2}" }}'
        - timestamp:
            source: time
            format: '2006-01-02T15:04:05.000000-07:00'

Dry run results:

level=warn ts=2023-09-18T16:18:39.530963821Z caller=promtail.go:263 msg="enable watchConfig"
2023-09-18T06:56:15.068572+0000{filename="/tmp/test.log", job="backend_logs"}	{"time":"2023-09-18T06:56:15.068572+00:00","seve":"INFO","mesg":"Logged in user: root (::1)","data":null}
2023-09-18T07:02:08.14113+0000	{filename="/tmp/test.log", job="backend_logs"}	{"time":"2023-09-18T07:02:08.141130+00:00","seve":"INFO","mesg":"Api request handled. Runtime(0.060374975204468), Status(200)","data":null}
2023-09-18T06:56:15.068572+0700{filename="/tmp/test.log", job="backend_logs"}	{"time":"2023-09-18T06:56:15.068572+07:00","seve":"INFO","mesg":"Logged in user: root (::1)","data":null}
2023-09-18T07:02:08.14113+0700	{filename="/tmp/test.log", job="backend_logs"}	{"time":"2023-09-18T07:02:08.141130+07:00","seve":"INFO","mesg":"Api request handled. Runtime(0.060374975204468), Status(200)","data":null}
2023-09-18T06:56:15.068572-0700{filename="/tmp/test.log", job="backend_logs"}	{"time":"2023-09-18T06:56:15.068572-07:00","seve":"INFO","mesg":"Logged in user: root (::1)","data":null}
2023-09-18T07:02:08.14113-0700	{filename="/tmp/test.log", job="backend_logs"}	{"time":"2023-09-18T07:02:08.141130-07:00","seve":"INFO","mesg":"Api request handled. Runtime(0.060374975204468), Status(200)","data":null}
level=info ts=2023-09-18T16:18:44.531199126Z caller=filetargetmanager.go:361 msg="Adding target" key="/tmp/test.log:{job=\"backend_logs\"}"
1 Like
5

Huh, I’m 100% sure I’ve tried that already, but it works now. Thanks, I appreciate it!

6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.