Need assistance recognizing timestamp from logs

Grafana Loki
1

The format is the following [02/Mar/2022 13:42:53] ERROR [django.request:230]

The log style is multiline

Logs messages are being inserted with timestamp at creation

The reason why you see the log date and the promtail date is different, is because I ingested a log with older entries as a target with promtail

Promtail config:

server:
   http_listen_address: 0.0.0.0
   http_listen_port: 8088

positions:
  filename: /tmp/positions.yaml
clients:
  - url: http://mylokiserver.com:3100/loki/api/v1/push

scrape_configs:

- job_name: containers

  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/log/containers/**/*log

  # --log-opt tag="{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}"
  pipeline_stages:

  - multiline:
      # Identify timestamps as first line of a multiline block. Enclose the string in single quotes.
      # 02/Mar/2022 13:42:5
      firstline: '^\[\d{2}\/[a-zA-Z]+\/\d{4} \d{1,2}:\d{2}:\d{2}\]'
      max_wait_time: 3s
  - regex:
      # Flag (?s:.*) needs to be set for regex stage to capture full traceback log in the extracted map.
      # expression: '^(?P<time>\[\d{2}\/[a-zA-Z]+\/\d{4} \d{1,2}:\d{2}:\d{2}\]) (?P<level>[A-Z]+)'
      expression: ^\[(?P<time>\d{2}\/[a-zA-Z]+\/\d{4} \d{1,2}:\d{2}:\d{2})\] (?P<level>[A-Z]+)
  - timestamp:
      source: time
      format: '02/Feb/2022 18:09:37'
  - regex:
      source: filename
      expression: .*\/(?P<container_name>.*)\/
  - labels:
      container_name:
      host: ${HOSTNAME}
      level:

promtail
promtail1528×563 25.2 KB

Dry run:
echo “[02/Mar/2022 13:42:53] ERROR [django.request:230]” | ./promtail --stdin --dry-run --inspect -config.file test.yml

Clients configured:
----------------------
url: http://mylokiserver.com:3100/loki/api/v1/push
batchwait: 1s
batchsize: 1048576
follow_redirects: false
backoff_config:
  min_period: 500ms
  max_period: 5m0s
  max_retries: 10
timeout: 10s
tenant_id: ""
stream_lag_labels: filename

[inspect: regex stage]:
{stages.Entry}.Extracted["level"]:
        +: ERROR
{stages.Entry}.Extracted["time"]:
        +: 02/Mar/2022 13:42:53
[inspect: timestamp stage]: none
[inspect: regex stage]: none
[inspect: labels stage]:
{stages.Entry}.Entry.Labels:
        -: {__path__="/var/log/containers/**/*log", job="containerlogs"}
        +: {__path__="/var/log/containers/**/*log", job="containerlogs", level="ERROR"}
2022-03-04T20:28:04.787766983+0000      {__path__="/var/log/containers/**/*log", job="containerlogs", level="ERROR"}    [02/Mar/2022 13:42:53] ERROR [django.request:230]
level=info ts=2022-03-04T20:28:04.789815311Z caller=server.go:260 http=[::]:8088 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2022-03-04T20:28:04.790009301Z caller=main.go:119 msg="Starting Promtail" version="(version=2.4.2, branch=HEAD, revision=525040a32)"type or paste code here

Also added debug log level
level=debug ts=2022-03-04T21:18:16.453927955Z caller=timestamp.go:196 component=pipeline msg=“failed to parse time” err=“parsing time "02/Mar/2022 13:42:53" as "02/Feb/2022 18:09:37": cannot parse "Mar/2022 13:42:53" as "/Feb/"” format=“02/Feb/2022 18:09:37” value=“02/Mar/2022 13:42:53”

2

I found the solution. The custom time format must be from the go list of constants. Any other value you select not in this list will cause your custom time format to fail the parse.

The format is: 01/Jan/2006 15:05:05

3

Update - it still doesn’t put the right timestamp into prometheus. The events are being inserted with the wrong timestamp still. I don’t understand why. Check out this dry run:

echo "[02/Mar/2022 13:42:53] ERROR [django.request:230]" | ./promtail --stdin --dry-run --inspect -config.file docker-config.yml
Clients configured:
----------------------
url: http://myloki.com:3100/loki/api/v1/push
batchwait: 1s
batchsize: 1048576
follow_redirects: false
backoff_config:
  min_period: 500ms
  max_period: 5m0s
  max_retries: 10
timeout: 10s
tenant_id: ""
stream_lag_labels: filename

[inspect: regex stage]:
{stages.Entry}.Extracted["level"]:
        +: ERROR
{stages.Entry}.Extracted["time"]:
        +: 02/Mar/2022 13:42:53
[inspect: timestamp stage]:
{stages.Entry}.Entry.Entry.Timestamp:
        -: 2022-03-09 19:00:49.649861423 +0000 UTC
        +: 2022-03-01 13:42:53 +0000 UTC

Timestamp is supposedly corrected, but I go search for the log entry for the date, and i dont see it. [Not from this dry run, specifically’]

4

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.