Can't query logs that emited before loki startup

Grafana Loki
, ,
1

Hey!

I’m a newbie with loki. I try to start loki with docker compose and then send my log files to it via promtail. Then I can’t query these logs with from loki. I think the cause is log timestamp is earlier then loki startup time. Can I somehow configure loki so it will return my logs?

In this example I hardcoded timestamp in promtail-config.yaml. In real it will be parsed from log.

Loki config:

auth_enabled: false

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

analytics:
 reporting_enabled: false

Promtail config:

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: my_job

  static_configs:
  - targets:
      - localhost
    labels:
      job: my_job
      __path__: /mnt/logs/test.log

  pipeline_stages:
  - match:
      selector: '{job="my_job"}'
      pipeline_name: default
      stages:
      - template:
          source: asctime
          template: '2024-05-21 11:33:01 +0000'
      - timestamp:
          source: asctime
          format: '2006-01-02 15:04:05 -0700'

docker-compose file:

services:
  loki:
    image: grafana/loki:3.0.0
    container_name: loki
    volumes:
    - ./loki-config.yaml:/etc/loki/config.yaml
    ports:
    - "3100:3100"
    network_mode: host
    command: "-config.file=/etc/loki/config.yaml -legacy-read-mode=false -server.log-request-at-info-level-enabled -server.log-request-headers"

  promtail:
    image: grafana/promtail:3.0.0
    container_name: promtail
    volumes:
    - "./promtail-config-test.yaml:/mnt/config/promtail-config.yaml"
    - "./logs:/mnt/logs:ro"
    ports:
    - "9080:9080"
    network_mode: host
    healthcheck:
      test: [ "CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:9080/ready || exit 1" ]
      interval: 10s
      timeout: 5s
      retries: 5
    command: "--config.file=/mnt/config/promtail-config.yaml --inspect"

test.log

INFO Some log message.

In promtail logs I see - my logs processed correctly

promtail  | [inspect: template stage]: 
promtail  | {stages.Entry}.Extracted["asctime"]:
promtail  | 	+: 2024-05-21 11:33:01 +0000
promtail  | [inspect: timestamp stage]: 
promtail  | {stages.Entry}.Entry.Entry.Timestamp:
promtail  | 	-: 2024-05-21 18:35:49.55811901 +0000 UTC
promtail  | 	+: 2024-05-21 11:33:01 +0000 UTC
promtail  | level=info ts=2024-05-21T18:35:49.549074638Z caller=filetargetmanager.go:372 msg="Adding target" key="/mnt/logs/test.log:{job=\"acm\"}"
promtail  | level=info ts=2024-05-21T18:35:49.549290594Z caller=filetarget.go:313 msg="watching new directory" directory=/mnt/logs
promtail  | level=info ts=2024-05-21T18:35:49.549577959Z caller=tailer.go:147 component=tailer msg="tail routine: started" path=/mnt/logs/test.log
promtail  | ts=2024-05-21T18:35:49.549522625Z caller=log.go:168 level=info msg="Seeked /mnt/logs/test.log - &{Offset:0 Whence:0}"

but if I query them I get empty response

$ curl -X GET -G localhost:3100/loki/api/v1/query -d query='{job="my_job"}' | jq '.status, .data.result'
 
"success"
[]

If I put some soon future timestamp in promtail-config.yaml log will be accessible.

2

This most likely is not the case. However the timestamp of logs can play a role to whether logs are accepted by Loki or not. For example, if reject_old_samples is set to true, logs older than reject_old_samples_max_age is rejected. Another less obvious scenario is if you have a log stream with newer logs (log stream is defined as a collection of logs with the same set of labels) you cannot send logs to the same log stream with older time stamp.

I would recommend:

  1. Make sure your Loki cluster is functional with a couple of push API calls.
  2. You can use the same API calls and tweak the timestamp to test a couple of scenarios.
  3. Check your configuration.
1 Like
3

Dunno what I’ve changed. Sometimes old logs becomes accessible after newer logs ingested.

I decided not to patch timestamp in promtail for static log files.

Thanks, for answer!

4

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.