Loki OTLP missing logs entries

Grafana Loki
, ,
1

Hi,

Currently im noticed that I have some missing log entries in Loki. Im currently using the OTLP endpoint with an otel-contrib collector infront of it.

Im also sending the logs to another otlp endpoint in the same pipeline but that backend has all the logs, but in loki which i currently preferer im missing some.

grafik
grafik845×613 82.6 KB

Did anyone also encounter this problem?

This is my loki configuration, its mostly basic changes.

auth_enabled: true # enables multi tenancy

server:
  http_listen_port: 3100
  log_level: info

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /loki

schema_config:
  configs:
  - from: 2020-05-15
    store: tsdb
    object_store: s3
    schema: v13
    index:
      prefix: index_
      period: 24h

compactor:
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  delete_request_store: s3

limits_config:
  max_query_lookback: 672h # 28 days
  retention_period: 672h   # 28 days

storage_config:
 tsdb_shipper:
   active_index_directory: /loki/index
   cache_location: /loki/index_cache
   cache_ttl: 24h
 aws:
   s3: minio
   s3forcepathstyle: true

Thanks.

2

Found the problem… after seeing where nginx send the otlp request to. I found out that docker load balanced the requests to two containers, which had the same hostname…