Logs disappear from Loki after some time

Grafana Loki
, , ,
21

Unfortunately, I have not found any root cause yet. Whenever I get to know how to fix this issue, I will write about it here

22

Hello,
Same problem here, Iā€™ve been looking for a few days, I checked the logs but nothing abnormal found

Itā€™s realy strange behavior :thinking:

23

I will give you more details about the issue.
Sorry for the long post :grimacing:

For my environement, Iā€™ve deployed Loki, Grafana, and promtail in docker with this docker-compose file:

version: "3"

networks:
  loki:

volumes:
  loki-data:

services:
  loki:
    image: grafana/loki:2.9.0
    volumes:
      - ./loki-config:/etc/loki
      - loki-data:/data
    ports:
      - "3100:3100"
    restart: always
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki
    logging:
      driver: json-file

  promtail:
    image: grafana/promtail:2.9.0
    volumes:
      - /var/log:/var/log
      - ./promtail-config:/etc/promtail
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - "3101:3101"
      - "9080:9080"
    restart: always
    command: -config.file=/etc/promtail/promtail.yaml
    networks:
      - loki
    logging:
      driver: json-file

  grafana:
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=false
      - GF_LOG_LEVEL=debug
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy
          orgId: 1
          url: http://loki:3100
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
    image: grafana/grafana:latest
    volumes:
      - ./grafana-config:/var/lib/grafana
    restart: always
    ports:
      - "3000:3000"
    networks:
      - loki
    logging:
      driver: json-file

The loki config:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /data/loki
  replication_factor: 1
  ring:
    kvstore:
      store: memberlist

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v12
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /data/loki/storage

ruler:
  alertmanager_url: http://localhost:9093

The promtail container is used to catch the system logs.
promtailā€™s config:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/log/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
    - targets:
       - localhost
      labels:
        job: varlogs
        __path__: /var/log/*log

The containerā€™s logs was catched by the Loki plugin for docker. The only cofig is the docker config (in /etc/docker/daemon.json):

{
        "debug": false,
        "log-driver": "loki",
        "log-opts": {
                "loki-url": "http://127.0.0.1:3100/loki/api/v1/push",
                "loki-batch-size": "400",
                "loki-timeout": "10s",
                "loki-max-backoff": "3000ms",
                "loki-retries": "5",
                "mode": "non-blocking",
                "keep-file": "true"
        }
}

Now that i have presented my config, i am going to give mode details about the issue:
For my test I have used the query given by @jangaraj earlier in the topic. I have activated the auto update every 5 seconds and recorded my screen. Here is what I was able to see.

Iā€™ve monitor the size of the data folder during the log disappear:

image
image777Ɨ230 54.6 KB

The last result of query before logs disappear:

image
image1340Ɨ735 61.9 KB

And a few secods later:

image
image1322Ɨ738 98.3 KB

And an extract of the logs between the time the data was there and the time the data was no longer there:

{"log":"level=info ts=2023-11-19T22:07:07.594182013Z caller=roundtrip.go:241 org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=3h0m0s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677059842Z"}
{"log":"level=info ts=2023-11-19T22:07:07.60200874Z caller=engine.go:234 component=querier org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=7m10s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677094447Z"}
{"log":"level=info ts=2023-11-19T22:07:07.602670725Z caller=engine.go:234 component=querier org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=22m50s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677100409Z"}
{"log":"level=info ts=2023-11-19T22:07:07.602810978Z caller=engine.go:234 component=querier org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677104559Z"}
{"log":"level=info ts=2023-11-19T22:07:07.60311597Z caller=engine.go:234 component=querier org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677108807Z"}
{"log":"level=info ts=2023-11-19T22:07:07.603407066Z caller=engine.go:234 component=querier org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677125117Z"}
{"log":"level=info ts=2023-11-19T22:07:07.614870785Z caller=engine.go:234 component=querier org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677129319Z"}
{"log":"level=info ts=2023-11-19T22:07:07.616083281Z caller=metrics.go:159 component=querier org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=7m10s start_delta=7m7.616075665s end_delta=-2.383924134s step=2s duration=13.939655ms status=200 limit=500 returned_lines=0 throughput=139MB total_bytes=1.9MB total_bytes_structured_metadata=0B lines_per_second=324828 total_lines=4528 post_filter_lines=4528 total_entries=47 store_chunks_download_time=0s queue_time=43.511Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.677134389Z"}
{"log":"level=info ts=2023-11-19T22:07:07.616530312Z caller=engine.go:234 component=querier org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677141044Z"}
{"log":"level=info ts=2023-11-19T22:07:07.631937873Z caller=metrics.go:159 component=querier org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=2h37m7.631930173s end_delta=2h7m9.631930405s step=2s duration=15.285502ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=32.79Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.677144924Z"}
{"log":"level=info ts=2023-11-19T22:07:07.632300803Z caller=metrics.go:159 component=querier org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=2h7m7.632296747s end_delta=1h37m9.632296927s step=2s duration=17.330015ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=837.192Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.677150399Z"}
{"log":"level=info ts=2023-11-19T22:07:07.632537037Z caller=metrics.go:159 component=querier org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=22m50s start_delta=2h59m59.632530439s end_delta=2h37m9.632530607s step=2s duration=29.795663ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=106.239Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.677159788Z"}
{"log":"level=info ts=2023-11-19T22:07:07.640437263Z caller=metrics.go:159 component=querier org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=1h37m7.640428764s end_delta=1h7m9.640429087s step=2s duration=37.489996ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=670.939Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.677165693Z"}
{"log":"level=info ts=2023-11-19T22:07:07.650635132Z caller=metrics.go:159 component=querier org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=37m7.650626646s end_delta=7m9.650626885s step=2s duration=47.092528ms status=200 limit=500 returned_lines=0 throughput=172MB total_bytes=8.1MB total_bytes_structured_metadata=0B lines_per_second=402080 total_lines=18935 post_filter_lines=18935 total_entries=201 store_chunks_download_time=392.587Āµs queue_time=107.437Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.677171085Z"}
{"log":"level=info ts=2023-11-19T22:07:07.668743002Z caller=metrics.go:159 component=querier org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=1h7m7.668726012s end_delta=37m9.668726262s step=2s duration=65.478519ms status=200 limit=500 returned_lines=0 throughput=119MB total_bytes=7.8MB total_bytes_structured_metadata=0B lines_per_second=278381 total_lines=18228 post_filter_lines=18228 total_entries=195 store_chunks_download_time=551.543Āµs queue_time=121.962Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.677176642Z"}
{"log":"level=info ts=2023-11-19T22:07:07.670337479Z caller=metrics.go:159 component=frontend org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=3h0m0s start_delta=2h59m59.355316698s end_delta=-644.683049ms step=2s duration=75.64674ms status=200 limit=500 returned_lines=0 throughput=236MB total_bytes=18MB total_bytes_structured_metadata=0B lines_per_second=551127 total_lines=41691 post_filter_lines=41691 total_entries=1 store_chunks_download_time=944.13Āµs queue_time=1.919999ms splits=7 shards=7 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.67718535Z"}
{"log":"level=info ts=2023-11-19T22:07:11.594944506Z caller=table_manager.go:136 index-store=tsdb-2020-10-24 msg=\"uploading tables\"\n","stream":"stderr","time":"2023-11-19T22:07:11.676273248Z"}
{"log":"level=info ts=2023-11-19T22:07:11.595007442Z caller=index_set.go:86 msg=\"uploading table index_19680\"\n","stream":"stderr","time":"2023-11-19T22:07:11.676340263Z"}
{"log":"level=info ts=2023-11-19T22:07:11.595017987Z caller=index_set.go:107 msg=\"finished uploading table index_19680\"\n","stream":"stderr","time":"2023-11-19T22:07:11.676346626Z"}
{"log":"level=info ts=2023-11-19T22:07:11.595026044Z caller=index_set.go:185 msg=\"cleaning up unwanted indexes from table index_19680\"\n","stream":"stderr","time":"2023-11-19T22:07:11.676351229Z"}
{"log":"level=info ts=2023-11-19T22:07:12.729425001Z caller=roundtrip.go:241 org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=3h0m0s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776569762Z"}
{"log":"level=info ts=2023-11-19T22:07:12.734183222Z caller=engine.go:234 component=querier org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776631671Z"}
{"log":"level=info ts=2023-11-19T22:07:12.734818821Z caller=engine.go:234 component=querier org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=22m46s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776637361Z"}
{"log":"level=info ts=2023-11-19T22:07:12.736031981Z caller=engine.go:234 component=querier org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=7m14s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776641789Z"}
{"log":"level=info ts=2023-11-19T22:07:12.737553888Z caller=metrics.go:159 component=querier org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=22m46s start_delta=3h0m0.737547621s end_delta=2h37m14.737547833s step=2s duration=2.623971ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=363.897Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.776646204Z"}
{"log":"level=info ts=2023-11-19T22:07:12.737974328Z caller=engine.go:234 component=querier org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776676384Z"}
{"log":"level=info ts=2023-11-19T22:07:12.74867471Z caller=engine.go:234 component=querier org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776681119Z"}
{"log":"level=info ts=2023-11-19T22:07:12.749778391Z caller=engine.go:234 component=querier org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776685094Z"}
{"log":"level=info ts=2023-11-19T22:07:12.750903491Z caller=engine.go:234 component=querier org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776688879Z"}
{"log":"level=info ts=2023-11-19T22:07:12.751571429Z caller=metrics.go:159 component=querier org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=1h7m12.75156367s end_delta=37m14.751563893s step=2s duration=13.518717ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=31.036Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.776692762Z"}
{"log":"level=info ts=2023-11-19T22:07:12.752137289Z caller=metrics.go:159 component=querier org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=7m14s start_delta=7m12.752130609s end_delta=-1.247869206s step=2s duration=16.00925ms status=200 limit=500 returned_lines=0 throughput=127MB total_bytes=2.0MB total_bytes_structured_metadata=0B lines_per_second=299014 total_lines=4787 post_filter_lines=4787 total_entries=49 store_chunks_download_time=0s queue_time=695.526Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.776698532Z"}
{"log":"level=info ts=2023-11-19T22:07:12.754857512Z caller=metrics.go:159 component=querier org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=2h37m12.754852513s end_delta=2h7m14.754852726s step=2s duration=3.869587ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=10.177266ms splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.776704063Z"}
{"log":"level=info ts=2023-11-19T22:07:12.755839898Z caller=metrics.go:159 component=querier org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=1h37m12.755832006s end_delta=1h7m14.755832259s step=2s duration=5.961109ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=949.207Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.776714015Z"}
{"log":"level=info ts=2023-11-19T22:07:12.756117261Z caller=metrics.go:159 component=querier org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=2h7m12.756112535s end_delta=1h37m14.756112719s step=2s duration=7.366933ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=58.604Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.77671959Z"}
{"log":"level=info ts=2023-11-19T22:07:12.765933081Z caller=metrics.go:159 component=querier org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=37m12.765919583s end_delta=7m14.765919831s step=2s duration=31.506087ms status=200 limit=500 returned_lines=0 throughput=188MB total_bytes=5.9MB total_bytes_structured_metadata=0B lines_per_second=439597 total_lines=13850 post_filter_lines=13850 total_entries=148 store_chunks_download_time=0s queue_time=165.512Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.776725403Z"}
{"log":"level=info ts=2023-11-19T22:07:12.76842301Z caller=metrics.go:159 component=frontend org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=3h0m0s start_delta=2h59m59.309407874s end_delta=-690.59189ms step=2s duration=38.357699ms status=200 limit=500 returned_lines=0 throughput=208MB total_bytes=8.0MB total_bytes_structured_metadata=0B lines_per_second=485873 total_lines=18637 post_filter_lines=18637 total_entries=1 store_chunks_download_time=0s queue_time=12.442ms splits=7 shards=7 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.776731025Z"}

Hopefully all this can help someone understand the problem.

24

With some tests, iā€™ve found the problem isnā€™t here on loki 2.8.0.

Iā€™ve open an Github issue here: Logs disapear after some time #11274

2 Likes
25

The problem seems to be fixed in Loki 2.9.2 so Iā€™m closing this topic.

Thank you all for your help :wink:

26

loki 3.1.1 problem still present.

Tried both loki 2.9.4 and loki 3.1.1.

After some hours logs disappear as they are lost.

After some hours they are present again. No significant error log on loki.

Example: tried to extract logs with logcli and also with grafana, from 2024-09-03T00:00:00+02:00" --to="2024-09-04T00:00:00+02:00.

The same query lauched on 2024-09-04T07:30:00+02:00 and on 2024-09-04T15:30:00+02:00.

Here the results (num lines, month, day, H)

First launch
476 Sep 3 00
390 Sep 3 01
415 Sep 3 02
487 Sep 3 03
398 Sep 3 04
388 Sep 3 05
558 Sep 3 06
398 Sep 3 07
922 Sep 3 08
809 Sep 3 09
580 Sep 3 10
405 Sep 3 11
486 Sep 3 12
402 Sep 3 13
400 Sep 3 14
495 Sep 3 15
413 Sep 3 16
275 Sep 3 17
1 Sep 3 22

Second one (hours later):
476 Sep 3 00
390 Sep 3 01
415 Sep 3 02
487 Sep 3 03
398 Sep 3 04
388 Sep 3 05
558 Sep 3 06
398 Sep 3 07
922 Sep 3 08
809 Sep 3 09
580 Sep 3 10
405 Sep 3 11
486 Sep 3 12
402 Sep 3 13
400 Sep 3 14
495 Sep 3 15
413 Sep 3 16
433 Sep 3 17
430 Sep 3 18
388 Sep 3 19
392 Sep 3 20
388 Sep 3 21
551 Sep 3 22
450 Sep 3 23