Logs disappear from Loki after some time

Grafana Loki
, , ,
21

Unfortunately, I have not found any root cause yet. Whenever I get to know how to fix this issue, I will write about it here

22

Hello,
Same problem here, Iā€™ve been looking for a few days, I checked the logs but nothing abnormal found

Itā€™s realy strange behavior :thinking:

23

I will give you more details about the issue.
Sorry for the long post :grimacing:

For my environement, Iā€™ve deployed Loki, Grafana, and promtail in docker with this docker-compose file:

version: "3"

networks:
  loki:

volumes:
  loki-data:

services:
  loki:
    image: grafana/loki:2.9.0
    volumes:
      - ./loki-config:/etc/loki
      - loki-data:/data
    ports:
      - "3100:3100"
    restart: always
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki
    logging:
      driver: json-file

  promtail:
    image: grafana/promtail:2.9.0
    volumes:
      - /var/log:/var/log
      - ./promtail-config:/etc/promtail
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - "3101:3101"
      - "9080:9080"
    restart: always
    command: -config.file=/etc/promtail/promtail.yaml
    networks:
      - loki
    logging:
      driver: json-file

  grafana:
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=false
      - GF_LOG_LEVEL=debug
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy
          orgId: 1
          url: http://loki:3100
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
    image: grafana/grafana:latest
    volumes:
      - ./grafana-config:/var/lib/grafana
    restart: always
    ports:
      - "3000:3000"
    networks:
      - loki
    logging:
      driver: json-file

The loki config:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /data/loki
  replication_factor: 1
  ring:
    kvstore:
      store: memberlist

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v12
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /data/loki/storage

ruler:
  alertmanager_url: http://localhost:9093

The promtail container is used to catch the system logs.
promtailā€™s config:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/log/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
    - targets:
       - localhost
      labels:
        job: varlogs
        __path__: /var/log/*log

The containerā€™s logs was catched by the Loki plugin for docker. The only cofig is the docker config (in /etc/docker/daemon.json):

{
        "debug": false,
        "log-driver": "loki",
        "log-opts": {
                "loki-url": "http://127.0.0.1:3100/loki/api/v1/push",
                "loki-batch-size": "400",
                "loki-timeout": "10s",
                "loki-max-backoff": "3000ms",
                "loki-retries": "5",
                "mode": "non-blocking",
                "keep-file": "true"
        }
}

Now that i have presented my config, i am going to give mode details about the issue:
For my test I have used the query given by @jangaraj earlier in the topic. I have activated the auto update every 5 seconds and recorded my screen. Here is what I was able to see.

Iā€™ve monitor the size of the data folder during the log disappear:

image
image777Ɨ230 54.6 KB

The last result of query before logs disappear:

image
image1340Ɨ735 61.9 KB

And a few secods later:

image
image1322Ɨ738 98.3 KB

And an extract of the logs between the time the data was there and the time the data was no longer there:

{"log":"level=info ts=2023-11-19T22:07:07.594182013Z caller=roundtrip.go:241 org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=3h0m0s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677059842Z"}
{"log":"level=info ts=2023-11-19T22:07:07.60200874Z caller=engine.go:234 component=querier org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=7m10s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677094447Z"}
{"log":"level=info ts=2023-11-19T22:07:07.602670725Z caller=engine.go:234 component=querier org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=22m50s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677100409Z"}
{"log":"level=info ts=2023-11-19T22:07:07.602810978Z caller=engine.go:234 component=querier org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677104559Z"}
{"log":"level=info ts=2023-11-19T22:07:07.60311597Z caller=engine.go:234 component=querier org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677108807Z"}
{"log":"level=info ts=2023-11-19T22:07:07.603407066Z caller=engine.go:234 component=querier org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677125117Z"}
{"log":"level=info ts=2023-11-19T22:07:07.614870785Z caller=engine.go:234 component=querier org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677129319Z"}
{"log":"level=info ts=2023-11-19T22:07:07.616083281Z caller=metrics.go:159 component=querier org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=7m10s start_delta=7m7.616075665s end_delta=-2.383924134s step=2s duration=13.939655ms status=200 limit=500 returned_lines=0 throughput=139MB total_bytes=1.9MB total_bytes_structured_metadata=0B lines_per_second=324828 total_lines=4528 post_filter_lines=4528 total_entries=47 store_chunks_download_time=0s queue_time=43.511Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.677134389Z"}
{"log":"level=info ts=2023-11-19T22:07:07.616530312Z caller=engine.go:234 component=querier org_id=fake traceID=3c4...065 msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:07.677141044Z"}
{"log":"level=info ts=2023-11-19T22:07:07.631937873Z caller=metrics.go:159 component=querier org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=2h37m7.631930173s end_delta=2h7m9.631930405s step=2s duration=15.285502ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=32.79Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.677144924Z"}
{"log":"level=info ts=2023-11-19T22:07:07.632300803Z caller=metrics.go:159 component=querier org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=2h7m7.632296747s end_delta=1h37m9.632296927s step=2s duration=17.330015ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=837.192Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.677150399Z"}
{"log":"level=info ts=2023-11-19T22:07:07.632537037Z caller=metrics.go:159 component=querier org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=22m50s start_delta=2h59m59.632530439s end_delta=2h37m9.632530607s step=2s duration=29.795663ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=106.239Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.677159788Z"}
{"log":"level=info ts=2023-11-19T22:07:07.640437263Z caller=metrics.go:159 component=querier org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=1h37m7.640428764s end_delta=1h7m9.640429087s step=2s duration=37.489996ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=670.939Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.677165693Z"}
{"log":"level=info ts=2023-11-19T22:07:07.650635132Z caller=metrics.go:159 component=querier org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=37m7.650626646s end_delta=7m9.650626885s step=2s duration=47.092528ms status=200 limit=500 returned_lines=0 throughput=172MB total_bytes=8.1MB total_bytes_structured_metadata=0B lines_per_second=402080 total_lines=18935 post_filter_lines=18935 total_entries=201 store_chunks_download_time=392.587Āµs queue_time=107.437Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.677171085Z"}
{"log":"level=info ts=2023-11-19T22:07:07.668743002Z caller=metrics.go:159 component=querier org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=1h7m7.668726012s end_delta=37m9.668726262s step=2s duration=65.478519ms status=200 limit=500 returned_lines=0 throughput=119MB total_bytes=7.8MB total_bytes_structured_metadata=0B lines_per_second=278381 total_lines=18228 post_filter_lines=18228 total_entries=195 store_chunks_download_time=551.543Āµs queue_time=121.962Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.677176642Z"}
{"log":"level=info ts=2023-11-19T22:07:07.670337479Z caller=metrics.go:159 component=frontend org_id=fake traceID=3c4...065 latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=3h0m0s start_delta=2h59m59.355316698s end_delta=-644.683049ms step=2s duration=75.64674ms status=200 limit=500 returned_lines=0 throughput=236MB total_bytes=18MB total_bytes_structured_metadata=0B lines_per_second=551127 total_lines=41691 post_filter_lines=41691 total_entries=1 store_chunks_download_time=944.13Āµs queue_time=1.919999ms splits=7 shards=7 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:07.67718535Z"}
{"log":"level=info ts=2023-11-19T22:07:11.594944506Z caller=table_manager.go:136 index-store=tsdb-2020-10-24 msg=\"uploading tables\"\n","stream":"stderr","time":"2023-11-19T22:07:11.676273248Z"}
{"log":"level=info ts=2023-11-19T22:07:11.595007442Z caller=index_set.go:86 msg=\"uploading table index_19680\"\n","stream":"stderr","time":"2023-11-19T22:07:11.676340263Z"}
{"log":"level=info ts=2023-11-19T22:07:11.595017987Z caller=index_set.go:107 msg=\"finished uploading table index_19680\"\n","stream":"stderr","time":"2023-11-19T22:07:11.676346626Z"}
{"log":"level=info ts=2023-11-19T22:07:11.595026044Z caller=index_set.go:185 msg=\"cleaning up unwanted indexes from table index_19680\"\n","stream":"stderr","time":"2023-11-19T22:07:11.676351229Z"}
{"log":"level=info ts=2023-11-19T22:07:12.729425001Z caller=roundtrip.go:241 org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=3h0m0s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776569762Z"}
{"log":"level=info ts=2023-11-19T22:07:12.734183222Z caller=engine.go:234 component=querier org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776631671Z"}
{"log":"level=info ts=2023-11-19T22:07:12.734818821Z caller=engine.go:234 component=querier org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=22m46s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776637361Z"}
{"log":"level=info ts=2023-11-19T22:07:12.736031981Z caller=engine.go:234 component=querier org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=7m14s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776641789Z"}
{"log":"level=info ts=2023-11-19T22:07:12.737553888Z caller=metrics.go:159 component=querier org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=22m46s start_delta=3h0m0.737547621s end_delta=2h37m14.737547833s step=2s duration=2.623971ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=363.897Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.776646204Z"}
{"log":"level=info ts=2023-11-19T22:07:12.737974328Z caller=engine.go:234 component=querier org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776676384Z"}
{"log":"level=info ts=2023-11-19T22:07:12.74867471Z caller=engine.go:234 component=querier org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776681119Z"}
{"log":"level=info ts=2023-11-19T22:07:12.749778391Z caller=engine.go:234 component=querier org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776685094Z"}
{"log":"level=info ts=2023-11-19T22:07:12.750903491Z caller=engine.go:234 component=querier org_id=fake traceID=676...4cf msg=\"executing query\" type=range query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" length=29m58s step=2s query_hash=3202193437\n","stream":"stderr","time":"2023-11-19T22:07:12.776688879Z"}
{"log":"level=info ts=2023-11-19T22:07:12.751571429Z caller=metrics.go:159 component=querier org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=1h7m12.75156367s end_delta=37m14.751563893s step=2s duration=13.518717ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=31.036Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.776692762Z"}
{"log":"level=info ts=2023-11-19T22:07:12.752137289Z caller=metrics.go:159 component=querier org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=7m14s start_delta=7m12.752130609s end_delta=-1.247869206s step=2s duration=16.00925ms status=200 limit=500 returned_lines=0 throughput=127MB total_bytes=2.0MB total_bytes_structured_metadata=0B lines_per_second=299014 total_lines=4787 post_filter_lines=4787 total_entries=49 store_chunks_download_time=0s queue_time=695.526Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.776698532Z"}
{"log":"level=info ts=2023-11-19T22:07:12.754857512Z caller=metrics.go:159 component=querier org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=2h37m12.754852513s end_delta=2h7m14.754852726s step=2s duration=3.869587ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=10.177266ms splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.776704063Z"}
{"log":"level=info ts=2023-11-19T22:07:12.755839898Z caller=metrics.go:159 component=querier org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=1h37m12.755832006s end_delta=1h7m14.755832259s step=2s duration=5.961109ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=949.207Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.776714015Z"}
{"log":"level=info ts=2023-11-19T22:07:12.756117261Z caller=metrics.go:159 component=querier org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=2h7m12.756112535s end_delta=1h37m14.756112719s step=2s duration=7.366933ms status=200 limit=500 returned_lines=0 throughput=0B total_bytes=0B total_bytes_structured_metadata=0B lines_per_second=0 total_lines=0 post_filter_lines=0 total_entries=0 store_chunks_download_time=0s queue_time=58.604Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.77671959Z"}
{"log":"level=info ts=2023-11-19T22:07:12.765933081Z caller=metrics.go:159 component=querier org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=29m58s start_delta=37m12.765919583s end_delta=7m14.765919831s step=2s duration=31.506087ms status=200 limit=500 returned_lines=0 throughput=188MB total_bytes=5.9MB total_bytes_structured_metadata=0B lines_per_second=439597 total_lines=13850 post_filter_lines=13850 total_entries=148 store_chunks_download_time=0s queue_time=165.512Āµs splits=0 shards=0 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.776725403Z"}
{"log":"level=info ts=2023-11-19T22:07:12.76842301Z caller=metrics.go:159 component=frontend org_id=fake traceID=676...4cf latency=fast query=\"sum(count_over_time({container_name=\\\"dev_container_1\\\"}[2s]))\" query_hash=3202193437 query_type=metric range_type=range length=3h0m0s start_delta=2h59m59.309407874s end_delta=-690.59189ms step=2s duration=38.357699ms status=200 limit=500 returned_lines=0 throughput=208MB total_bytes=8.0MB total_bytes_structured_metadata=0B lines_per_second=485873 total_lines=18637 post_filter_lines=18637 total_entries=1 store_chunks_download_time=0s queue_time=12.442ms splits=7 shards=7 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s\n","stream":"stderr","time":"2023-11-19T22:07:12.776731025Z"}

Hopefully all this can help someone understand the problem.

24

With some tests, iā€™ve found the problem isnā€™t here on loki 2.8.0.

Iā€™ve open an Github issue here: Logs disapear after some time #11274

1 Like
25

The problem seems to be fixed in Loki 2.9.2 so Iā€™m closing this topic.

Thank you all for your help :wink: