Loki "forgets" logs

[crown0815]

I am encountering a rather odd behavior from Grafana Loki.

We are currently testing a setup with Grafana Loki installed through docker on server A, and Grafana installed through docker on server B.

We added Loki as a data source to Grafana and were able to find the logs that we posted to it, but just a few hours later they could no longer be found. We already observed this a few days ago.

Today, I changed the query to include the last 30 days (which I knew ought to contain logs) and was able to resolve logs in the morning, but when I came back to the page a few hours later and reran the query the logs were gone (the logs were from 4 and 9 days ago, so they did not rotate out of the 30 day range). At this point, I am not able to find those logs anymore. Neither the Loki, nor the Grafana server have been restarted in between the two queries. The Loki server has been restarted within the last few days multiple times.

For completeness, here are the configuration files: docker-compose.yaml for Loki:

services:
  loki:
    image: grafana/loki:2.9.0
    user: "1000"
    ports:
      - "443:443"
    volumes:
      - ./loki-config.yaml:/mnt/config/loki-config.yaml
      - ./certificates:/etc/loki/certificates
      - ./loki-data:/loki
    command: -config.file=/mnt/config/loki-config.yaml
    restart: unless-stopped

and loki-config.yaml

auth_enabled: false

server:
  grpc_listen_port: 9096
  http_listen_port: 443
  http_tls_config: &tls_server_config
    cert_file: /etc/loki/certificates/fullchain.pem
    key_file: /etc/loki/certificates/privkey.pem

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

Most of the Loki configuration was taken from an example and to be fair I do not fully understand it. So far I assumed, that storing everything in `/loki/ should be sufficient and it seems like it is stored for longer time (see my query that found logs from 4 and 9 days ago).

Regarding the queries themselves, they have been fairly simple:

{application="testing"}

I know that the application label with the testing value was logged because I was able to find the logs initially.

Another side note, when I save & test the Loki settings within Grafana, the error

Data source connected, but no labels were received. Verify that Loki and Promtail are correctly configured.

I am not using prom tail to feed in logs, but am sending them directly from a .NET application using the serilog-contrib/serilog-sinks-grafana-loki library.

Thank you very much for your help!

[tonyswumac]

1. Enable debug log and see if there is anything obvious.
2. Check your chunk store and see if older chunks are still there.
3. Make sure your Loki container is allowed to use a lot of file handlers.