How does one test Loki storage is correctly configured?

Grafana Loki
1

I’m trying to validate storage configuration (specifically Azure) in isolation, using a “minimal” Loki instance running in a Docker container. I’m using the “local” config file provided in the “Install Loki with Docker or Docker Compose” documentation, here, and then modifying it with Azure storage configuration, shown below.

I’m having trouble validating that the configuration works. The behavior I’m seeing doesn’t match my expectations, and so before digging deeper I need to validate that my expectations are correct. Here’s my test:

  1. Start the Loki container: docker run --name loki -v .:/mnt/config -p 3100:3100 grafana/loki:3.0.0 --config.file=/mnt/config/loki-config.yaml
  2. curl http://localhost:3100/ready until the response is ready
  3. curl -X POST http://localhost:3100/loki/api/v1/push -H "Content-Type: application/json" -d @log.json where log.json has a log payload containing entries with timestamps from around now: e.g.
{
    "streams": [
      {
        "stream": {
          "label": "foo"
        },
        "values": [
            [ "1727285016000000000", "Foo bar baz" ],
            [ "1727285016000000001", "Biz baz boz" ]
        ]
      }
    ]
  }
  1. curl -G -s "http://localhost:3100/loki/api/v1/query" --data-urlencode 'query=sum(rate({label="foo"}[90m])) by (level)' | jq > query.json and verify that my push is reflected in the results.
  2. Verify no errors written in the output of the Docker container.
  3. Stop the container, delete it, and recreate. Wait for it to be ready, and re-query.

Expected:
The query after the recreate returns similar results (because the logs were persisted).

Actual:
The query after the recreate doesn’t return any entries/lines. Additionally, the only thing in the Blob storage container is a loki_cluster_seed.json file, with a size of 251 bytes that never changes.

Provisional Conclusion:
Loki is not configured correctly for persisting logs.

Configuration:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    azure:
        account_key: <key>
        account_name: <account name>
        container_name: loki
        request_timeout: 0
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
  - from: "2020-12-11"
    index:
      period: 24h
      prefix: index_
    object_store: azure
    schema: v13
    store: tsdb
storage_config:
  tsdb_shipper:
    active_index_directory: /tmp/loki/tsdb-index
    cache_location: /tmp/loki/tsdb-cache
ruler:
  alertmanager_url: http://localhost:9093

I’ve also performed the same test with the common.storage.azure object defined in storage_config instead, with the same results.

2

Logs aren’t written into persistent storage right away. There are a couple of things that affect the duration of it, such as desired chunk size and chunk idle time. By default if you only send one line of log it’ll be written to persistent storage after 30 minutes, so you should at least wait that period of time before you re-create your loki instance, or modify the configuration chunk_idle_period.

3

you should be able to thoroughly test and verify that your Loki storage is correctly configured and functioning as intended. If you encounter any issues, refer to the logs for troubleshooting and consult the Loki documentation for further guidance on configuration options and best practices onstream.