Loki monolithic cluster

Grafana Loki
1

Hi Team,

I am trying to create a three node monolithic loki cluster on virtual machines, GCS as object storage for the same. Everything is working fine with the cluster. I am able to integrate Loki with Grafana and check logs, but not able to see log data on the GCS bucket. Please check the below configuration and let me know what I am missing, I have also attached the data of the bucket.
We will be collecting logs from 500+ servers and store it for 100 days, it would be great if you could suggest any performance tuning changes.

loki.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: gcs
  gcs:
      bucket_name: arista-loki
common:
  path_prefix: /opt/loki
  storage:
    gcs:
     bucket_name: loki-bucket
  replication_factor: 3
  ring:
    kvstore:
      store: memberlist
query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

memberlist:
  join_members:
   - loki11001:7946
   - loki11002:7946
   - loki11003:7946

Loki Service

[Unit]
Description=Loki
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/loki -config.file /etc/loki/loki.yaml -log.level=info
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartSec=42s

[Install]
WantedBy=multi-user.target

Content of Loki GCS bucket

[root@loki11001 ~]# gsutil ls -l gs://loki-bucket/*
       251  2023-03-01T03:26:09Z  gs://loki-bucket/loki_cluster_seed.json

gs://loki-bucket/fake/:
       711  2023-03-02T10:30:14Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186a1c3b31b:186a1c3baf3:1bc13193
       707  2023-03-03T10:30:35Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186a6ea10dc:186a6ea18b6:44f8082b
       710  2023-03-04T10:30:06Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186ac106ba0:186ac10737d:23ef1276
       711  2023-03-05T10:30:06Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186b136c605:186b136cddc:81770bc0
       720  2023-03-06T10:30:36Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186b65d253c:186b65d3216:97b4ff2e
       713  2023-03-07T10:30:35Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186bb83801c:186bb8388ed:10a6bcc
       712  2023-03-08T10:30:06Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186c0a9da11:186c0a9e1f5:189fd01a
       714  2023-03-09T10:30:35Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186c5d03764:186c5d04448:11618673
       703  2023-03-10T10:30:35Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186caf69441:186caf69c20:aa2c6b24
       706  2023-03-11T10:30:06Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186d01cef4b:186d01cf721:fd103e59
       712  2023-03-12T10:30:06Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186d54349c1:186d5435197:3deca28f
       715  2023-03-13T09:30:06Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186da32b7ca:186da32bfa1:2c847e4e
       709  2023-03-14T09:30:35Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186df59150b:186df591ddd:c881282c
       711  2023-03-15T09:30:35Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186e47f71e0:186e47f79b5:e89a0e7e
       713  2023-03-16T09:30:36Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186e9a5cdd7:186e9a5dae3:373da62a
       714  2023-03-17T09:30:35Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186eecc29e6:186eecc32b7:12fcc5a7
       709  2023-03-18T09:30:35Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186f3f285d2:186f3f28da9:63d4d7f1
       713  2023-03-19T09:30:35Z  gs://loki-bucket/fake/1426b0a9b8a0010e:186f918e1c5:186f918e99b:2ccf6c38
       887  2023-03-01T11:21:29Z  gs://loki-bucket/fake/19e7983edfbe813c:1869b0f0dbd:1869b1cc92e:f2d72e8d
       857  2023-03-02T07:30:14Z  gs://loki-bucket/fake/19e7983edfbe813c:186a1112434:186a11edf3e:3b7fadd
       840  2023-03-02T08:45:14Z  gs://loki-bucket/fake/19e7983edfbe813c:186a155d01f:186a163899e:b78b0137
       864  2023-03-02T10:00:14Z  gs://loki-bucket/fake/19e7983edfbe813c:186a19a7a20:186a1a8363f:3afc5806
gs://loki-bucket/index/:
                                 gs://loki-bucket/index/index_19416/
                                 gs://loki-bucket/index/index_19417/
                                 gs://loki-bucket/index/index_19418/
                                 gs://loki-bucket/index/index_19419/
                                 gs://loki-bucket/index/index_19420/
                                 gs://loki-bucket/index/index_19421/
                                 gs://loki-bucket/index/index_19422/
                                 gs://loki-bucket/index/index_19423/
                                 gs://loki-bucket/index/index_19424/
                                 gs://loki-bucket/index/index_19425/
                                 gs://loki-bucket/index/index_19426/
                                 gs://loki-bucket/index/index_19427/
                                 gs://loki-bucket/index/index_19428/
                                 gs://loki-bucket/index/index_19429/
                                 gs://loki-bucket/index/index_19430/
                                 gs://loki-bucket/index/index_19431/
                                 gs://loki-bucket/index/index_19432/
                                 gs://loki-bucket/index/index_19433/
                                 gs://loki-bucket/index/index_19434/
                                 gs://loki-bucket/index/index_19435/
                                 gs://loki-bucket/index/index_19436/
2

  1. Not sure what you mean by not seeing data on your GCS, there seems to be data there.

  2. What is your anticipated log volume looking like? If you anticipate vastly different access pattern for read and writer traffic I’d recommend investing in figuring out simple scalable mode, that way you can scale your Loki cluster separately based on read or write traffic. Make sure to tune your chunk idle period to ensure you don’t write too many small chunk files (many small files means slower read performance). And if you anticipate more than a couple of people querying the logs, you’ll want to implement query frontend.

3

Screenshot 2023-03-22 at 2.00.53 AM
Screenshot 2023-03-22 at 2.00.53 AM1812×938 113 KB

Previously I used to see data like this. After deploying to the latest version of Loki I don’t see the directories.

4

What do you see in GCS now? You mentioned you can still see the data from Grafana, is it perhaps writing to local directory?

5

I can only see two directories in GCS bucket /fake/ and /index/. I have already attached a content of GCS bucket in first message
I was also curious is there any cli to check loki cluster health and verify if data is being stored locally or on GCS bucket.

6

/fake and /index is what you should see. /fake is the default organization ID if you don’t enable auth_enabled.

There is a command line tool logcli, but it’s not going to tell you where the data is from. You’ll just have to watch GCS and see if new index / chunks are written and check your writer logs.

1 Like
7

Thanks Tony for the help.

1 Like
9

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.