Two loki instances on same cluster

Grafana Loki
1

At the moment we have one loki with 4-5 promtails feeding logs into that one loki instance. That loki use object storage minio (in loki namespace). We want to separate them, to control retention easier and to save logs in different buckets in s3. I couldn’t find any other solution, before we used retention_stream, but seems that option is no longer available.

old rentention_stream:

    # retention_stream:
    # - selector: '{namespace="dev"}'
    #   priority: 1
    #   period: 24h
    # - selector: '{namespace="staging"}'
    #   priority: 1
    #   period: 336h
    # - selector: '{namespace="prod"}'
    #   priority: 1
    #   period: 336h

But now we want to separate dev, staging and prod environments, each in a different loki instance (all loki instances will be on the same cluster). When deploying second loki (in different namespace), backend wont connect to the minio instance with following error:

 failed to init delete store: failed to get s3 object: RequestError: send request failed                                                                                                                                                              
 caused by: Get "http://chunkssys.loki-minio.loki.svc:9000/index/delete_requests/delete_requests.gz": 3 errors occurred:                                                                                                                               
     * dial tcp: lookup chunkssys.loki-minio.loki.svc on 10.43.0.10:53: no such host                                                                                                                                                                   
     * dial tcp: lookup chunkssys.loki-minio.loki.svc on 10.43.0.10:53: no such host                                                                                                                                                                   
     * dial tcp: lookup chunkssys.loki-minio.loki.svc on 10.43.0.10:53: no such host

Why does it add chunkssys in the s3 url for minio?

This bucket exists in minio, created with the helm deployment (from loki).

minio:
  enabled: true
  buckets:
    - name: chunks
      policy: none
      purge: false
    - name: ruler
      policy: none
      purge: false
    - name: admin
      policy: none
      purge: false
    - name: chunkssys
      policy: none
      purge: false
    - name: rulersys
      policy: none
      purge: false
    - name: adminsys
      policy: none
      purge: false

Chart version: 5.31.0
Loki version: 2.9.2

Do you have any suggestions how we can achieve that?

2

You kinda answered your own question. I’d double check configurations for all your Loki instances, and make sure they are configured with the intended minio storage.

3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.