Too many unhealthy instances in the ring

Grafana Configuration
,
1

  • What Grafana version and what operating system are you using?

  • What are you trying to achieve?

Install a full LGTM observability stack in a kubernetes cluster

All components are installed and are running, but when I try to access the Mimir or Loki datasources from grafana, I get a too many unhealthy instances in the ring error

  • What did you expect to happen?

This to work

  • Can you copy/paste the configuration(s) that you are having problems with?
grafana:
  enabled: true
  datasources:
    datasources.yaml:
      apiVersion: 1
      datasources:
        - name: Loki
          uid: loki
          type: loki
          url: http://{{ .Release.Name }}-loki-gateway
          isDefault: false
        - name: Mimir
          uid: prom
          type: prometheus
          url: http://{{ .Release.Name }}-mimir-nginx/prometheus
          isDefault: true
        - name: Tempo
          uid: tempo
          type: tempo
          url: http://{{ .Release.Name }}-tempo-query-frontend:3100
          isDefault: false
          jsonData:
            tracesToLogsV2:
              datasourceUid: loki
            lokiSearch:
              datasourceUid: loki
            tracesToMetrics:
              datasourceUid: prom
            serviceMap:
              datasourceUid: prom
loki:
  enabled: true
mimir:
  enabled: true
  alertmanager:
    enabled: true
    replicas: 1
    resources:
      requests:
        cpu: 20m
    persistentVolume:
      enabled: false
  compactor:
    resources:
      requests:
        cpu: 20m
    persistentVolume:
      enabled: false
  distributor:
    resources:
      requests:
        cpu: 20m
  ingester:
    replicas: 1
    zoneAwareReplication:
      enabled: false
    resources:
      requests:
        cpu: 20m
    persistentVolume:
      enabled: false
  overrides_exporter:
    resources:
      requests:
        cpu: 20m
  querier:
    replicas: 1
    resources:
      requests:
        cpu: 20m
  query_frontend:
    resources:
      requests:
        cpu: 20m
  query_scheduler:
    replicas: 1
    resources:
      requests:
        cpu: 20m
  ruler:
    resources:
      requests:
        cpu: 20m
  store_gateway:
    zoneAwareReplication:
      enabled: false
    resources:
      requests:
        cpu: 20m
    persistentVolume:
      enabled: false
  minio:
    resources:
      requests:
        cpu: 20m
    persistence:
      size: 20Gi
      storageClass: gp2
  rollout_operator:
    resources:
      requests:
        cpu: 20m
tempo:
  enabled: true
  ingester:
    replicas: 1

grafana-oncall:
  enabled: false
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

In grafana UI “too many unhealthy instances in the ring”.

I can’t see any other error in the logs of the pods. And this is the status reported by mimir

image
image1377×881 92.6 KB

  • Did you follow any online instructions? If so, what is the URL?
2

Apparently, the problem was that the ingester can’t run with one replica. The minimum seems to be 2 but this is not documented and it is not validated by the chart.