Hi
Situation
I’ve been struggling with HA configuration of loki.
I deployed loki-stack with subcharts loki and fluent-bit using ArgoCD. At the very begin, I just changed statefulset replicas from 1 to 3 in loki chart.
It seems working well because there were 3 pods of loki named loki-0, loki-1 and loki-2, bounding with backend storage each.
However, when I tried pull log data using data_source “loki:3100/” through Grafana, it randomly showed nodes.(sometimes there were log_labels I can choose, but sometimes there were nothing to choose)
I made data_source using FQDN form “loki-0.loki-headless.monitor.svc.cluster.local:3100/”. It worked fine pulling log data.
When I tried with data_source “loki-1.loki-headless.monitor.svc.cluster.local:3100/”, there were no log_labels like I went through previously.
If a node, that loki-0 deployed, down there might be no reliable data_source I can use.
Question
Is there any suggestion to make loki in high availablity architecture using loki-stack chart??