Writes To Loki (Simple Scalable Architecture) Fail With 503 Error

Grafana Loki
1

I am trying setup Loki in a simple scalable architecture on Kubernetes. My intentions are to have:

  1. One loki-gateway
  2. Two loki-read
  3. One loki-write
  4. One loki-backend

I am able to run helm and get the pods setup and running as expected. However, when I attempt to write to Loki, I get 503 response from the gateway. I have a feeling my config map or member list is not setup right. This is my first attempt at deploying Loki and its all new to me.

This is the curl command I use to write to Loki:

curl -v -H "Content-Type: application/json" -XPOST -s "http://{kubernetes-cluster-domain}/loki/api/v1/push" --data-raw '{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "1570818238000000000", "fizzbuzz" ] ] }]}'

This is what runtime-configmap.yaml looks like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: loki-runtime
  labels:
    helm.sh/chart: loki-5.5.0
    app.kubernetes.io/name: loki
    app.kubernetes.io/instance: my-loki-loki
    app.kubernetes.io/version: "2.8.2"
    app.kubernetes.io/managed-by: Helm
data:
  runtime-config.yaml: |
    
    {}

service-memberlist.yaml

apiVersion: v1
kind: Service
metadata:
  name: loki-memberlist
  labels:
    helm.sh/chart: loki-5.5.0
    app.kubernetes.io/name: loki
    app.kubernetes.io/instance: my-loki-loki
    app.kubernetes.io/version: "2.8.2"
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - name: tcp
      port: 7946
      targetPort: http-memberlist
      protocol: TCP
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/name: loki
    app.kubernetes.io/instance: my-loki-loki
    app.kubernetes.io/part-of: memberlist

service-gateway.yaml:

apiVersion: v1
kind: Service
metadata:
  name: loki-gateway
  labels:
    helm.sh/chart: loki-5.5.0
    app.kubernetes.io/name: loki
    app.kubernetes.io/instance: my-loki-loki
    app.kubernetes.io/version: "2.8.2"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: gateway
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 80
      targetPort: http
      protocol: TCP
  selector:
    app.kubernetes.io/name: loki
    app.kubernetes.io/instance: my-loki-loki
    app.kubernetes.io/component: gateway

service-write-headless.yaml

apiVersion: v1
kind: Service
metadata:
  name: loki-write-headless
  labels:
    app.kubernetes.io/name: loki
    app.kubernetes.io/instance: my-loki-loki
    app.kubernetes.io/component: write
    prometheus.io/service-monitor: "false"
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - name: http-metrics
      port: 3100
      targetPort: http-metrics
      protocol: TCP
    - name: grpc
      port: 9095
      targetPort: grpc
      protocol: TCP
      appProtocol: tcp
  selector:
    app.kubernetes.io/name: loki
    app.kubernetes.io/instance: my-loki-loki
    app.kubernetes.io/component: write

service-write.yaml

apiVersion: v1
kind: Service
metadata:
  name: loki-write
  labels:
    helm.sh/chart: loki-5.5.0
    app.kubernetes.io/name: loki
    app.kubernetes.io/instance: my-loki-loki
    app.kubernetes.io/version: "2.8.2"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: write
spec:
  type: ClusterIP
  ports:
    - name: http-metrics
      port: 3100
      targetPort: http-metrics
      protocol: TCP
    - name: grpc
      port: 9095
      targetPort: grpc
      protocol: TCP
  selector:
    app.kubernetes.io/name: loki
    app.kubernetes.io/instance: my-loki-loki
    app.kubernetes.io/component: write

Any help will be greatly appreciated.