Loki write pods does not get into ready state

Grafana Loki
1

Hello,

I just deployed Loki using helm chart on an eks cluster. Almost all got up and running except the write pods which are failing the **Readiness probe failed: HTTP probe failed with statuscode: 503** . In the logs I can get something like this:

level=error ts=2024-01-11T18:15:56.892357205Z caller=flush.go:143 org_id=fake msg="failed to flush" err="failed to flush chunks: store put chunk: WebIdentityErr: failed to retrieve credentials\ncaused by: SerializationError: failed to unmarshal error message\n\tstatus code: 405, request id: \ncaused by: UnmarshalError: failed to unmarshal error message\n\t00000000  3c 3f 78 6d 6c 20 76 65  72 73 69 6f 6e 3d 22 31  |<?xml version=\"1|\n00000010  2e 30 22 20 65 6e 63 6f  64 69 6e 67 3d 22 55 54  |.0\" encoding=\"UT|\n00000020  46 2d 38 22 3f 3e 0a 3c  45 72 72 6f 72 3e 3c 43  |F-8\"?>.<Error><C|\n00000030  6f 64 65 3e 4d 65 74 68  6f 64 4e 6f 74 41 6c 6c  |ode>MethodNotAll|\n00000040  6f 77 65 64 3c 2f 43 6f  64 65 3e 3c 4d 65 73 73  |owed</Code><Mess|\n00000050  61 67 65 3e 54 68 65 20  73 70 65 63 69 66 69 65  |age>The specifie|\n00000060  64 20 6d 65 74 68 6f 64  20 69 73 20 6e 6f 74 20  |d method is not |\n00000070  61 6c 6c 6f 77 65 64 20  61 67 61 69 6e 73 74 20  |allowed against |\n00000080  74 68 69 73 20 72 65 73  6f 75 72 63 65 2e 3c 2f  |this resource.</|\n00000090  4d 65 73 73 61 67 65 3e  3c 4d 65 74 68 6f 64 3e  |Message><Method>|\n000000a0  50 4f 53 54 3c 2f 4d 65  74 68 6f 64 3e 3c 52 65  |POST</Method><Re|\n000000b0  73 6f 75 72 63 65 54 79  70 65 3e 53 45 52 56 49  |sourceType>SERVI|\n000000c0  43 45 3c 2f 52 65 73 6f  75 72 63 65 54 79 70 65  |CE</ResourceType|\n000000d0  3e 3c 52 65 71 75 65 73  74 49 64 3e 34 58 53 37  |><RequestId>4XS7|\n000000e0  44 35 4e 53 47 32 54 53  58 39 32 51 3c 2f 52 65  |D5NSG2TSX92Q</Re|\n000000f0  71 75 65 73 74 49 64 3e  3c 48 6f 73 74 49 64 3e  |questId><HostId>|\n00000100  59 31 63 32 73 33 34 58  69 41 2f 51 66 75 4a 6a  |Y1c2s34XiA/QfuJj|\n00000110  2f 56 44 4d 77 4c 66 55  44 74 32 6a 4e 62 62 50  |/VDMwLfUDt2jNbbP|\n00000120  4e 5a 65 46 49 56 34 75  32 4c 2b 65 57 42 73 73  |NZeFIV4u2L+eWBss|\n00000130  45 6d 41 6a 42 75 58 32  58 4f 72 35 44 6b 77 62  |EmAjBuX2XOr5Dkwb|\n00000140  38 48 68 7a 45 45 73 43  6e 4b 38 3d 3c 2f 48 6f  |8HhzEEsCnK8=</Ho|\n00000150  73 74 49 64 3e 3c 2f 45  72 72 6f 72 3e           |stId></Error>|\n\ncaused by: unknown error response tag, {{ Error} []}, num_chunks: 2, labels: {app=\"prometheus\", container=\"config-reloader\", filename=\"/var/log/pods/monitoring_prometheus-prometheus-kube-prometheus-prometheus-0_d8acbc12-277d-497b-927c-982bd9360613/config-reloader/0.log\", instance=\"prometheus-kube-prometheus-prometheus\", job=\"monitoring/prometheus\", namespace=\"monitoring\", node_name=\"ip-172-21-15-42.ec2.internal\", pod=\"prometheus-prometheus-kube-prometheus-prometheus-0\", stream=\"stdout\"}"
level=info ts=2024-01-11T18:15:56.892437306Z caller=flush.go:167 msg="flushing stream" user=fake fp=12133b108c9b4096 immediate=true num_chunks=1 labels="{app=\"timescaledb\", container=\"timescaledb-full\", filename=\"/var/log/pods/test_timescaledb-full-weekly-28261344-5w67h_c4fbc583-e893-4ac7-8626-b877649cc4f7/timescaledb-full/2.log\", job=\"test/timescaledb\", namespace=\"test\", node_name=\"ip-172-21-15-42.ec2.internal\", pod=\"timescaledb-full-weekly-28261344-5w67h\", stream=\"stderr\"}"
level=info ts=2024-01-11T18:15:56.892514196Z caller=flush.go:167 msg="flushing stream" user=fake fp=b4fd60a8141d0278 immediate=true num_chunks=1 labels="{app=\"timescaledb\", container=\"timescaledb-full\", filename=\"/var/log/pods/test_timescaledb-full-weekly-28260945-rkdhn_ff7b2ee6-350f-4d43-a8fb-4e75a46e5f85/timescaledb-full/2.log\", job=\"test/timescaledb\", namespace=\"test\", node_name=\"ip-172-21-15-42.ec2.internal\", pod=\"timescaledb-full-weekly-28260945-rkdhn\", stream=\"stderr\"}"

but nothing about the readiness probes.
When I run a curl to the /ready I get the following output:

Some services are not Running: 
Running: 8 
Starting: 1

Any suggestion on how can I fix it?

2

Sounds like your storage configuration may be incorrect. Can you share your configuration please?

3

figured it out. Also had to modify the IAM role. Thank you for your reply.

4

The question how would I differentiate the logs from which eks is comming from?

5

Can you tell me how you resolved this? Same error here.

6

I was using IAM role to give access to the s3 bucket and we had to give more access to the role. So the fix has to do with proper access to the s3 bucket.

7

Can you tell me how you resolved this? Same error here.

kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: promtail
  namespace: observability
  uid: b30306a6-16c6-4cdf-963a-2032252e109a
  resourceVersion: '24899106'
  generation: 1
  creationTimestamp: '2024-02-09T16:36:35Z'
  labels:
    app.kubernetes.io/instance: promtail
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: promtail
    app.kubernetes.io/version: 2.9.3
    helm.sh/chart: promtail-6.15.5
  annotations:
    deprecated.daemonset.template.generation: '1'
    meta.helm.sh/release-name: promtail
    meta.helm.sh/release-namespace: observability
  managedFields:
    - manager: helm
      operation: Update
      apiVersion: apps/v1
      time: '2024-02-09T16:36:35Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:deprecated.daemonset.template.generation: {}
            f:meta.helm.sh/release-name: {}
            f:meta.helm.sh/release-namespace: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/instance: {}
            f:app.kubernetes.io/managed-by: {}
            f:app.kubernetes.io/name: {}
            f:app.kubernetes.io/version: {}
            f:helm.sh/chart: {}
        f:spec:
          f:revisionHistoryLimit: {}
          f:selector: {}
          f:template:
            f:metadata:
              f:annotations:
                .: {}
                f:checksum/config: {}
              f:labels:
                .: {}
                f:app.kubernetes.io/instance: {}
                f:app.kubernetes.io/name: {}
            f:spec:
              f:containers:
                k:{"name":"promtail"}:
                  .: {}
                  f:args: {}
                  f:env:
                    .: {}
                    k:{"name":"HOSTNAME"}:
                      .: {}
                      f:name: {}
                      f:valueFrom:
                        .: {}
                        f:fieldRef: {}
                  f:image: {}
                  f:imagePullPolicy: {}
                  f:name: {}
                  f:ports:
                    .: {}
                    k:{"containerPort":3101,"protocol":"TCP"}:
                      .: {}
                      f:containerPort: {}
                      f:name: {}
                      f:protocol: {}
                  f:readinessProbe:
                    .: {}
                    f:failureThreshold: {}
                    f:httpGet:
                      .: {}
                      f:path: {}
                      f:port: {}
                      f:scheme: {}
                    f:initialDelaySeconds: {}
                    f:periodSeconds: {}
                    f:successThreshold: {}
                    f:timeoutSeconds: {}
                  f:resources: {}
                  f:securityContext:
                    .: {}
                    f:allowPrivilegeEscalation: {}
                    f:capabilities:
                      .: {}
                      f:drop: {}
                    f:readOnlyRootFilesystem: {}
                  f:terminationMessagePath: {}
                  f:terminationMessagePolicy: {}
                  f:volumeMounts:
                    .: {}
                    k:{"mountPath":"/etc/promtail"}:
                      .: {}
                      f:mountPath: {}
                      f:name: {}
                    k:{"mountPath":"/run/promtail"}:
                      .: {}
                      f:mountPath: {}
                      f:name: {}
                    k:{"mountPath":"/var/lib/docker/containers"}:
                      .: {}
                      f:mountPath: {}
                      f:name: {}
                      f:readOnly: {}
                    k:{"mountPath":"/var/log/pods"}:
                      .: {}
                      f:mountPath: {}
                      f:name: {}
                      f:readOnly: {}
              f:dnsPolicy: {}
              f:enableServiceLinks: {}
              f:restartPolicy: {}
              f:schedulerName: {}
              f:securityContext:
                .: {}
                f:runAsGroup: {}
                f:runAsUser: {}
              f:serviceAccount: {}
              f:serviceAccountName: {}
              f:terminationGracePeriodSeconds: {}
              f:tolerations: {}
              f:volumes:
                .: {}
                k:{"name":"config"}:
                  .: {}
                  f:name: {}
                  f:secret:
                    .: {}
                    f:defaultMode: {}
                    f:secretName: {}
                k:{"name":"containers"}:
                  .: {}
                  f:hostPath:
                    .: {}
                    f:path: {}
                    f:type: {}
                  f:name: {}
                k:{"name":"pods"}:
                  .: {}
                  f:hostPath:
                    .: {}
                    f:path: {}
                    f:type: {}
                  f:name: {}
                k:{"name":"run"}:
                  .: {}
                  f:hostPath:
                    .: {}
                    f:path: {}
                    f:type: {}
                  f:name: {}
          f:updateStrategy:
            f:rollingUpdate:
              .: {}
              f:maxSurge: {}
              f:maxUnavailable: {}
            f:type: {}
    - manager: kube-controller-manager
      operation: Update
      apiVersion: apps/v1
      time: '2024-02-14T13:07:12Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:currentNumberScheduled: {}
          f:desiredNumberScheduled: {}
          f:numberAvailable: {}
          f:numberReady: {}
          f:numberUnavailable: {}
          f:observedGeneration: {}
          f:updatedNumberScheduled: {}
      subresource: status
spec:
  selector:
    matchLabels:
      app.kubernetes.io/instance: promtail
      app.kubernetes.io/name: promtail
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: promtail
        app.kubernetes.io/name: promtail
      annotations:
        checksum/config: 2ef0f14afc8ed4b72495a244ef20e42a4bc14afd488033fe94e04c341b97529a
    spec:
      volumes:
        - name: config
          secret:
            secretName: promtail
            defaultMode: 420
        - name: run
          hostPath:
            path: /run/promtail
            type: ''
        - name: containers
          hostPath:
            path: /var/lib/docker/containers
            type: ''
        - name: pods
          hostPath:
            path: /var/log/pods
            type: ''
      containers:
        - name: promtail
          image: docker.io/grafana/promtail:2.9.3
          args:
            - '-config.file=/etc/promtail/promtail.yaml'
          ports:
            - name: http-metrics
              containerPort: 3101
              protocol: TCP
          env:
            - name: HOSTNAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
          resources: {}
          volumeMounts:
            - name: config
              mountPath: /etc/promtail
            - name: run
              mountPath: /run/promtail
            - name: containers
              readOnly: true
              mountPath: /var/lib/docker/containers
            - name: pods
              readOnly: true
              mountPath: /var/log/pods
          readinessProbe:
            httpGet:
              path: /ready
              port: http-metrics
              scheme: HTTP
            initialDelaySeconds: 10
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 5
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              drop:
                - ALL
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: promtail
      serviceAccount: promtail
      securityContext:
        runAsUser: 0
        runAsGroup: 0
      schedulerName: default-scheduler
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
      enableServiceLinks: true
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 0
  revisionHistoryLimit: 10
status:
  currentNumberScheduled: 16
  numberMisscheduled: 0
  desiredNumberScheduled: 16
  numberReady: 15
  observedGeneration: 1
  updatedNumberScheduled: 16
  numberAvailable: 15
  numberUnavailable: 1
8

Get error

level=warn ts=2024-02-14T12:56:01.29122951Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T12:56:15.880214534Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T12:56:38.801582325Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T12:56:55.298525149Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T12:57:46.536263223Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T12:59:00.239110929Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:02:44.367017881Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=error ts=2024-02-14T13:02:44.367060669Z caller=client.go:430 component=client host=loki-gateway msg="final error sending batch" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:02:45.386267559Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:02:46.266937527Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:02:57.622440626Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:03:00.268672021Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:03:15.923348769Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:03:35.544348244Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:03:58.511198812Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:04:44.29905186Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:06:46.912740573Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:09:35.535411128Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=error ts=2024-02-14T13:09:35.535469886Z caller=client.go:430 component=client host=loki-gateway msg="final error sending batch" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:09:43.819323719Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:09:44.704950618Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:09:53.291821659Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:09:56.020764517Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:10:03.562996534Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:10:22.016775966Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:10:52.175425586Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:11:39.444773648Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:13:25.967490592Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:17:09.523304782Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=error ts=2024-02-14T13:17:09.523340021Z caller=client.go:430 component=client host=loki-gateway msg="final error sending batch" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:17:20.584789183Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:17:31.174531171Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:17:39.467062575Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:17:49.451097041Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:17:55.492331987Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:18:20.938906029Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:18:57.620944747Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:20:04.956056356Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:21:46.371829832Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:24:45.625712299Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=error ts=2024-02-14T13:24:45.62574786Z caller=client.go:430 component=client host=loki-gateway msg="final error sending batch" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:24:56.685456042Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:24:58.539436952Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:25:09.988064524Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:25:20.523520133Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:25:36.022127149Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:25:57.213450304Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:26:28.877360361Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:27:17.674981346Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:28:41.812478573Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:32:37.396143992Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=error ts=2024-02-14T13:32:37.396194618Z caller=client.go:430 component=client host=loki-gateway msg="final error sending batch" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:32:41.519271349Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:32:49.547162011Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:33:00.76148981Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:33:07.183546398Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:33:21.29289927Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:33:41.384869637Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:34:16.168067097Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:35:13.707063325Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:37:00.68855199Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:41:12.133246521Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=error ts=2024-02-14T13:41:12.133286887Z caller=client.go:430 component=client host=loki-gateway msg="final error sending batch" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:41:16.203169405Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:41:24.107291619Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:41:32.559184592Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:41:44.610737129Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:41:49.129518889Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:42:07.26469013Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:42:38.091340101Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:43:20.795431305Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:45:25.769382278Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:49:35.075399232Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=error ts=2024-02-14T13:49:35.075440566Z caller=client.go:430 component=client host=loki-gateway msg="final error sending batch" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:49:36.086413199Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:49:46.604991582Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:49:47.74550944Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:49:50.158866509Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:49:56.697263144Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:50:12.331135299Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:50:40.328659758Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:51:39.21517343Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:53:31.247180238Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:56:31.115836557Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=error ts=2024-02-14T13:56:31.115884769Z caller=client.go:430 component=client host=loki-gateway msg="final error sending batch" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:56:32.186597171Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:56:33.142236699Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:56:44.396154079Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:56:53.901263828Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:57:11.49930075Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T13:57:26.234010977Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:58:00.20303977Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T13:58:59.76246498Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T14:00:56.536483324Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=-1 tenant= error="Post \"http://loki-gateway/loki/api/v1/push\": context deadline exceeded"
level=warn ts=2024-02-14T14:03:44.159667521Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=error ts=2024-02-14T14:03:44.159705431Z caller=client.go:430 component=client host=loki-gateway msg="final error sending batch" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T14:03:46.187050165Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T14:03:54.255189383Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T14:03:56.523048204Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
level=warn ts=2024-02-14T14:04:01.899290662Z caller=client.go:419 component=client host=loki-gateway msg="error sending batch, will retry" status=502 tenant= error="server returned HTTP status 502 Bad Gateway (502): <html>"
9

can you please add the output of kubectl get pods -n observability?

Mostly interested in the loki pod states.

1 Like
10

loki
loki878×688 106 KB

11

This does not seam to be the same issue as I was having the pods constnatly restarting and never getting into ready state.

12

The promtail doesn’t seem to want to be executed

promtail
promtail1119×304 22.3 KB

13

loki-0
loki-01214×255 21.8 KB

I redid the Loki installation now with the stack - helm install grafana/loki-stack, but I still have errors

log error

level=info ts=2024-02-27T17:45:31.635765382Z caller=main.go:103 msg="Starting Loki" version="(version=2.6.1, branch=HEAD, revision=6bd05c9a4)"
level=info ts=2024-02-27T17:45:31.636231214Z caller=server.go:288 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2024-02-27T17:45:31.636663212Z caller=modules.go:736 msg="RulerStorage is not configured in single binary mode and will not be started."
level=warn ts=2024-02-27T17:45:31.645357126Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksfifocache"
level=info ts=2024-02-27T17:45:31.646605071Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.574µs distinct_users_len=0
level=info ts=2024-02-27T17:45:31.647049677Z caller=shipper.go:124 msg="starting index shipper in RW mode"
level=info ts=2024-02-27T17:45:31.647107048Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2024-02-27T17:45:31.647679675Z caller=shipper_index_client.go:79 msg="starting boltdb shipper in RW mode"
level=info ts=2024-02-27T17:45:31.64779592Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2024-02-27T17:45:31.647983175Z caller=modules.go:761 msg="RulerStorage is nil.  Not starting the ruler."
level=info ts=2024-02-27T17:45:31.649559659Z caller=worker.go:112 msg="Starting querier worker using query-scheduler and scheduler ring for addresses"
level=info ts=2024-02-27T17:45:31.651885279Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2024-02-27T17:45:31.652692371Z caller=module_service.go:82 msg=initialising module=query-frontend-tripperware
level=info ts=2024-02-27T17:45:31.653181423Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=error ts=2024-02-27T17:45:31.654030419Z caller=loki.go:386 msg="module failed" module=memberlist-kv error="invalid service state: Failed, expected: Running, failure: service &{0xc000950d20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc00011aa00 loki false {    false}} 0xc00011aa00 loki [{ringDesc 0xe43cc0} {}]} 0xc0000a1a90 0xc00011aa00 0xc0003ddb20 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0000d70e0 0xc000959c20 0xc000959ce0 0xc000959da0 0xc000959e60 0xc0009660c0 0xc000959f20 0xc000966180 0xc000966000 0xc000936d00 0xc000936d40 0xc000966300 0xc0009663c0 0xc000966540 0xc000966480 0xc00000fea8 0xc0004aa620 0xc00000fe98 0xc00000fea0 0xc000936d80 0xc000936dc0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-02-27T17:45:31.654197017Z caller=loki.go:386 msg="module failed" module=ingester error="failed to start ingester, because it depends on module store, which has failed: context canceled"
level=error ts=2024-02-27T17:45:31.654357344Z caller=loki.go:386 msg="module failed" module=ingester-querier error="failed to start ingester-querier, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc000950d20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc00011aa00 loki false {    false}} 0xc00011aa00 loki [{ringDesc 0xe43cc0} {}]} 0xc0000a1a90 0xc00011aa00 0xc0003ddb20 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0000d70e0 0xc000959c20 0xc000959ce0 0xc000959da0 0xc000959e60 0xc0009660c0 0xc000959f20 0xc000966180 0xc000966000 0xc000936d00 0xc000936d40 0xc000966300 0xc0009663c0 0xc000966540 0xc000966480 0xc00000fea8 0xc0004aa620 0xc00000fe98 0xc00000fea0 0xc000936d80 0xc000936dc0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-02-27T17:45:31.654911118Z caller=loki.go:386 msg="module failed" module=distributor error="failed to start distributor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc000950d20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc00011aa00 loki false {    false}} 0xc00011aa00 loki [{ringDesc 0xe43cc0} {}]} 0xc0000a1a90 0xc00011aa00 0xc0003ddb20 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0000d70e0 0xc000959c20 0xc000959ce0 0xc000959da0 0xc000959e60 0xc0009660c0 0xc000959f20 0xc000966180 0xc000966000 0xc000936d00 0xc000936d40 0xc000966300 0xc0009663c0 0xc000966540 0xc000966480 0xc00000fea8 0xc0004aa620 0xc00000fe98 0xc00000fea0 0xc000936d80 0xc000936dc0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-02-27T17:45:31.655111378Z caller=loki.go:386 msg="module failed" module=store error="failed to start store, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc000950d20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc00011aa00 loki false {    false}} 0xc00011aa00 loki [{ringDesc 0xe43cc0} {}]} 0xc0000a1a90 0xc00011aa00 0xc0003ddb20 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0000d70e0 0xc000959c20 0xc000959ce0 0xc000959da0 0xc000959e60 0xc0009660c0 0xc000959f20 0xc000966180 0xc000966000 0xc000936d00 0xc000936d40 0xc000966300 0xc0009663c0 0xc000966540 0xc000966480 0xc00000fea8 0xc0004aa620 0xc00000fe98 0xc00000fea0 0xc000936d80 0xc000936dc0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-02-27T17:45:31.655453111Z caller=loki.go:386 msg="module failed" module=query-scheduler error="failed to start query-scheduler, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc000950d20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc00011aa00 loki false {    false}} 0xc00011aa00 loki [{ringDesc 0xe43cc0} {}]} 0xc0000a1a90 0xc00011aa00 0xc0003ddb20 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0000d70e0 0xc000959c20 0xc000959ce0 0xc000959da0 0xc000959e60 0xc0009660c0 0xc000959f20 0xc000966180 0xc000966000 0xc000936d00 0xc000936d40 0xc000966300 0xc0009663c0 0xc000966540 0xc000966480 0xc00000fea8 0xc0004aa620 0xc00000fe98 0xc00000fea0 0xc000936d80 0xc000936dc0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-02-27T17:45:31.656030916Z caller=loki.go:386 msg="module failed" module=compactor error="failed to start compactor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc000950d20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc00011aa00 loki false {    false}} 0xc00011aa00 loki [{ringDesc 0xe43cc0} {}]} 0xc0000a1a90 0xc00011aa00 0xc0003ddb20 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0000d70e0 0xc000959c20 0xc000959ce0 0xc000959da0 0xc000959e60 0xc0009660c0 0xc000959f20 0xc000966180 0xc000966000 0xc000936d00 0xc000936d40 0xc000966300 0xc0009663c0 0xc000966540 0xc000966480 0xc00000fea8 0xc0004aa620 0xc00000fe98 0xc00000fea0 0xc000936d80 0xc000936dc0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-02-27T17:45:31.656143505Z caller=loki.go:386 msg="module failed" module=usage-report error="failed to start usage-report, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc000950d20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc00011aa00 loki false {    false}} 0xc00011aa00 loki [{ringDesc 0xe43cc0} {}]} 0xc0000a1a90 0xc00011aa00 0xc0003ddb20 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0000d70e0 0xc000959c20 0xc000959ce0 0xc000959da0 0xc000959e60 0xc0009660c0 0xc000959f20 0xc000966180 0xc000966000 0xc000936d00 0xc000936d40 0xc000966300 0xc0009663c0 0xc000966540 0xc000966480 0xc00000fea8 0xc0004aa620 0xc00000fe98 0xc00000fea0 0xc000936d80 0xc000936dc0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-02-27T17:45:31.656332025Z caller=loki.go:386 msg="module failed" module=ring error="failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc000950d20 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc00011aa00 loki false {    false}} 0xc00011aa00 loki [{ringDesc 0xe43cc0} {}]} 0xc0000a1a90 0xc00011aa00 0xc0003ddb20 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0000d70e0 0xc000959c20 0xc000959ce0 0xc000959da0 0xc000959e60 0xc0009660c0 0xc000959f20 0xc000966180 0xc000966000 0xc000936d00 0xc000936d40 0xc000966300 0xc0009663c0 0xc000966540 0xc000966480 0xc00000fea8 0xc0004aa620 0xc00000fe98 0xc00000fea0 0xc000936d80 0xc000936dc0 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-02-27T17:45:31.656460418Z caller=loki.go:386 msg="module failed" module=querier error="failed to start querier, because it depends on module store, which has failed: context canceled"
level=error ts=2024-02-27T17:45:31.656646382Z caller=loki.go:386 msg="module failed" module=query-frontend error="failed to start query-frontend, because it depends on module usage-report, which has failed: context canceled"
level=info ts=2024-02-27T17:45:31.656733282Z caller=module_service.go:114 msg="module stopped" module=query-frontend-tripperware
level=info ts=2024-02-27T17:45:31.658772872Z caller=modules.go:1090 msg="server stopped"
level=info ts=2024-02-27T17:45:31.658801859Z caller=module_service.go:114 msg="module stopped" module=server
level=info ts=2024-02-27T17:45:31.658819899Z caller=loki.go:375 msg="Loki stopped"
failed services
github.com/grafana/loki/pkg/loki.(*Loki).Run
	/src/loki/pkg/loki/loki.go:421
main.main
	/src/loki/cmd/loki/main.go:105
runtime.main
	/usr/local/go/src/runtime/proc.go:255
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1581
level=error ts=2024-02-27T17:45:31.659004411Z caller=log.go:103 msg="error running loki" err="failed services\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:421\nmain.main\n\t/src/loki/cmd/loki/main.go:105\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"
14

Hi, i am also facing the same issue with my loki-write pods, They are not getting to a healthy state. the error i am getting is Readiness probe failed: HTTP probe failed with statuscode: 503.

15

Any help will be appreciated. thanks

16

without some configuration it is very hard to give some indications. Usually 503 means there is no backend available.