Error: Data source connected, but no labels were received

Grafana Grafana Agent
,
1

Hi everyone!

Recently I installed Loki + Grafana Agent (flow mode) but when I added Loki as a data source I got the error: Data source connected, but no labels were received. Verify that Loki and Promtail are correctly configured.

Does anyone know how I can fix it?

Loki configuration:

loki:
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 30
    timeoutSeconds: 1
  auth_enabled: true
  storage:
    bucketNames:
      chunks: bucket-name
      ruler: bucket-name
      admin: bucket-name
    type: s3
    s3:
      region: us-east-2
      s3ForcePathStyle: false
      insecure: false
serviceAccount:
  create: true
  name: loki-service-account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam:::role/grafama-loki-role
test:
  enabled: false
monitoring:
  dashboards:
    enabled: false
  rules:
    enabled: false
  serviceMonitor:
    enabled: false
  selfMonitoring:
    enabled: false
  lokiCanary:
    enabled: true
write:
  replicas: 3
read:
  replicas: 4
backend:
  replicas: 3
gateway:
  enabled: true
  replicas: 1
  verboseLogging: true
  service:
    port: 80
    type: NodePort
    nodePort: 30001
  basicAuth:
    enabled: true
    username: test
    password: test
  readinessProbe:
    httpGet:
      path: /
      port: http
    initialDelaySeconds: 15
    timeoutSeconds: 1

Agent configuration:

agent:
  mode: 'flow'
  configMap:
    create: false
    name: grafana-agent-config
    key: config.river

  clustering:
    enabled: false
rbac:
  create: true

serviceAccount:
  create: true
  annotations: {}
  name: grafana-agent-service-account

configReloader:
  enabled: true

controller:
  type: 'daemonset'

service:
  enabled: true
  type: ClusterIP
  clusterIP: ''
  annotations: {}

ingress:
  enabled: false

config.river:

discovery.kubernetes "pods" {
  role = "pod"
}
loki.source.kubernetes "test" {
    targets    = discovery.kubernetes.pods.targets
    forward_to = [loki.write.loki.receiver]
}
loki.write "loki" {
  endpoint {
    url = env("LOKI_URL")
    basic_auth {
       username = env("LOKI_USERNAME")
       password = env("LOKI_PASSWORD")
    }
  }
}

Thanks in advance

2

I have similar situation

discovery.kubernetes "pods" {
                role = "pod"
              }
              loki.source.kubernetes "pods" {
                targets    = discovery.kubernetes.pods.targets
                forward_to = [loki.write.multitenant.receiver]
              }
              loki.relabel "app" {
                forward_to = [loki.write.multitenant.receiver]
                rule {
                  action        = "replace"
                  source_labels = ["__meta_kubernetes_pod_controller_name"]
                  regex         = "([0-9a-z-.]+?)(-[0-9a-f]{8,10})?"
                  target_label  = "__tmp_controller_name"
                }
                rule {
                  action        = "replace"
                  source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name","__meta_kubernetes_pod_label_app","__tmp_controller_name","__meta_kubernetes_pod_name"]
                  regex         = "^;*([^;]+)(;.*)?$"
                  target_label  = "app"
                }
              }
              loki.relabel "podName" {
                forward_to = [loki.write.multitenant.receiver]
                rule {
                  action        = "replace"
                  source_labels = ["__meta_kubernetes_pod_name"]
                  target_label  = "pod"
                }
              }              
              loki.source.kubernetes_events "local" {
                forward_to = [loki.write.multitenant.receiver]
                job_name = "events"
              }
              loki.write "multitenant" {
                endpoint {
                  url = "https://loki-multi.domain.com/loki/api/v1/push"
                  tenant_id = "{{metadata.labels.tier}}"
                }
                external_labels = { 
                  cluster = "{{name}}",
                }

but have only this labels

job = loki.source.kubernetes.pods
instance="vault/vault-0:vault-init"