Missing k8sattributes in OTEL data with Grafana Alloy

Grafana Loki
1

We currently migrate to Grafana Alloy but unfortunately do not get k8s metadata for our infrastructure signals, especially Logs in Loki. I can’t find proper documentation for OTLP integration for Loki unfortunately. We use Alloy’s “loki.source.kubernetes” and “otelcol.processor.k8sattributes” but dont get any of the k8sattributes in our OTLP data in Loki - only the labels from “discovery.relabel” are available.

This is our current alloy config (the part for logs):

logging {
  level  = "info"
  format = "logfmt"
}

discovery.kubernetes "pods" {
  role = "pod"

  namespaces {
    names = ["ingress-nginx"]
  }
}

discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pods.targets

  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    action = "replace"
    target_label = "k8s_namespace_name"
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_node_name"]
    action = "replace"
    target_label = "k8s_node_name"
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    action = "replace"
    target_label = "k8s_pod_name"
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
    action = "replace"
    target_label = "service_name"
  }
}

loki.source.kubernetes "poc" {
  targets    = discovery.relabel.pod_logs.output
  forward_to = [otelcol.receiver.loki.default.receiver]
}

otelcol.receiver.loki "default" {
  output {
    logs = [otelcol.processor.k8sattributes.default.input]
  }
}

otelcol.receiver.otlp "default" {
  http {}
  grpc {}

  output {
    logs    = [otelcol.processor.k8sattributes.default.input]
  }
}

otelcol.processor.k8sattributes "default" {
  extract {
    metadata = [
      "k8s.namespace.name",
      "k8s.deployment.name",
      "k8s.statefulset.name",
      "k8s.daemonset.name",
      "k8s.cronjob.name",
      "k8s.job.name",
      "k8s.node.name",
      "k8s.pod.name",
      "k8s.pod.uid",
      "k8s.pod.start_time",
    ]

    label {
      from = "pod"
      key_regex = "app.kubernetes.io/(.*)"
      tag_name  = "$1"
    }
  }

  output {
    logs    = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    logs    = [otelcol.exporter.loki.default.input]
  }
}

otelcol.exporter.loki "default" {
  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Structured Metadata is enabled in Loki. We also tried using the OTLPHTTP exporter, but it didnt solve the problem.

Thanks in advance!

2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.