Logs are not tailed since regex and replacement are failing

Grafana Alloy
, ,
1

Hello,

I’m using Alloy discovery.kubernetes with role as pod .

My logs are present at the location on hostPath:

/var/log/pods/app-namespace_app-pod-deployment-856f4c4d55-vh4d8_c5f09df9-9676-48ac-9c33-df2a7ef457e4/app-containername/0.log

I’m using source_labels for app-namespace_app, app-pod-deployment-856f4c4d55-vh4d8, c5f09df9-9676-48ac-9c33-df2a7ef457e4, container-name
“__meta_kubernetes_namespace”,
“__meta_kubernetes_pod_name”,
“__meta_kubernetes_pod_uid”,
“__meta_kubernetes_pod_container_name”

My rule looks something like this

rule {
source_labels = [“__meta_kubernetes_namespace”, “__meta_kubernetes_pod_name”, “__meta_kubernetes_pod_uid”, “_meta_kubernetes_pod_container_name"]
action = “replace”
target_label = “path”
separator = “”
regex = "(.*)_(.)_(.)_(.*)"
replacement = "/var/log/pods/$1_$2_$3/$4/*.log”
}

By using _ , my paths are printed something like this(noticied during the debug)
/var/log/pods/c5f09df9-9676-48ac-9c33-df2a7ef457e4/container-name/.log

Just to see if the source_labels have any values at all, i changed my replacement to use /, as below

rule {
source_labels = [“__meta_kubernetes_namespace”, “__meta_kubernetes_pod_name”, “__meta_kubernetes_pod_uid”, “__meta_kubernetes_pod_container_name”]
action = “replace”
target_label = “path”
separator = “"
regex = "(.*)_(.)_(.)_(.)"
replacement = "/var/log/pods/$1/$2/$3/$4/.log”
}
I get below value
/var/log/pods/app-namespace_app/app-pod-deployment-856f4c4d55-vh4d8/c5f09df9-9676-48ac-9c33-df2a7ef457e4/container-name/.log

The wildcard is not getting resolved in both the cases.

I’m using alloy:v1.10.2

Could you please help ? I’m stuck here from days!!

My Entire Config

    discovery.kubernetes "pods" {
      role = "pod"
    }

    discovery.relabel "pod_logs" {
      targets = discovery.kubernetes.pods.targets

      rule {
        source_labels = ["__meta_kubernetes_namespace"]
        action = "replace"
        target_label = "namespace"
      }

      rule {
        source_labels = ["__meta_kubernetes_pod_name"]
        action = "replace"
        target_label = "pod"
      }

      rule {
        source_labels = ["__meta_kubernetes_pod_container_name"]
        action = "replace"
        target_label = "container"
      }

      rule {
        source_labels = ["__meta_kubernetes_pod_label_app_kubernetes_io_name"]
        action = "replace"
        target_label = "app"
      }

      rule {
        source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
        action = "replace"
        target_label = "job"
        separator = "/"
        replacement = "$1"
      }

      rule {
        source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_name", "__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
        action = "replace"
        target_label = "__path__"
        separator = "_"
        regex = "(.*)_(.*)_(.*)_(.*)"
        replacement = "/var/log/pods/$1_$2_$3/$4/0.log"
      }

      rule {
        source_labels = ["__meta_kubernetes_pod_container_id"]
        action = "replace"
        target_label = "container_runtime"
        regex = "^(\\S+):\\/\\/.+$"
        replacement = "$1"
      }
    }

    loki.write "loki_output" {
      endpoint {
        url = "https://vm.de.xx.com/logs/loki/api/v1/push"
        tls_config {
          insecure_skip_verify = true
        }
      }

      external_labels = {
        source = "alloy_k8s",
      }
    }

    loki.process "pod_logs" {
      stage.static_labels {
        values = {
          cluster = env("HOSTNAME"),
        }
      }

      forward_to = [loki.write.loki_output.receiver]
    }

    loki.source.file "pod_logs" {
      targets    = discovery.relabel.pod_logs.output
      forward_to = [loki.process.pod_logs.receiver]
    }
    discovery.relabel "container_logs" {
      targets = discovery.kubernetes.pods.targets

      rule {
        source_labels = ["__meta_kubernetes_namespace"]
        action = "replace"
        target_label = "namespace"
      }

      rule {
        source_labels = ["__meta_kubernetes_pod_name"]
        action = "replace"
        target_label = "pod"
      }

      rule {
        source_labels = ["__meta_kubernetes_pod_container_name"]
        action = "replace"
        target_label = "container"
      }

      rule {
        source_labels = ["__meta_kubernetes_pod_name", "__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name", "__meta_kubernetes_pod_uid"]
        separator = "_"
        regex = "(.*)_(.*)_(.*)_(.*)"
        target_label = "__path__"
        replacement = "/var/log/containers/$1_$2_$3-$4.log"
        action = "replace"
      }
    }

    loki.source.file "container_logs" {
      targets    = discovery.relabel.container_logs.output
      forward_to = [loki.process.pod_logs.receiver]
    }
2

If you are using discovery.kubernetes, it’s better to just use loki.source.kubernetes (see Collect Kubernetes logs and forward them to Loki | Grafana Alloy documentation).