Promtail - Kubernetes_sd_configs coupled with labels

[mikemclean1]

Hi,

I’m trying desperately to get kubernetes pod logs to extract labels from the detected fields. I’ve tried a staggering amount of things, read all the docs, yet … I’m stuck.

Here’s the setup:

• Everything running in a local k8s cluster
• Grafana is fine and I can see my logs properly
• Promtail is running and is getting logs properly to Loki for all containers running in the cluster

What I’m trying to achieve:
Given the following log (for example): { "breed" : "german shorthaired pointer", "name" : "Jax", "good_boy" : true }

I want to get the good_boy detected field into the labels. At present, no matter what I do, the good_boy field remains as a “Detected Field” in Grafana and is never available as a label.

Here’s what my configuration looks like:

- pipeline_stages:
        - docker:
        - labels: 
             good_boy:
        job_name: kubernetes-pods-name
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels:
          - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: replace
          replacement: $1
          separator: /
          source_labels:
          - __meta_kubernetes_namespace
          - __service__
          target_label: k8s_app    
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
          target_label: __path__
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
          target_label: __path__

Thanks for any insights!

[b0b]

Hi @mikemclean1 ,

I’m just about to start experimenting with log parsing in Promtail so this is just a guess from my side but I think you need a json pipeline before the labels pipeline like in this example.

Hope that helps.

[mikemclean1]

Hi @b0b , thanks for the reply! Using a JSON pipeline is one of the things I tried, but unfortunately, didn’t get any results .