Can't add dynamic labels to my static_configs in promtail?

Grafana Loki
1

So I am trying to add the Kubernetes node name to my static_config scrape using relabel_configs, but these don’t get added for some reason. I think I’m either doing something wrong or missing the point of relabelling. Here’s a manifest that can reproduce this -

extraVolumes:
  - name: syslog-vol
    hostPath:
      path: /vol
extraVolumeMounts:
  - name: syslog-vol
    mountPath: /vol
    readOnly: true
config:
  logLevel: debug
  lokiAddress: http://loki-stack.logging:3100/loki/api/v1/push
  snippets:
    extraScrapeConfigs: |
      - job_name: syslog
        static_configs:
        - targets:
            - localhost
          labels:
            job: syslog
            __path__: /vol/system.log
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - action: replace
            source_labels: ['__meta_kubernetes_node_name']
            target_label: node_name

Any help would be appreciated! Thanks!

2

Scrape configs looks correct!. Not sure why you still couldn’t get node_name label.

My only suspect is may yaml in-correct indentation may be? (I think I see some extra spaces in action and source_labels).

One option to test is use --stdin and --dry-run option in promtail. To see if you are getting proper labels (even before sending it to loki).

Can you try that?

3

The indentation is only an issue on this forum. I have verified correct yaml indentation locally. I’m unable to run dry-run but I’ve ssh’d onto a pod and here are the contents of /etc/promtail/promtail.yaml -

  - job_name: syslog
    static_configs:
    - targets:
        - localhost
      labels:
        job: syslog
        __path__: /vol/system.log
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - action: replace
        source_labels: ['__meta_kubernetes_node_name']
        target_label: node_name
4

I did a little more digging and looking at the service discovery you can see that the promtail job exists, as do the kubernetes_sd_config labels. However, for some reason the $1 is being evaluated to null. If I change replacement: $1 to replacement: foo-$1, I see the syslog job with the node_name: foo- label. Any idea whats going on?

service-discovery
service-discovery1920×498 61.6 KB

5

I am having the same issue, did you find any solution for this?
With the same config, I cant seem to find the kubernetes labels in Discovered labels section