Can't add dynamic labels to my static_configs in promtail?

So I am trying to add the Kubernetes node name to my static_config scrape using relabel_configs, but these don’t get added for some reason. I think I’m either doing something wrong or missing the point of relabelling. Here’s a manifest that can reproduce this -

extraVolumes:
  - name: syslog-vol
    hostPath:
      path: /vol
extraVolumeMounts:
  - name: syslog-vol
    mountPath: /vol
    readOnly: true
config:
  logLevel: debug
  lokiAddress: http://loki-stack.logging:3100/loki/api/v1/push
  snippets:
    extraScrapeConfigs: |
      - job_name: syslog
        static_configs:
        - targets:
            - localhost
          labels:
            job: syslog
            __path__: /vol/system.log
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - action: replace
            source_labels: ['__meta_kubernetes_node_name']
            target_label: node_name

Any help would be appreciated! Thanks!

Scrape configs looks correct!. Not sure why you still couldn’t get node_name label.

My only suspect is may yaml in-correct indentation may be? (I think I see some extra spaces in action and source_labels).

One option to test is use --stdin and --dry-run option in promtail. To see if you are getting proper labels (even before sending it to loki).

Can you try that?

The indentation is only an issue on this forum. I have verified correct yaml indentation locally. I’m unable to run dry-run but I’ve ssh’d onto a pod and here are the contents of /etc/promtail/promtail.yaml -

  - job_name: syslog
    static_configs:
    - targets:
        - localhost
      labels:
        job: syslog
        __path__: /vol/system.log
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - action: replace
        source_labels: ['__meta_kubernetes_node_name']
        target_label: node_name

I did a little more digging and looking at the service discovery you can see that the promtail job exists, as do the kubernetes_sd_config labels. However, for some reason the $1 is being evaluated to null. If I change replacement: $1 to replacement: foo-$1, I see the syslog job with the node_name: foo- label. Any idea whats going on?