Promtail drop/allow label pipeline stage not working as expected

Grafana Loki
,
1

Hi,

we have below Promtail config to scrape kubernetes level pod logs,

apiVersion: v1
kind: ConfigMap
metadata:
name: promtail-config
namespace: promtail-test
data:
promtail.yaml: |
server:
http_listen_port: 9080
grpc_listen_port: 0

clients:
- url: http://loki-test-gateway.loki-test.svc.cluster.local/loki/api/v1/push

positions:
  filename: /tmp/positions.yaml

target_config:
  sync_period: 10s

scrape_configs:
- job_name: pod-logs

  kubernetes_sd_configs:
    - role: pod

  relabel_configs:
    - source_labels: 
        - __meta_kubernetes_pod_node_name
      target_label: __host__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - action: replace
      replacement: $1
      separator: /
      source_labels:
        - __meta_kubernetes_namespace
        - __meta_kubernetes_pod_name
      target_label: job
    - action: replace
      source_labels:
        - __meta_kubernetes_namespace
      target_label: namespace
    - action: replace
      source_labels:
        - __meta_kubernetes_pod_name
      target_label: pod
    - action: replace
      source_labels:
        - __meta_kubernetes_pod_container_name
      target_label: container
    - replacement: /var/log/pods/*$1/*.log
      separator: /
      source_labels:
        - __meta_kubernetes_pod_uid
        - __meta_kubernetes_pod_container_name
      target_label: __path__
    
    - action: labeldrop
      regex: "app_kubernetes_io_.instance" 

  pipeline_stages:
    - docker: {}
    - labeldrop:
      - app_kubernetes_io_instance
      - app_kubernetes_io_managed_by
    - labelallow:
      - app_kubernetes_io_component
      - pod

however, I see many unnecessary/not required labels are coming to loki server, when i try labeldrop/labelallow , this is not actually not working as expected, we are still able to see labels, this might cause high cardinality,

image
image2827×537 57.5 KB

please suggest how we can remove them,

Many thanks,

2
  1. You should use either labeldrop or labelallow, not both. labelallow allows only the labels you are set, doesn’t make much sense to combine that with labeldrop.
  2. If you already sent logs with other labels prior to config change, your label browser will still show them. you should actually do some query and verify it the labels are actually active or not.

Other than that I don’t see anything obviously wrong with your config.

3

Thanks @tonyswumac ,
I have migrated to grafana alloy, since promtail had EOL soon,
I would want to know is there any way to delete existing loaded labels into loki server anyway from alloy or anything?

Please guide.

Thanks Much,

4

I think the only way is to use the delete API and remove the logs with the labels you don’t want. Personally I would recommend to not bother with it. Eventually the labels will be phased out and disappear from label browser.