Filtering Deployments in Promtail and K8S

Grafana Loki
,
1

My problem: I am only using promtail to send logs from my Kubernetes cluster to my Grafana cloud instance. It works, my logs arrive perfectly. My problem is that I don’t want to send logs from the whole cluster, I only want to send those that are conveniently tagged. I also don’t want to tag all objects in the cluster prematurely to indicate they should not be logged, because I don’t find it scalable. My idea is to take, for example, a Deployment, tag it in a convenient way (“logs-enabled,” for example), and then receive only its logs. Below is the configuration I currently have.

I installed Promtail simply using helm: helm repo add grafana https://grafana.github.io/helm-charts, and then helm repo update

So I created this yml file, which works perfectly for sending my logs to my Grafana Cloud:

config:
  clients:
    - url: my-url
      basic_auth:
        username: my-user
        password: my-pass

  positions:
    filename: /run/promtail/positions.yaml

  pipeline_stages:
    - docker: {}

  scrape_configs:
    - job_name: kubernetes-pods
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        # ---- FILTERING ----
        # Try to only keep pods that have logging=logs-enabled label
        # This should drop ALL pods that don't have this exact label
        - source_labels: [__meta_kubernetes_pod_label_logging]
          regex: ^logs-enabled$
          action: keep

        # ---- STANDARD LABELS ----
        - source_labels: [__meta_kubernetes_namespace]
          target_label: namespace
        - source_labels: [__meta_kubernetes_pod_name]
          target_label: pod
        - source_labels: [__meta_kubernetes_container_name]
          target_label: container

        # ---- MAP ALL POD LABELS ----
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)

And finally I just execute helm upgrade --install promtail grafana/promtail -f promtail-values.yaml

As you can see, I’m trying to filter by the label “logs-enabled” inside “scrape_configs,” but this isn’t working. Continue sending logs of absolutely everything.

I also tried filtering by namespace, leaving only the “default” namespace, and got no results; absolutely everything continues to be logged:

        - source_labels: [__meta_kubernetes_namespace]

          action: keep

          regex: ^default$

Thanks in advance!

2

Deployment labels/annotations are not propagated as pod labels/annotations usually. So it doesn’t make sense to filter on Pod labels, when you labeled Deployment.

I would ditch promtail and use more powerfull tools, e. g.

See doc for pod logs:

You can exclude namespace out of the box. In theory you can also filter on deployment labels, that will need more work there. You will need to enrich pod log - add those deployement labels to pod logs and then filter on them. It will be complex, but it is possible. Example for log enrichment (of course not copy&paste for your case):

3

Wow Jangara, you really are the Grafana Champion! Thanks!

Well, regarding your suggestions, I’ll keep them in mind for the future, but today it’s overkill for me. I just need to send some logs; my cluster is being tracked by another tool.

Regarding my mention of Deployments labeling, I apologize for the confusion; I also tried Pods labeling and it didn’t work either.

With a little trial and error, I might be able to find a solution for my use case. I’ll leave it here in case anyone needs it. Basically, I use the app tag, and all the elements mentioned will be logged, while the rest will be discarded:

  snippets:

    pipelineStages:

    - match:

        selector: '{app!~"app1|app2|app3|postgres|mongo|etc"}'

        action: drop

It’s interesting because it’s written in several similar pipelines that haven’t worked and syntactically looked fine.

If you have any recommendations for me on how I’m doing this (adjusting it, improving it), they would be welcome. Otherwise, you can close the post.

Thank you!