Confused where to put the match statement

Grafana Loki
1

Hi!

I just installed promtail in one of our K8S clusters using the directions on Grafana Cloud.

curl -fsS https://raw.githubusercontent.com/grafana/loki/master/tools/promtail.sh | sh -s 1122334 <Your Grafana.com API Key> logs-prod-us-central1.grafana.net default | kubectl apply --namespace=default -f  -

This worked, and I have cluster logs flowing into Loki. But what I really want to do is restrict what logs are sent to Loki. Mostly because we have a lot of pods we don’t need logging on, and I’m trying to limit the logs going into Grafana Cloud and stay under the 50GB limit until we can get our own Loki environment locally.

I’d really like to have an annotation on the pods that promtail uses to determine if the logs should be scrapped or not. Like with prometheus having an annotation for prometheus.io/scrape=true. Next best solution would be to restrict it based on the namespaces we want to scrape pods in.

So I believe I need to do a match with a selector and an action of drop.

       - match:
         selector: '{promtail="true"}
         action: drop

However the promtail.yml configmap created by the promtail.sh script doesn’t seem to match the documentation on the promtail site about how the file should be.

According to the documentation, I should see something like:

scrape_configs:
- job_name: kubernetes-pods-name
  kubernetes_sd_configs: ....
  pipeline_stages:

  # This stage is only going to run if the scraped target has a label
  # of "name" with value "promtail".
  - match:
      selector: '{name="promtail"}'
      stages:
      - docker{}

But the configmap created is completely different, with the parsing stage of docker before the kubernetes_sd_configs - and I put the match at the end?

  8 scrape_configs:
  9     - pipeline_stages:
 10       - docker:
 11       job_name: kubernetes-pods-name
 12       kubernetes_sd_configs:
 13       - role: pod
 14       relabel_configs:
 15       - source_labels:
 16         - __meta_kubernetes_pod_label_name
 17         target_label: __service__
 18       - source_labels:
 19         - __meta_kubernetes_pod_node_name
 20         target_label: __host__
 21       - action: drop
 22         regex: ^$
 23         source_labels:
 24         - __service__
 25       - action: replace
 26         replacement: $1
 27         separator: /
 28         source_labels:
 29         - __meta_kubernetes_namespace
 30         - __service__
 31         target_label: job
 32       - action: replace
 33         source_labels:
 34         - __meta_kubernetes_namespace
 35         target_label: namespace
 36       - action: replace
 37         source_labels:
 38         - __meta_kubernetes_pod_name
 39         target_label: instance
 40       - action: replace
 41         source_labels:
 42         - __meta_kubernetes_pod_container_name
 43         target_label: container_name
 44       - replacement: /var/log/pods/*$1/*.log
 45         separator: /
 46         source_labels:
 47         - __meta_kubernetes_pod_uid
 48         - __meta_kubernetes_pod_container_name
 49         target_label: __path__
 50       - match:
 51         selector: '{promtail="true"}
 52         action: drop

Can anyone provide any insight into how I can accomplish this?

Thanks!

2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.