Promtail helm chart extraScrapeConfigs not working as intended

Grafana Loki
, ,
1

Hi,

we’re using Loki and Promtail on Azure on AKS. We install/update and manage them through helm, so far we didn’t really do changes in the configuration files but now we would like to drop some of the messages from our ingress nginx controller (messages coming to two specific endpoints from on-premise services).

I have tried to modify the values.yaml but unfortunately although the config is picked up and appears in the promtail.yaml file on promtail’s pods it doesn’t actually does what we expected.

promtail chart version: 6.7.4
loki chart (simple scalable) version: 1.8.11

Could you help me figure out what did we miss in the config?
The goal is: drop all lines that are coming from app: ingress-nginx and contains the string “/getdate”

config:
  clients:
    - url: http://loki-gateway/loki/api/v1/push
  
  snippets:
    extraScrapeConfigs: |
      - job_name: nginx-getdate-lines
        pipeline_stages:
        - match:
            selector: '{job="ingress/ingress-nginx"}'
            stages:
            - drop:
                expression: .*\/getdate.*

Thank you!

2

Try this:

- job_name: nginx-getdate-lines
  pipeline_stages:
    - match:
        selector: '{job="ingress/ingress-nginx"}'
        action: drop
        drop_counter_reason: ingress-nginx
3

Hi,

I think this configuration would drop all log lines, wouldn’t it? I only need to drop the ones where “/getdate” is in it.

4

Ah yes, I see what you mean. In that case what you have should actually work. Do you have some sample logs and your entire promtail config?

You can also try matching it and turn it into label. This is however a bit of a round-about way:

pipeline_stages:
  - regex:
      expression: .*(?P<getdate>\/getdate).*
  - labels:
      getdate:
  - match: # Drop logs
      selector: '{getdate="getdate"}'
      action: drop
      drop_counter_reason: non_essential_log
  - match: # Don't drop logs
      selector: '{container_name!~"getdate"}'
      stages:
        ...
5

Sure, this is the promtail config that is currently on the pods (I removed the default kubernetes part as it’s long, but let me know if I should put it back in):

root@promtail-txxtq:/# cat /etc/promtail/promtail.yaml
server:
  log_level: info
  http_listen_port: 3101


clients:
  - url: http://loki-gateway/loki/api/v1/push

positions:
  filename: /run/promtail/positions.yaml

scrape_configs:
  # See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
  - job_name: kubernetes-pods
        ... the default kubernetes related settings here ...

  - job_name: nginx-getdate-lines
    pipeline_stages:
    - match:
        selector: '{job="ingress/ingress-nginx"}'
        stages:
        - drop:
            expression: .*\/getdate.*

limits_config:

Also here are some sample logs:

2023-02-17 20:01:02	80.78.121.54 - - [17/Feb/2023:19:01:02 +0000] "POST /json/getdate HTTP/2.0" 200 346 "-" "unirest-java/3.1.00" 283 0.001 [backend-prod-daterecognizer-80] [] 10.0.13.207:80 358 0.000 200 4c4077429993a20c9eb59c0c7e78f61e
2023-02-17 20:01:02	80.78.121.54 - - [17/Feb/2023:19:01:02 +0000] "POST /json/getdate HTTP/2.0" 200 346 "-" "unirest-java/3.1.00" 282 0.002 [backend-prod-daterecognizer-80] [] 10.0.13.207:80 358 0.000 200 496347a2665aba15ae1bf78a4439c7ae
2023-02-17 20:01:02	80.78.121.54 - - [17/Feb/2023:19:01:02 +0000] "POST /json/getdate HTTP/2.0" 200 346 "-" "unirest-java/3.1.00" 281 0.001 [backend-prod-daterecognizer-80] [] 10.0.13.207:80 358 0.000 200 b48a917266152549e2cfa2495035db9e
6

I tested your configuration, and it seems to work. Here is what I have:

  1. Test log (/tmp/test.log). Added 4th line to not match the regex for testing purpose:
2023-02-17 20:01:02 80.78.121.54 - - [17/Feb/2023:19:01:02 +0000] "POST /json/getdate HTTP/2.0" 200 346 "-" "unirest-java/3.1.00" 283 0.001 [backend-prod-daterecognizer-80] [] 10.0.13.207:80 358 0.000 200 4c4077429993a20c9eb59c0c7e78f61e
2023-02-17 20:01:02 80.78.121.54 - - [17/Feb/2023:19:01:02 +0000] "POST /json/getdate HTTP/2.0" 200 346 "-" "unirest-java/3.1.00" 282 0.002 [backend-prod-daterecognizer-80] [] 10.0.13.207:80 358 0.000 200 496347a2665aba15ae1bf78a4439c7ae
2023-02-17 20:01:02 80.78.121.54 - - [17/Feb/2023:19:01:02 +0000] "POST /json/getdate HTTP/2.0" 200 346 "-" "unirest-java/3.1.00" 281 0.001 [backend-prod-daterecognizer-80] [] 10.0.13.207:80 358 0.000 200 b48a917266152549e2cfa2495035db9e
2023-02-17 20:01:02 80.78.121.54 - - [17/Feb/2023:19:01:02 +0000] "POST /json/getnotadate HTTP/2.0" 200 346 "-" "unirest-java/3.1.00" 281 0.001 [backend-prod-daterecognizer-80] [] 10.0.13.207:80 358 0.000 200 b48a917266152549e2cfa2495035db9e
  1. Test config (/tmp/config.yml):
server:
  disable: true

positions:
  filename: /tmp/positions.yml

clients:
- url: http://127.0.0.1:3100/loki/api/v1/push

scrape_configs:
- job_name: test
  static_configs:
  - targets:
      - localhost
    labels:
      job: test
      __path__: /tmp/test.log

  pipeline_stages:
  - drop:
      expression: .*\/getmpdate.*
  1. Actually running the test:
cat /tmp/test.log | promtail-linux-amd64-2.6.1 -config.file /tmp/config.yml --stdin --dry-run
  1. Result printout (notice only one line of log got sent, which is the 4th that doesn’t match regex):
Clients configured:
----------------------
url: http://127.0.0.1:3100/loki/api/v1/push
batchwait: 1s
batchsize: 1048576
follow_redirects: false
backoff_config:
  min_period: 500ms
  max_period: 5m0s
  max_retries: 10
timeout: 10s
tenant_id: ""
stream_lag_labels: ""

level=info ts=2023-02-21T20:56:39.694458075Z caller=main.go:121 msg="Starting Promtail" version="(version=2.6.1, branch=HEAD, revision=6bd05c9a4)"
2023-02-21T20:56:39.694702385+0000	{__path__="/tmp/test.log", job="test"}	2023-02-17 20:01:02 80.78.121.54 - - [17/Feb/2023:19:01:02 +0000] "POST /json/getnotadate HTTP/2.0" 200 346 "-" "unirest-java/3.1.00" 281 0.001 [backend-prod-daterecognizer-80] [] 10.0.13.207:80 358 0.000 200 b48a917266152549e2cfa2495035db9e
level=info ts=2023-02-21T20:56:39.694923017Z caller=server.go:275 msg="received shutdown signal" sig=terminated

I went back and looked at your config again, where is this part from?

- match:
        selector: '{job="ingress/ingress-nginx"}'

Are you sure the job label actually exist there? Might want to double check on that and maybe use namespace label from kubernetes.

7

Hi,

thanks for checking, based on this I think the selector might be wrong that we use as the regex seems to work as expected in your example.

This selector is used when we query for the log lines on grafana with loki. Not sure if this is usable as is in promtail, maybe I have to separate the namespace or do some label conversion first?

I’ve tried this, but it’s still not working:

      - job_name: nginx-getdate-lines
        kubernetes_sd_configs:
          - role: pod
        pipeline_stages:
        - match:
            selector: '{namespace="ingress", app="ingress-nginx"}'
            stages:
            - drop:
                expression: .*\/getdate.*
8

When you look at your logs in Grafana, what labels do they have?

9

This is what I see under the log lines:

image
image1199×294 27 KB

Not sure about the inner workings of promtail, so I don’t know if all these are available there too.

10

The labels look correct. I want to backtrack to your configuration above:

  - job_name: kubernetes-pods
        ... the default kubernetes related settings here ...

  - job_name: nginx-getdate-lines
       ...

What’s happening under the job_name: kubernetes-pods block? Is it possible that the ingress logs are picked up there? If so, then your nginx-getdate-lines would’ve never been invoked and would explain why the drop isn’t being processed.

11

This is the whole file without the reduction (I removed that part as it gets there from the standard helm config I assume and we haven’t changed it’s values). By the look of it it’s mostly about the renaming of the labels:

root@promtail-vfv2m:/# cat /etc/promtail/promtail.yaml
server:
  log_level: info
  http_listen_port: 3101


clients:
  - url: http://loki-gateway/loki/api/v1/push

positions:
  filename: /run/promtail/positions.yaml

scrape_configs:
  # See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
  - job_name: kubernetes-pods
    pipeline_stages:
      - cri: {}
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels:
          - __meta_kubernetes_pod_controller_name
        regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
        action: replace
        target_label: __tmp_controller_name
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_name
          - __meta_kubernetes_pod_label_app
          - __tmp_controller_name
          - __meta_kubernetes_pod_name
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: app
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_instance
          - __meta_kubernetes_pod_label_release
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: instance
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_component
          - __meta_kubernetes_pod_label_component
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: component
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: node_name
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - namespace
        - app
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - action: replace
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_uid
        - __meta_kubernetes_pod_container_name
        target_label: __path__
      - action: replace
        regex: true/(.*)
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
        - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
        - __meta_kubernetes_pod_container_name
        target_label: __path__

  - job_name: nginx-getdate-lines
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
    - match:
        selector: '{namespace="ingress", app="ingress-nginx"}'
        stages:
        - drop:
            expression: .*\/getdate.*


limits_config:
12

I’ve not used promtail with kubernetes much, so I could be wrong, but what happens if you try to move your pipeline stage from job_name: nginx-getdate-lines into the first pipeline stages?

13

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.