Howto drop log levels with structured logs (JSON) relabel_configs?

Hi There,

I’m trying to only log log levels of type “Error|Critical|Warning|Info|Trace” all other “Debug” i’m not interested in to be logged. I’m using Serilog to create structured logs (JSON) which look like this:

{"@t":"2022-05-03T06:57:13.2534723Z","@m":"Batch acquisition of 0 triggers","@i":"5f643198","@l":"Debug","0":0,"SourceContext":"Quartz.Core.QuartzSchedulerThread"}

To log my kubernetes environment I received an config of Grafana Cloud which looks like follow:

  kind: ConfigMap
  metadata:
    name: grafana-agent-logs
    namespace: monitoring
  apiVersion: v1
  data:
    agent.yaml: |
      metrics:
        wal_directory: /tmp/grafana-agent-wal
        global:
          scrape_interval: 60s
          external_labels:
            cluster: cloud
        configs:
        - name: integrations
          remote_write:
          - url: XXX
            basic_auth:
              username: XXX
              password: XXX
      integrations:
        prometheus_remote_write:
        - url: XXX
          basic_auth:
            username: XXX
            password: XXX


      logs:
        configs:
        - name: integrations
          clients:
          - url: XXX
            basic_auth:
              username: XXX
              password: XXX
            external_labels:
              cluster: cloud
          positions:
            filename: /tmp/positions.yaml
          target_config:
            sync_period: 10s
          scrape_configs:
          - job_name: integrations/kubernetes/pod-logs
            kubernetes_sd_configs:
              - role: pod
            pipeline_stages:
              - cri: {}
            relabel_configs:
              - source_labels:
                  - __meta_kubernetes_pod_node_name
                target_label: __host__
              - action: labelmap
                regex: __meta_kubernetes_pod_label_(.+)
              - action: replace
                replacement: $1
                separator: /
                source_labels:
                  - __meta_kubernetes_namespace
                  - __meta_kubernetes_pod_name
                target_label: job
              - action: replace
                source_labels:
                  - __meta_kubernetes_namespace
                target_label: namespace
              - action: replace
                source_labels:
                  - __meta_kubernetes_pod_name
                target_label: pod
              - action: replace
                source_labels:
                  - __meta_kubernetes_pod_container_name
                target_label: container
              - replacement: /var/log/pods/*$1/*.log
                separator: /
                source_labels:
                  - __meta_kubernetes_pod_uid
                  - __meta_kubernetes_pod_container_name
                target_label: __path__

My output in Grafana Cloud looks like this:

And when I inspect the data using Grafana Cloud en download the logs this looks like this:

ommon labels: {"stream":"stdout","cluster":"cloud"}
Line limit: 1000
Total bytes processed: "837  kB"


2022-05-03T09:02:20+02:00	{"@t":"2022-05-03T07:02:20.2007423Z","@m":"Batch acquisition of 0 triggers","@i":"5f643198","@l":"Debug","0":0,"SourceContext":"Quartz.Core.QuartzSchedulerThread"}
2022-05-03T09:01:51+02:00	{"@t":"2022-05-03T07:01:51.4765404Z","@m":"Batch acquisition of 0 triggers","@i":"5f643198","@l":"Debug","0":0,"SourceContext":"Quartz.Core.QuartzSchedulerThread"}
2022-05-03T09:01:51+02:00	{"@t":"2022-05-03T07:01:51.4765404Z","@m":"Batch acquisition of 0 triggers","@i":"5f643198","@l":"Debug","0":0,"SourceContext":"Quartz.Core.QuartzSchedulerThread"}
2022-05-03T09:01:24+02:00	{"@t":"2022-05-03T07:01:24.0343157Z","@m":"Batch acquisition of 0 triggers","@i":"5f643198","@l":"Debug","0":0,"SourceContext":"Quartz.Core.QuartzSchedulerThread"}
2022-05-03T09:01:24+02:00	{"@t":"2022-05-03T07:01:24.0343157Z","@m":"Batch acquisition of 0 triggers","@i":"5f643198","@l":"Debug","0":0,"SourceContext":"Quartz.Core.QuartzSchedulerThread"}
2022-05-03T09:00:54+02:00	{"@t":"2022-05-03T07:00:54.0420947Z","@m":"Batch acquisition of 0 triggers","@i":"5f643198","@l":"Debug","0":0,"SourceContext":"Quartz.Core.QuartzSchedulerThread"}
2022-05-03T09:00:54+02:00	{"@t":"2022-05-03T07:00:54.0420947Z","@m":"Batch acquisition of 0 triggers","@i":"5f643198","@l":"Debug","0":0,"SourceContext":"Quartz.Core.QuartzSchedulerThread"}
2022-05-03T09:00:29+02:00	{"@t":"2022-05-03T07:00:29.0588847Z","@m":"Batch acquisition of 0 triggers","@i":"5f643198","@l":"Debug","0":0,"SourceContext":"Quartz.Core.QuartzSchedulerThread"}
2022-05-03T09:00:29+02:00	{"@t":"2022-05-03T07:00:29.0588847Z","@m":"Batch acquisition of 0 triggers","@i":"5f643198","@l":"Debug","0":0,"SourceContext":"Quartz.Core.QuartzSchedulerThread"}
2022-05-03T09:00:00+02:00	{"@t":"2022-05-03T07:00:00.3187026Z","@m":"Batch acquisition of 0 triggers","@i":"5f643198","@l":"Debug","0":0,"SourceContext":"Quartz.Core.QuartzSchedulerThread"}
2022-05-03T09:00:00+02:00	{"@t":"2022-05-03T07:00:00.3187026Z","@m":"Batch acquisition of 0 triggers","@i":"5f643198","@l":"Debug","0":0,"SourceContext":"Quartz.Core.QuartzSchedulerThread"}

Below some extra display which could help or not ?

I hope someone can help me out, I’ve tried the documentation … but it’s just not working for when I thought I understood the problem.

Kind regards,

Nick

Thanks to some awesome help by @Maarten De Wispelaere on the Slack community channel of Grafana who pushed me in the right direction.

I only had to add the following configuration:

pipeline_stages:
  - cri: {}
  - replace:
      expression: \"(@l)\"
      replace: level
  - json:
      expressions:
        level:
  - drop:
      source: level
      expression: "(Debug|T