Filter logs coming from K8s cluster, and send it to Grafana

[devcluster]

I want to filter logs coming from my Kubernetes cluster, so that I can keep the logs usage within the limits.
What shall I do, if I only want to receive the ‘error’ logs and ‘warning’ logs.
I tried the following method to send only error logs, could someone confirm if this is the correct way or not:

---

pod_logs:

enabled: true

namespaces: [ns-1, ns-2]

loggingFormat: docker

extraRelabelingRules: 

extraStageBlocks: |-
  stage.drop {
    expression  = ".*debug.*"
  }

Settings for scraping Kubernetes cluster events

cluster_events:
# – Scrape Kubernetes cluster events
enabled: true

namespaces:
  - ns-1
  - ns-2

extraConfig: |-
loki.relabel “keep_error_only” {
forward_to = [loki.write.grafana_cloud_loki.receiver]

  rule {
    action        = "keep"
    source_labels = ["log.level"]
    regex         = "error"
  }
}

If I don’t do any filtering, the 100gb limit crosses.
I want to keep my logs with the 100 gb logs limit.
Any more methods, if anyone could suggest, to only focus on the error or warning logs.
I had read an article about dropping the unused metrics, is there any similar approach to be followed for logs.

[tonyswumac]

I can think of two ways to do this:

1. You use the drop action and write a regex to match logs that do NOT contain “error” or “warning”.

2. You match for logs that contain “error” or “warning”, then drop the rest.

In general I find it painful writing regex to match “not containing”, so I’d probably recommend #2 personally. Mock logic can look like this (mock logic, not tested):

- match:
    selector: '{SELECTOR} |~ "((?i)error|(?i)warning)"'
    stages:
      static_label:
        keep_log: "yes"

- match:
    selector: '{keep_lob!="yes"}'
    action: drop