Help promtail filtering from prometheus

Grafana Loki
1

I have an time error from the supervisor vs my vm generating a log line similar to this one:

ts=2023-10-20T03:37:19.120Z caller=scrape.go:1685 level=warn component=“scrape manager” scrape_pool=cadvisor target=http://1xx.x.xxx.xxx:29100/metrics msg=“Error on ingesting out-of-order samples”

So I managed to send the prometheus log into a file on my server.

Then I want to configure promtail to grab the information from this file. The log file is huge and I want:

  • to insert into Loki only the lines containing “out-of-order”
  • to manage a metric to count each line.

I’ve spend 3 hours on the subject and I want to kill someone (or drink a coffee), finally finishing with an erroneous yml file:

- job_name: prometheus  
  static_configs:
  - targets:
      - localhost
    labels:
      job: prometheus 
      host: dagobert
      connection: localhost
      owner: atlanticlog
      __path__: /var/log/prometheus/prometheus*.log
  pipeline_stages:
  - match:
    selector: 'msg="Error on ingesting out-of-order samples"'
      stages:
      - regex:
        expression: "out-of-order"
      - metrics:
        out_of_order_ingestion:
          type: Counter
          description: "total count of out of order ingestion, probably time error"
          source: ingestion
          config:
            action: inc

I don’t understand the pipelinestages system.

Please help! :sob: :sob: :sob:

2

Your promtail doesn’t seem to match your intent. If you are trying to send logs to Loki, then you don’t want to use metrics because it turns logs into metrics and expose them at /metrics instead of forwarding them. See metrics | Grafana Loki documentation.

So what you want instead is something like this

- match:
    selector: '{job="prometheus"} !~ ".*out-of-order.*"'
    action: drop
    drop_counter_reason: not_out_of_order
3

Thanks, going to test it.

4

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.