Help promtail filtering from prometheus

I have an time error from the supervisor vs my vm generating a log line similar to this one:

ts=2023-10-20T03:37:19.120Z caller=scrape.go:1685 level=warn component=“scrape manager” scrape_pool=cadvisor target= msg=“Error on ingesting out-of-order samples”

So I managed to send the prometheus log into a file on my server.

Then I want to configure promtail to grab the information from this file. The log file is huge and I want:

  • to insert into Loki only the lines containing “out-of-order”
  • to manage a metric to count each line.

I’ve spend 3 hours on the subject and I want to kill someone (or drink a coffee), finally finishing with an erroneous yml file:

- job_name: prometheus  
  - targets:
      - localhost
      job: prometheus 
      host: dagobert
      connection: localhost
      owner: atlanticlog
      __path__: /var/log/prometheus/prometheus*.log
  - match:
    selector: 'msg="Error on ingesting out-of-order samples"'
      - regex:
        expression: "out-of-order"
      - metrics:
          type: Counter
          description: "total count of out of order ingestion, probably time error"
          source: ingestion
            action: inc

I don’t understand the pipelinestages system.

Please help! :sob: :sob: :sob:

Your promtail doesn’t seem to match your intent. If you are trying to send logs to Loki, then you don’t want to use metrics because it turns logs into metrics and expose them at /metrics instead of forwarding them. See metrics | Grafana Loki documentation.

So what you want instead is something like this

- match:
    selector: '{job="prometheus"} !~ ".*out-of-order.*"'
    action: drop
    drop_counter_reason: not_out_of_order

Thanks, going to test it.