Alert for every error in logs

Hello. I want to receive an alert on every error in logs. I have Logstash now and it sends me emails on every error in some application. But I can’t sort out how to do it in Loki. I’ve tried several rules like:

- alert: ErrorsInAppLogs
  expr: |
    sum(count_over_time({label="app",level="error"} | regexp `(?P<log>.+)`[5m])) by (log, label, container, hostname, job, logger, pod) > 0
  for: 15s
  labels:
    severity: critical

and

- alert: ErrorsInAppLogs
  expr: |
    count_over_time({label="app",level="error"}[5m]) > 0
  for: 15s
  labels:
    severity: critical

In both cases, I’ve received either zero or double alerts for errors in logs. But Logstash still sent emails on every error without doubling or skipping.
Alertmanager config:

global:
  smtp_from: some@email.com
  smtp_smarthost: smtp.email.com:587
  smtp_hello: smtp.email.com
  smtp_auth_username: some@email.com
receivers:
- name: admins
  email_configs:
  - to: me@email.com
    send_resolved: true
    html: '{{ template "email.html" . }}'
    headers:
      subject: '{{ template "email-subject" . }}'
- name: logs
  email_configs:
  - to: me@email.com
    send_resolved: false
    html: '{{ template "email.html" . }}'
    headers:
      subject: '{{ template "email-subject" . }}'
inhibit_rules:
- source_match:
    severity: critical
  target_match:
    severity: warning
  equal:
  - alertname
route:
  group_by:
  - job
  - node
  group_interval: 1s
  group_wait: 1s
  receiver: admins
  repeat_interval: 12h
  routes:
  - receiver: logs
    repeat_interval: 10m
    match_re:
      alertname: ErrorsInAppLogs
templates:
- '*.tmpl'

With the settings above, I receive a duplicate message after 5 or more minutes or don’t receive one. Is it possible to get an alert for every single error in logs? It doesn’t matter if the alerts are grouped or not. I just don’t want Loki to skip or duplicate alerts. Thanks in advance.

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.