Hello. I want to receive an alert on every error in logs. I have Logstash now and it sends me emails on every error in some application. But I can’t sort out how to do it in Loki. I’ve tried several rules like:
- alert: ErrorsInAppLogs
expr: |
sum(count_over_time({label="app",level="error"} | regexp `(?P<log>.+)`[5m])) by (log, label, container, hostname, job, logger, pod) > 0
for: 15s
labels:
severity: critical
and
- alert: ErrorsInAppLogs
expr: |
count_over_time({label="app",level="error"}[5m]) > 0
for: 15s
labels:
severity: critical
In both cases, I’ve received either zero or double alerts for errors in logs. But Logstash still sent emails on every error without doubling or skipping.
Alertmanager config:
global:
smtp_from: some@email.com
smtp_smarthost: smtp.email.com:587
smtp_hello: smtp.email.com
smtp_auth_username: some@email.com
receivers:
- name: admins
email_configs:
- to: me@email.com
send_resolved: true
html: '{{ template "email.html" . }}'
headers:
subject: '{{ template "email-subject" . }}'
- name: logs
email_configs:
- to: me@email.com
send_resolved: false
html: '{{ template "email.html" . }}'
headers:
subject: '{{ template "email-subject" . }}'
inhibit_rules:
- source_match:
severity: critical
target_match:
severity: warning
equal:
- alertname
route:
group_by:
- job
- node
group_interval: 1s
group_wait: 1s
receiver: admins
repeat_interval: 12h
routes:
- receiver: logs
repeat_interval: 10m
match_re:
alertname: ErrorsInAppLogs
templates:
- '*.tmpl'
With the settings above, I receive a duplicate message after 5 or more minutes or don’t receive one. Is it possible to get an alert for every single error in logs? It doesn’t matter if the alerts are grouped or not. I just don’t want Loki to skip or duplicate alerts. Thanks in advance.