How to get a separate alert for each firing series?

Grafana Alerting
,
1

I have an alert rule query against Loki like:

sum(count_over_time({unit=“auditd-log-alerts.service”} | logfmt | error = “” and level_code <= 3 [10s])) by (_id, type, desc, uid, path).

But Grafaana alert manager is sending one alert for many time series. With the default template I get an alert like this for example:

…
Value: [ metric=‘{_id=“132389”, desc=“Unauthorized access attempts to files (unsuccessful)”, path=“/dev/tty0”, type=“SYSCALL”, uid=“1000”}’ labels={_id=132389, desc=Unauthorized access attempts to files (unsuccessful), path=/dev/tty0, type=SYSCALL, uid=1000} value=1 ], [ metric=‘{_id=“132391”, desc=“Unauthorized access attempts to files (unsuccessful)”, path=“/dev/tty0”, type=“SYSCALL”, uid=“1000”}’ labels={_id=132391, desc=Unauthorized access attempts to files (unsuccessful), path=/dev/tty0, type=SYSCALL, uid=1000} value=1 ], [ metric=‘{_id=“132392”, desc=“Unauthorized access attempts to files (unsuccessful)”, path=“/dev/console”, type=“SYSCALL”, uid=“1000”}’ labels={_id=132392, desc=Unauthorized access attempts to files (unsuccessful), path=/dev/console, type=SYSCALL, uid=1000} value=1 ], [ metric=‘{_id=“132393”, desc=“Unauthorized access attempts to files (unsuccessful)”, path=“/dev/console”, type=“SYSCALL”, uid=“1000”}’ labels={_id=132393, desc=Unauthorized access attempts to files (unsuccessful), path=/dev/console, type=SYSCALL, uid=1000} value=1 ], [ metric=‘{_id=“132394”, desc=“Unauthorized access attempts to files (unsuccessful)”, path=“/dev/console”, type=“SYSCALL”, uid=“1000”}’ labels={_id=132394, desc=Unauthorized access attempts to files (unsuccessful), path=/dev/console, type=SYSCALL, uid=1000} value=1 ]
…

The Value is very hard to read. I actually want one alert per series that is firing. How can I make Grafana send one alert for every timeseries that is firing?

Note there is a section in the grafana alerting docs that says:

“… select Classic condition to create a single alert rule, or choose from Math, Reduce, Resample options to generate separate alert for each series.”

But none of Reduce, Math, or Resample seem to achieve state outcome of “separate alert for each series”. Reduce by definition, reduce many series to one, Math is just a more general form of classic, Resample isn’t explained but I couldn’t get anything out of it.

2

are you using a multi-dimensional rule?

3

Hi, @mattabrams!
I also have the same question
I think my rule is multi-dimensional, because im using sum by (cluster, deployment, namespace) in my query.
Results look like this:

[ var='B0' metric='Value' labels={cluster=a-cluster, deployment=a-deployment, namespace=default, pod=a-pod} value=100 ],
[ var='B1' metric='Value' labels={cluster=b-cluster, deployment=b-deployment, namespace=default, pod=b-pod} value=100 ],
[ var='B2' metric='Value' labels={cluster=c-cluster, deployment=c-deployment, namespace=default, pod=c-pod} value=100 ]

I want to have these three alerts separately…
This example might be three, but with real data it can have a lot more so i really want to avoid making separate queries for each alert i wanna get,

Is there a way?

4

Hi! Classic Conditions are uni-dimensional, and when given multi-dimensional data will reduce it to a single dimension. Reduce, Math, and Threshold create multi-dimensional alerts when given multi-dimensional data. You need to Reduce the input of the query to create a single value per dimension, and then you can use a Math or Threshold expression to compare it to a threshold.

1 Like