This is kind of a crazy question, I guess, but I’m trying to avoid repeating things and keeping the number of alerts more manageable.
Let’s say I have an alert tagged with priority=p5 and a rule that says, for example, that if CPU usage is above 80% it should fire.
Now let’s assume that if the CPU usage is above 99% then it should be priority=p3.
I can have these with separate alerts, each with its own priority. That’s simple enough but if I have 5000 alerts, I could cut down the number of alerts in half if they had some way to define that priority label based on certain parameters.
Is there anything out there like that? Is it even doable in Grafana?
And if it’s not doable, if really wanted to develop something to enable Grafana to accept, say, a table of parameters (a templated alert?), how would I go about this? A plugin?
You can make priority a custom label or a custom annotation, which will let you have a single alert rule for all priorities. Whether you make it a label or an annotation depends whether you want each priority to be a separate alert, or the same alert that is updated over time.
I think I don’t follow. You mean maybe having a label priority_cpu80 and priority_cpu99 in the same alert and then in some annotation and/or notification template use that label to, say, inform some 3rd-party system of the priority?
No 
You know that labels are the unique identifier for an alert right? If the labels change, the alert changes.
What I’m suggesting is templating a custom label using the result of your query to set priority_cpu to p1, p2, etc - depending on the result of the query.
For example, if you have a query with Ref ID “A” and a Reduce expression with Ref ID “B”:
Then create a custom label called priority, such as:
{{ $v := $values.B.Value }}{{ if gt $v 99.0 }}p1{{ else if gt $v 90.0 }}p2{{ else if gt $v 80.0 }}p3{{ else if gt $v 75.0 }}p4{{ else }}p5{{ end }}
And then you can have a single alert rule fire alerts with different priorities for different CPUs:
2 Likes
Wow, I didn’t know we could have templates inside a label’s value field. Amazing.
Thanks a lot, this will do it.
Hi @georgerobinson , may I know what Grafana version you using?
I couldn’t make it work with custom labels, I’m using version 10.0.3. I think using the If statement is only doable in templates.
I get this part. But how do I then send it to a contact group?
I couldn’t make it work with custom labels, I’m using version 10.0.3. I think using the If statement is only doable in templates.
Templating custom labels works in all versions of Grafana since 8.0. However, $values was added in Grafana 8.1.
via notification policies. Have you seen this graph on the Alerting landing page in Grafana?
This is what I get in my subject and Labels - Priority
[FIRING:2] Memory Internals ({{ $v := $values.B.Value }}{{ if gt $v 90 }}Critical{{ else }}Warning{{ end }} server)
My expressions and labels:
1 Like
My goal is to contact the sysadmin contact. This works with sysadmin = server. But I need the priority correct
I finally got around to trying this and I’m having the same issue.
The priority label is populated with the raw value, not the evaluation of the Go template.
