Alerting By Threshold Status

Grafana Alerting
1

I have a scenario I am trying to create. I would like to have three notification groups dynamically responding to the following:

In my alert under custom labels have the tags:

Team = Sysadmin
Severity = {{ if gt $values.C.Value 90.0 }}Critical{{ else }}Warning{{ end }}

For notification policies I would like the actions for this team to get the following:

Email alerts for:
Warnings - Monday-Friday 8 am to 5 pm, and none on Saturday and Sunday
Critical - 24/7

Team = Sysadmin Oncall
Severity = {{ if gt $values.C.Value 90.0 }}Critical{{ else }}Warning{{ end }}

Email alerts for all:
24/7

I think I can do the two teams, Sysadmin and Sysadmin Oncall and have two different muting policies, but what I would like to figure out how to do dynamically is have it where emails will be sent 24/7 to Sysadmin team if the threshold is critical.

Is there a way to do this?

2

Hi! :wave:

Team = Sysadmin Oncall
Severity = {{ if gt $values.C.Value 90.0 }}Critical{{ else }}Warning{{ end }}

Is this your notification policy? If so, you cannot have templates there. If only critical alerts should match this policy then it should simply be:

Team = Sysadmin Oncall
Severity = Critical
3

I figured out how to do it.

I created a Teams Critical = Sysadmin Critical and a Teams OnCall = Sysadmin OnCall
The warnings are sent during business hours, and to the on call after hours and critical goes to everyone on the sysadmin team