Grafana alert with multiple "stages"

Grafana Alerting
1

I want to create an alert on a multicondition case.
value >= 0.8 alert with lable severity=medium
value >= 0.95 alert with lable severity=high
value >= 0.99 alert with lable severity=urgent

if the condition of a higer severity is fulfilled, the lower alert can be quite. (I do not say silent, because it is a special word)

my only solution is to setup following alert rules:

  • >=0.8 && <0.95
  • >=0.95 && <0.99
  • >=0.99

on the downside: with every change there would be a resolve-notification as well.

Is there a more elegant way to do that?

2

Hello,

Can you show us your query? Is “severity” a label that is generated by the query?

3

My query gives me just the values and I set a Threshold on it. The ‘severity’ is a label I set on the alert to react on it in policies. I want to use different contactpoints.

The query:

sum(kubelet_volume_stats_used_bytes) by (persistentvolumeclaim, namespace) 
/
sum (
 sum(kube_persistentvolumeclaim_info) by (persistentvolumeclaim, namespace, storageclass)
 * on (persistentvolumeclaim, namespace) group_right(storageclass)
 sum(kubelet_volume_stats_capacity_bytes) by (persistentvolumeclaim, namespace)
) by (namespace,persistentvolumeclaim)
4

You can use the label_replace function to create the label based on the condition you want.

In this example, I create a new query for every condtion, and wrap the metric in the label_replace:

label_replace(v instant-vector, dst_label string, replacement string, src_label string, regex string)
v: your query
dst_label: the label name
replacement: the label value
set src_label and regex to empty strings as you don't want to modify existing labels.

image
image1010×746 59.3 KB

I’m not sure if you set different notification policies with each label and value linked to a contact point will capture the label generated in the alert without specifying it in the “Notifications” section.

Try it, if it doesn’t work then you should create 3 alerts each linked to a notification policy based on the label value.

5

@clevernessisamyth thank you for that input! That works so far as you can see here

image
image1476×996 89 KB

It is nice to use the metrics intuitivly and get some successes :wink:

Currently I have three queries and they are responding some data.
How have you combined them to create alerts from them. I can just set one of these config blocks as ‘alert condition’. Also in my current configuration I get ‘No data’ for the rule. But two of the queries are responding data.

image
image1073×583 30.7 KB

6

This thread may also help.

2 Likes
7

Wow, I didn’t know it was possible to use go template expressions in the notifications section.

8

@grant2, thank you. That was helpfull.

Now I got alerts with dynamic severity label. It reacts on the value of a calculation.

You If I configure a label more than one time, just one of these lables are saved to the final configuration. I do that via GUI.
If I try to do it in the yaml, I get the hint from my IDE, that it in no valid yaml.

          labels:
            ressource: storage
            severity: '{{if gt $values.B.Value 80.0 }}warning{{else}}none{{end}}'
            severity: '{{if gt $values.B.Value 90.0 }}high{{else}}none{{end}}'
            severity: '{{if gt $values.B.Value 95.0 }}critical{{else}}none{{end}}'

As you´ve shown here, you setup a multi condition case. How have you done that?

9

Hello,

In YAML, each key should be unique within a mapping. Try this:

labels:
  ressource: storage
  severity: '{{if gt $values.B.Value 95.0 }}critical{{else if gt $values.B.Value 90.0 }}high{{else if gt $values.B.Value 80.0 }}warning{{else}}none{{end}}'
1 Like