We create Grafana Alerts using a YAML file:
- name: KubePodCrashLoopBackTools
expr: (sum by (cluster_type) (kube_pod_container_status_waiting_reason{cluster_type="tools", reason="CrashLoopBackOff", environment="xxx", job="yyy/metircs"})) / count by (cluster_type) ((kube_pod_info{cluster_type="tools", environment="x", job="y/metircs"})) * 100 > 5
for: 25m
labels:
severity: critical
annotations:
description: In the {$label.cluster_type} cluster, at {$labels.environment}, in the region { $labels.region}, there is above 5% of pods in ${labels.reason} state.
summary: In the {$label.cluster_type} cluster, at {$labels.environment}, in the region { $labels.region}, there is above 5% of pods in ${labels.reason} state.
runbook: tbd
Right now, the threshold 5%
is been displayed statically in the summary and description.
I would like to be able to display it dynamically.
Something like : above {{ $threshold }}% of pods
or above {{ $value }}%
.
So, if I decided to change the threshold in the query from 5 to any other value, I don’t have to update the summary and description with the new value.
Do anyone knows if this is possible and/or supported?