Request for Assistance in Configuring Grafana Alert Templates and Silencing Specific Alerts

Grafana Alerting
, , ,
1

Hi All,

We are using Grafana as our monitoring and alerting tool. Among other alerts, we have configured it to notify us if a Lambda function executes for more than 30 seconds. However, we have a few AWS Lambda functions that are expected to run for more than 30 seconds, and we keep receiving alerts for these, indicating they are executing beyond the threshold.

To address this, my first step was to update our alerting template.

Below is our existing template:

{{ if gt (len .Alerts.Firing) 0 }}{{ len .Alerts.Firing }} Alert Firing{{ end }} {{ if gt (len .Alerts.Resolved) 0 }}{{ len .Alerts.Resolved }} Resolved{{ end }}

I updated it as follows:

{{ if gt (len .Alerts.Firing) 0 }}{{ len .Alerts.Firing }} Alert(s) Firing for {{ .Labels.job }}{{ end }} {{ if gt (len .Alerts.Resolved) 0 }}{{ len .Alerts.Resolved }} Resolved for {{ .Labels.job }}{{ end }}

However, I’m not receiving alerts as expected—the part where I included .Labels.job is being ignored.

image
image822×444 32.7 KB

Is there any detailed documentation on Grafana alert templates or any tool to verify my custom alert template? Alternatively, how do you silence certain types of known alerts?

Other than that I tried to ignore it in the query it self like below

REMOVE_EMPTY(SEARCH('{AWS/Lambda,FunctionName!="cdkWorkshopStack-ErrorHandler4C8EBDAE-aMML3F6k2AoH"} MetricName="Duration"', 'Average', 300))

But still its returning the lambda cdkWorkshopStack-ErrorHandler4C8EBDAE-aMML3F6k2AoH in the results.

Is my query wrong ?
Any help is appreciated thanks.

2

Perhaps you do not group by job label. Try check out docs Notification template reference | Prometheus.

.Lables.job is attribute of a single alert but the .Alerts is a list. You need to use GroupLabels (make sure you add this label to route’s group_by setting) or CommonLabels (won’t work if there are two alerts with different job label).