Group alerts within notification by start time?

Grafana Alerting
,
1

I’m setting up alert notifications for an ISP, we monitor several hundred nodes across three service areas. We may have dozens of alerts firing at a time (power outages etc). Having an aggregated notification is wonderful, and a major improvement over our old system. However the default alert is far too verbose for our needs. I have simplified the template, to show a concise summary of the instances and timestamps. Here’s the template we’re using:

{{ define "email.message_outage" }}
{{ if .Alerts.Firing -}} 

The following {{ toLower (index (index .Alerts 1).Labels "system") }} {{ if eq (len .Alerts.Firing) 1}}site is{{ else -}}sites are{{ end }} offline in {{ title (index (index .Alerts 1).Labels "franchise") }}: 


{{ range .Alerts.Firing }} - {{ .StartsAt.Format "1/2/06 15:04 (UTC)" }} {{ index .Labels "instance" }} 
{{ end }}
{{ end }}
{{ if .Alerts.Resolved -}}

The following {{ toLower (index (index .Alerts 1).Labels "system") }} {{ if eq (len .Alerts.Resolved) 1}}site has{{ else -}}sites have{{ end }} been restored in {{ title (index (index .Alerts 1).Labels "franchise") }}: 

{{ range .Alerts.Resolved }} - {{ .EndsAt.Format "1/2/06 15:04 (UTC)" }} {{ index .Labels "instance" }} 
{{ end }}
{{ end }}

{{ end }}

However, during an outage several sites can be “flapping” ie going up and down repeatedly while the rest of the offlines are continually down. It can be hard to tell by glancing at an aggregated alert, which one triggered it. Here’s an example alert message:

The following sites are offline in Brooks:
1/12/24 20:18 (UTC) PS02-639 HOLSCLAW HILL RD
1/12/24 20:18 (UTC) PS03-5851 KNOB CREEK RD
1/12/24 20:18 (UTC) PS04-4769 KNOB CREEK RD
1/15/24 16:05 (UTC) PS06-1008 KNOB CREEK RD
1/12/24 20:18 (UTC) PS08-2878 WEAVERS RUN
1/12/24 20:18 (UTC) PS09-2171 SKYVIEW RD
1/12/24 20:18 (UTC) PS16-5400 BROOKS HILL RD
1/12/24 20:18 (UTC) PS19-455 COWBRANCH RD
1/12/24 20:18 (UTC) PS21-713 YORKSHIRE BLVD
1/12/24 20:18 (UTC) PS22-100 HURST RD
1/12/24 20:18 (UTC) PS23-1800 WEAVERS RUN
1/12/24 20:18 (UTC) PS24-995 ROE HILL RD

You can see the fourth instance is the notification trigger, as the rest of the sites have been offline for days. But it’s hard to pick that out, especially as the list gets longer. I would like to format it more like this:

The following site just now went offline in Brooks:
1/15/24 16:05 (UTC) PS06-1008 KNOB CREEK RD

The following sites continue to be offline in Brooks:
1/12/24 20:18 (UTC) PS02-639 HOLSCLAW HILL RD
1/12/24 20:18 (UTC) PS03-5851 KNOB CREEK RD
1/12/24 20:18 (UTC) PS04-4769 KNOB CREEK RD
1/12/24 20:18 (UTC) PS08-2878 WEAVERS RUN
1/12/24 20:18 (UTC) PS09-2171 SKYVIEW RD
1/12/24 20:18 (UTC) PS16-5400 BROOKS HILL RD
1/12/24 20:18 (UTC) PS19-455 COWBRANCH RD
1/12/24 20:18 (UTC) PS21-713 YORKSHIRE BLVD
1/12/24 20:18 (UTC) PS22-100 HURST RD
1/12/24 20:18 (UTC) PS23-1800 WEAVERS RUN
1/12/24 20:18 (UTC) PS24-995 ROE HILL RD

Math functions seem pretty limited in the template language, but I do see some comparators. How can I sort alerts by start time, or better yet group them by unique time stamps?

1 Like