.EndAt same as .StartsAt when alarm ends

Grafana Alerting
1

I have Grafana v11.5.1 (c6c701cf5b) with Prometheus.

Sometimes when i get notification about ended alert it has same end time as start time.
For example:

Started: 2025-03-10 20:18:30 UTC
Ended: 2025-03-10 20:18:30 UTC

Here is snipped from notification template.

Started: <code>{{ .StartsAt | date "2006-01-02 15:04:05 MST" -}}</code>
{{ if eq .Status "resolved" -}}
Ended: <code>{{ .EndsAt | date "2006-01-02 15:04:05 MST" -}}</code>
{{ end }}

What can be the cause of this?

2

I have the same problem. Did you manage to figure out what the problem is?

3

I have also the same problem. Any news regarding this? @chdanilpro @dimabzz

4

This problem mysteriously appeared after upgrading to Grafana 12, then just as inexplicably, it suddenly disappeared following service restarts.

My suspicion here is that, in our case, the issue was incompatible alertmanager versions. The flow went something along the lines of: Grafana 11.6.0->Grafana 12.0.0 (issue appears), alertmanager some earlier version->altermanager latest version (issue persists), grafana-server service restarted (issue solved).

Assumption: Update and [re]start alertmanager before grafana-server to avoid this.

5

I retract that. Restarting merely altered the behaviour somewhat. Instead of almost always firing off with startsAt = endsAt, it would only sometimes do this, but still tag the previous start time with an end time beforehand.

We are now running with the feature flag alertRuleUseFiredAtForStartsAt enabled, which we believe will correct this issue. (It is also believed to be resolved in 12.1)