I have Grafana v11.5.1 (c6c701cf5b) with Prometheus.
Sometimes when i get notification about ended alert it has same end time as start time.
For example:
Started: 2025-03-10 20:18:30 UTC
Ended: 2025-03-10 20:18:30 UTC
Here is snipped from notification template.
Started: <code>{{ .StartsAt | date "2006-01-02 15:04:05 MST" -}}</code>
{{ if eq .Status "resolved" -}}
Ended: <code>{{ .EndsAt | date "2006-01-02 15:04:05 MST" -}}</code>
{{ end }}
What can be the cause of this?
I have the same problem. Did you manage to figure out what the problem is?
I have also the same problem. Any news regarding this? @chdanilpro @dimabzz
This problem mysteriously appeared after upgrading to Grafana 12, then just as inexplicably, it suddenly disappeared following service restarts.
My suspicion here is that, in our case, the issue was incompatible alertmanager versions. The flow went something along the lines of: Grafana 11.6.0->Grafana 12.0.0 (issue appears), alertmanager some earlier version->altermanager latest version (issue persists), grafana-server service restarted (issue solved).
Assumption: Update and [re]start alertmanager before grafana-server to avoid this.
I retract that. Restarting merely altered the behaviour somewhat. Instead of almost always firing off with startsAt = endsAt, it would only sometimes do this, but still tag the previous start time with an end time beforehand.
We are now running with the feature flag alertRuleUseFiredAtForStartsAt enabled, which we believe will correct this issue. (It is also believed to be resolved in 12.1)
Hi, will this issue be resolved in grafana v12.3? . I want to know the result, I am currently facing this case.
Exact same problem here
grafana 12.3.0 problem exists
An explanation and fix for this issue can be found here: Alerting: Add FiredAt field to State by fayzal-g · Pull Request #104046 · grafana/grafana · GitHub
Until the fix is generally included in an upcoming Grafana version, the current solution is to set the alertRuleUseFiredAtForStartsAt feature toggle to true.
I have the same problem. I want to include the timestamp of when the alert was fired in the resolved alert message, but I cannot make it work.
For this reason, I tried adding the following parameter to my custom.ini file:
alertRuleUseFiredAtForStartsAt = true
as shown in the attached screenshot.
What should I do next in the notification template? Do I need to add something like .FiredAt inside the template in order to get the result I want? Is there anything else that needs to be configured?
I still cannot find the correct solution.