The Annotation list panel is no longer updating from alert annotation history for me, is this a bug or a known issue? ( using 9.4.3, manual annotations gets added, but not the once from alerting)
Hi! It is working for me in Grafana 9.4.3 (please see screenshot):
Can you turn on debug logging in Grafana and look for
Alert state changed creating annotation in the logs:
DEBUG[03-17|13:16:00] Saving alert states logger=ngalert.state.manager rule_uid=efb5df22-9a7e-4193-a22a-dc8ba1a88e04 org_id=1 count=2 DEBUG[03-17|13:16:00] Alert state changed creating annotation logger=ngalert.state.historian rule_uid=efb5df22-9a7e-4193-a22a-dc8ba1a88e04 org_id=1 newState=Alerting oldState=Normal DEBUG[03-17|13:16:00] Alert state changed creating annotation logger=ngalert.state.historian rule_uid=efb5df22-9a7e-4193-a22a-dc8ba1a88e04 org_id=1 newState=Alerting oldState=Normal INFO [03-17|13:16:00] Sending alerts to local notifier logger=ngalert.sender.router rule_uid=efb5df22-9a7e-4193-a22a-dc8ba1a88e04 org_id=1 count=2
They show up in the timeseries panel, but not in the annotation list panel:
Same issue as this guy it seems:
Found that one after
Also tested with nightly build and same issue there.
Ah apologies, I misunderstood. I’m not familiar with that panel, but it seems like it could be a bug. Could you open an issue for this on Issues · grafana/grafana · GitHub?
I’ve reported here now: Alerts states annotations not shown on Annotation list panel · Issue #64961 · grafana/grafana · GitHub
The annotation panel does not show Alerts since it’s filtering annotations with a specific type.
We recently created a Grafana HTTP API data source that allows retrieving annotations and alerts and then displaying them using any available panel in your style.
Blog post with details.
Thanks, very interesting, I tested the HTTP API plugin, it seems to retrieve only one alert ID and when I select Dashboards “This” on a dashboard I have active alarms and annotations it does not show any, is it working properly for 9.4.3?
@Skywalker, try to increase Max Limit to 1000 or more. By default, API retrieves 100.
Are you sure alarms and annotation have Dashboard UID set to the current dashboard? You can check the Dashboard UID in the returned results for all dashboards.
The dashboard UID is the same as in the alert and using the config below:
There are both active and pending alerts on the dashboard but it only returns No data using the plugin.
I did notice that when I select Dashboard ALL and check Dashboard UID none show the Dashboard UID, Actually, it only shows the ones that do not have a dashboard UID.
Do you have any other tips?
I did figure out what was wrong.
The API user was set as a viewer, but when I changed to admin I could see all dashboard UIDs and the plugin was working as expected.
I have a follow-up question, Regarding the string that is in the text column, is it possible to separate the labels into rows? I tried using transform → extract fields but it creates a bit of a mess.
@Skywalker As we discussed in Slack, a feature request opened Extract Annotation Alarm labels · Issue #16 · VolkovLabs/volkovlabs-grapi-datasource · GitHub
PR merged and will be a part of the upcoming release.