The Annotation list panel is no longer updating from alert annotation

Skywalker · March 17, 2023, 9:53am

Hello,
The Annotation list panel is no longer updating from alert annotation history for me, is this a bug or a known issue? ( using 9.4.3, manual annotations gets added, but not the once from alerting)

georgerobinson · March 17, 2023, 1:18pm

Hi! It is working for me in Grafana 9.4.3 (please see screenshot):

Can you turn on debug logging in Grafana and look for Alert state changed creating annotation in the logs:

DEBUG[03-17|13:16:00] Saving alert states                      logger=ngalert.state.manager rule_uid=efb5df22-9a7e-4193-a22a-dc8ba1a88e04 org_id=1 count=2
DEBUG[03-17|13:16:00] Alert state changed creating annotation  logger=ngalert.state.historian rule_uid=efb5df22-9a7e-4193-a22a-dc8ba1a88e04 org_id=1 newState=Alerting oldState=Normal
DEBUG[03-17|13:16:00] Alert state changed creating annotation  logger=ngalert.state.historian rule_uid=efb5df22-9a7e-4193-a22a-dc8ba1a88e04 org_id=1 newState=Alerting oldState=Normal
INFO [03-17|13:16:00] Sending alerts to local notifier         logger=ngalert.sender.router rule_uid=efb5df22-9a7e-4193-a22a-dc8ba1a88e04 org_id=1 count=2

Skywalker · March 17, 2023, 1:43pm

Hello,

They show up in the timeseries panel, but not in the annotation list panel:
Same issue as this guy it seems:

Found that one after
Also tested with nightly build and same issue there.

georgerobinson · March 17, 2023, 1:45pm

Ah apologies, I misunderstood. I’m not familiar with that panel, but it seems like it could be a bug. Could you open an issue for this on Issues · grafana/grafana · GitHub?

Skywalker · March 17, 2023, 1:51pm

Hello,
I’ve reported here now: Alerts states annotations not shown on Annotation list panel · Issue #64961 · grafana/grafana · GitHub

mikhailvolkov · March 17, 2023, 5:23pm

The annotation panel does not show Alerts since it’s filtering annotations with a specific type.

We recently created a Grafana HTTP API data source that allows retrieving annotations and alerts and then displaying them using any available panel in your style.

Blog post with details.

Skywalker · March 17, 2023, 8:53pm

Hello,
Thanks, very interesting, I tested the HTTP API plugin, it seems to retrieve only one alert ID and when I select Dashboards “This” on a dashboard I have active alarms and annotations it does not show any, is it working properly for 9.4.3?
Thanks!

mikhailvolkov · March 17, 2023, 9:11pm

@Skywalker, try to increase Max Limit to 1000 or more. By default, API retrieves 100.

Are you sure alarms and annotation have Dashboard UID set to the current dashboard? You can check the Dashboard UID in the returned results for all dashboards.

Skywalker · March 17, 2023, 10:31pm

Hello,

The dashboard UID is the same as in the alert and using the config below:

Alert conf:

url:

There are both active and pending alerts on the dashboard but it only returns No data using the plugin.

I did notice that when I select Dashboard ALL and check Dashboard UID none show the Dashboard UID, Actually, it only shows the ones that do not have a dashboard UID.

Do you have any other tips?

Skywalker · March 18, 2023, 11:47am

Hello,

I did figure out what was wrong.
The API user was set as a viewer, but when I changed to admin I could see all dashboard UIDs and the plugin was working as expected.

I have a follow-up question, Regarding the string that is in the text column, is it possible to separate the labels into rows? I tried using transform → extract fields but it creates a bit of a mess.

mikhailvolkov · March 18, 2023, 6:18pm

@Skywalker As we discussed in Slack, a feature request opened Extract Annotation Alarm labels · Issue #16 · VolkovLabs/volkovlabs-grapi-datasource · GitHub

PR merged and will be a part of the upcoming release.