The Annotation list panel is no longer updating from alert annotation

[Skywalker]

Hello,
The Annotation list panel is no longer updating from alert annotation history for me, is this a bug or a known issue? ( using 9.4.3, manual annotations gets added, but not the once from alerting)

[georgerobinson]

Hi! It is working for me in Grafana 9.4.3 (please see screenshot):

Can you turn on debug logging in Grafana and look for Alert state changed creating annotation in the logs:

DEBUG[03-17|13:16:00] Saving alert states                      logger=ngalert.state.manager rule_uid=efb5df22-9a7e-4193-a22a-dc8ba1a88e04 org_id=1 count=2
DEBUG[03-17|13:16:00] Alert state changed creating annotation  logger=ngalert.state.historian rule_uid=efb5df22-9a7e-4193-a22a-dc8ba1a88e04 org_id=1 newState=Alerting oldState=Normal
DEBUG[03-17|13:16:00] Alert state changed creating annotation  logger=ngalert.state.historian rule_uid=efb5df22-9a7e-4193-a22a-dc8ba1a88e04 org_id=1 newState=Alerting oldState=Normal
INFO [03-17|13:16:00] Sending alerts to local notifier         logger=ngalert.sender.router rule_uid=efb5df22-9a7e-4193-a22a-dc8ba1a88e04 org_id=1 count=2

[Skywalker]

Hello,

They show up in the timeseries panel, but not in the annotation list panel:
Same issue as this guy it seems:

Alerts states automatic annotations not shown on Annotation list panel

Hi, everyone! I am using Grafana 8.4.3 and the Unified Alerting system. In former versions I used to use Alert List panel to show the historical state of the alerts defined on the system, but this feature seems not to be supported anymore on the 8.x.x versions. I am trying to get the same functionality by other means, so I decided to use the new Annotation list panel. According to the documentation this panel should be able to take the annotations from one Dashboard (or the whole system) and …

Found that one after
Also tested with nightly build and same issue there.

[georgerobinson]

Ah apologies, I misunderstood. I’m not familiar with that panel, but it seems like it could be a bug. Could you open an issue for this on Issues · grafana/grafana · GitHub?

[Skywalker]

Hello,
I’ve reported here now: Alerts states annotations not shown on Annotation list panel · Issue #64961 · grafana/grafana · GitHub

[mikhailvolkov]

The annotation panel does not show Alerts since it’s filtering annotations with a specific type.

We recently created a Grafana HTTP API data source that allows retrieving annotations and alerts and then displaying them using any available panel in your style.

Grafana HTTP API plugin for Grafana | Grafana Labs

Blog post with details.

Grafana HTTP API Data Source 1.0.0 | Volkov Labs

[Skywalker]

Hello,
Thanks, very interesting, I tested the HTTP API plugin, it seems to retrieve only one alert ID and when I select Dashboards “This” on a dashboard I have active alarms and annotations it does not show any, is it working properly for 9.4.3?
Thanks!

[mikhailvolkov]

@Skywalker, try to increase Max Limit to 1000 or more. By default, API retrieves 100.

Are you sure alarms and annotation have Dashboard UID set to the current dashboard? You can check the Dashboard UID in the returned results for all dashboards.

[Skywalker]

Hello,

The dashboard UID is the same as in the alert and using the config below:

Alert conf:

url:

There are both active and pending alerts on the dashboard but it only returns No data using the plugin.

I did notice that when I select Dashboard ALL and check Dashboard UID none show the Dashboard UID, Actually, it only shows the ones that do not have a dashboard UID.

Do you have any other tips?

[Skywalker]

Hello,

I did figure out what was wrong.
The API user was set as a viewer, but when I changed to admin I could see all dashboard UIDs and the plugin was working as expected.

I have a follow-up question, Regarding the string that is in the text column, is it possible to separate the labels into rows? I tried using transform → extract fields but it creates a bit of a mess.

[mikhailvolkov]

@Skywalker As we discussed in Slack, a feature request opened Extract Annotation Alarm labels · Issue #16 · VolkovLabs/volkovlabs-grapi-datasource · GitHub

PR merged and will be a part of the upcoming release.

[Pugachev_Ilay]

Hello everyone,

I think I have the same problem. Recently, I have been working with Grafana 7.5.5, and in this version, the system for alerting and generating annotations was working different. The alert query was created on the panel in the dashboard, and then the alert was fired, after which Grafana automatically created the annotation line on the time series panel.

Now I am using Grafana 9.3.1, and if I use the standard Prometheus dashboard and a simple alert rule, then the alert works, but the annotation is not automatically created.

I have watched many tutorials, but they haven’t helped me.

Do you find the solution for this issue?

[mikhailvolkov]

@Pugachev_Ilay Have you tried in 9.4.7 or 9.5.0? How do you know that annotations are not automatically created? You don’t see them on the panel.

Do you see anything returned from the /api/annotations endpoint in the Browser Console → Network?

Looking at the latest screenshot, Alert rules generate Alert Annotations when Annotations are Native manual annotations. To retrieve Alert annotations generated from Alert rules, you need to select All or Alert.

Have you seen this video? It’s for Timescale and Grafana 9.4.7, but the steps for Prometheus are the same.

How to setup Annotations and Alerts in Grafana with Timescale | Demo to reveal the basics

[Pugachev_Ilay]

It’s unbelievable, I installed Grafana 9.4.7 and all annotations are finally working!

I think it’s sad and funny at the same time, because I tested versions 9.3.1, 9.3.6 and 9.4.3 and it wasn’t working. But with version 9.4.7, it worked “out of the box”. I just used the included instruments of Grafana and it worked!

Also, I tested the “Grafana HTTP API” plugin on versions 9.3.6, 9.4.3 and 9.4.7. Unfortunately, it worked only on 9.4.7. Maybe I’m doing something wrong, I don’t know. But I have to say that I literally did everything I saw in Daria Volkova’s video.

Anyway, I’d like to thank you so much for your support and help! I’m really enjoying the Volkov Labs YouTube channel too!

[mikhailvolkov]

@Pugachev_Ilay Great to hear.

The latest version of Grafana HTTP API requires 9.4.0 to retrieve Alert Rules to get Alert Name and UID. The required API endpoint was introduced in 9.4.0, which is why it’s working for 9.4.7 for you.
The issue was fixed in the main and will be a part of the upcoming release.

[Pugachev_Ilay]

Good news, thank you!

[Pugachev_Ilay]

Hi Mikhail, tell me please, сauld I install Grafana HTTP API plugin selectively without sudo?
I’m start grafana from user on my server and I don’t have any sudo rights. Could you write the steps to run your plugin without sudo commands?

[mikhailvolkov]

@Pugachev_Ilay Who owns the plugins folder on your server? If the directory has write permissions for your user, you should not have any issues copying the plugin files in the folder.

1. Download the latest version from GitHub.
2. Extract the plugin in the plugins folder.

Let me know if it helps.

[Pugachev_Ilay]

Hello Mikhail
Thank you, your advice helped me. I’m successfuly install grapi. But it doesn’t helped me to resolve finally issue.

I have a complicated question. At first, I want to explain what is my “finally issue”. I have dashboard on grafana with many pannel and different datasourse. On several pannels I have custom alert rules. And when alert rule is triggered I saw an annotations on pennel with some text (name of alert rule, dashboard folder, dashboard pannel name, A=…, B=… ). Maybe I forgot something.

We have large and incomprehensible description of alert, because of this, the annotation becomes useless.

I think, perfect results is it.
Alert triggered → activate annotation on panel → discribe of annotation, something like that (Error: query name, A=…, B=…)

On grafana 7.5.5, when we have old system of alerting it was possible “from the box” but now it doesn’t look simple.

Maybe I can do that with Grafana HTTP API plugin?
I tried to fix this via Grafana HTTP API but it seems like text of annotation doesn’t have many ways to customisation.
Maybe, I do something wrong

Now I use grafana 9.5.1, datasourse elasticsearch. Query and alert rules manedge from UI grafana

[mikhailvolkov]

@Pugachev_Ilay So you want to have customization for text in the Annotations? Not just being able to select a corresponding field?

Unfortunately, there is no customization in Alerts of how Text should look like. Maybe it’s on the roadmap. I can think of adding a customization option for the Text if it’s not.

Please open an issue in the repository, GitHub - VolkovLabs/volkovlabs-grapi-datasource: Grafana HTTP API Data Source for @grafana.