Months of Annotations Not Showing on Dashboard

Grafana Dashboards
1
  • What Grafana version and what operating system are you using?

Grafana v10.0.3 (eb8dd72637) on Docker

  • What are you trying to achieve?

I am trying to view all my annotations on a dashboard. These annotations have been visible for months and now they are gone. As of today, only annotations made today are showing. I recently upgraded from Grafana v9.x to v10.0.3 but I’m not 100% sure that this issue is related to the upgrade.

  • How are you trying to achieve it?

I am using the SQLite database to store annotations. I can see the annotations (in the annotations table) in the database using DB Browser for SQLite. I can also query annotations using the API. For example, here are two entry that is in the database and that is return by the API. The first one shows in the GUI but the second one does not.

Shows in GUI

Annotation Table

id org_id alert_id user_id dashboard_id panel_id category_id type title text metric prev_state new_state data epoch region_id tags created updated epoch_end
84909 1 0 4 20 0 Start eraserltemp.vbs {} 1691596803000 0 [eraserltemp.vbs] 1691596803677 1691596865124 1691596865000

Annotation Tag Table

id annotation_id tag_id
1092 84909 10

Does NOT Show in GUI

Annotation Table

id org_id alert_id user_id dashboard_id panel_id category_id type title text metric prev_state new_state data epoch region_id tags created updated epoch_end
84367 1 0 4 20 0 Start eraserltemp.vbs {} 1691510404000 0 [eraserltemp.vbs] 1691510405502 1691510476088 1691510466000

Annotation Tag Table

id annotation_id tag_id
1045 84367 10

JSON Query to Annotation API

Query:

http://localhost:3003/api/annotations?tags=eraserltemp.vbs&limit=2

Results:

[
{
“id”: 84909,
“alertId”: 0,
“alertName”: “”,
“dashboardId”: 20,
“dashboardUID”: “dPGkzem4k”,
“panelId”: 0,
“userId”: 0,
“newState”: “”,
“prevState”: “”,
“created”: 1691596803677,
“updated”: 1691596865124,
“time”: 1691596803000,
“timeEnd”: 1691596865000,
“text”: “Start eraserltemp.vbs”,
“tags”: [
“eraserltemp.vbs”
],
“login”: “sa-pause-alerts”,
“email”: “sa-pause-alerts”,
“avatarUrl”: “/avatar/783a6ae320953901d76160ccef2a4639”,
“data”: {}
},
{
“id”: 84367,
“alertId”: 0,
“alertName”: “”,
“dashboardId”: 20,
“dashboardUID”: “dPGkzem4k”,
“panelId”: 0,
“userId”: 0,
“newState”: “”,
“prevState”: “”,
“created”: 1691510405502,
“updated”: 1691510476088,
“time”: 1691510404000,
“timeEnd”: 1691510466000,
“text”: “Start eraserltemp.vbs”,
“tags”: [
“eraserltemp.vbs”
],
“login”: “sa-pause-alerts”,
“email”: “sa-pause-alerts”,
“avatarUrl”: “/avatar/783a6ae320953901d76160ccef2a4639”,
“data”: {}
}
]

Screenshot from last 7 days

The visible annotation has time in the json response as:
“time”: 1691596803000,
which translates to GMT : Wednesday, August 9, 2023 4:00:03 PM

The non-visible annotation has time in the json response as :
“time”: 1691510404000
which translates to GMT : Tuesday, August 8, 2023 4:00:04 PM

image
image1920×964 136 KB

JSON Model snippet

I cannot include the entire dashboard because of character limitations in the post but here you see the UID for the dashboard.

{
“timezone”: “browser”,
“title”: “Windows Host Overview”,
“uid”: “dPGkzem4k”,
“version”: 12,
“weekStart”: “”
}

  • What happened?

I don’t know exactly. Maybe the upgrade?

  • What did you expect to happen?

I expect to see all my annotations. I have a lot of them on this system but I’m not seeing any beside the one from today.

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

No

  • Did you follow any online instructions? If so, what is the URL?

No

1 Like
2

In case, someone else stumbles on a similar issue, I found what can be described as a work around. The call on the webpage to get the annotations was returning not only annotations but alerts events. I had one alert that was generating a lot of entries. I’m not sure why it was generating all those entries just yet but I’m working through that. In the end, I can also just disable that alert. I then used the Annotations API to retrieve and then delete all the alert events that had a state of “Normal”. Once I did that, I then was able to see my past annotations.

At this point, I’m left to guess that the upgrade was the event that lead to a change in behavior in how annotations are rendered. I did not delve deeply into the root cause but I few things stood out:

  1. The limit of the number of annotations is 100 on the API call on the webpage. Maybe this is too small?
  2. It’s not clear to me why the webpage would need to have all the alert events. The vast, vast majority of the alert events were reporting a state change from “Normal” to “Normal (Updated)” (the table has a previous state and state column…I think that’s the labels). I question the need to return alert events indicating that an alert went from being OK to being OK. But again, that’s just a hunch.

Paolo

1 Like