Annotations Table size huge , heavy write IO to DB

Grafana Configuration
,
1

Hi ,

Noticing - Huge size of Annotations Table (40G) and heavy write I/O to DB.
Got around 3k Alerts running . Added purging / rotation to bring down the capacity.
purging dropped it to 25G, i believe that will not bring down the heavy writes to DB..

i can see that , grafana alert use annotations as backend.
As there is no usecase of annotations via dashboard , is there any harm to disable this globally ? any experience. Any one attempted and how ?

Am using Grafana 10.0.1 . these are the current config.

[alerting]
max_annotation_age = 1M
[annotations]
api = map[max_age:1M]
cleanupjob_batchsize = 10000
dashboard = map[max_age:1M]

Thanks and Regards
GVK

2

Hi, I have like a week of annotations max age (~2.5k rules, we had problems when updating Grafana, migrations wouldn’t end), no one said anything yet (we disabled it more than half a year ago). Also with that many alerts you can check alert_rule_version table and the setting for that - I don’t remember its name but it was something like how many versions should be kept in database.

3

Hi @dawiddebowski ,

Thank You . that helps.
May i know how do you disable the annotations for this alerting backend.

4

to reduce load on the annotation table you can update alert rules to not refer to any dashboards\panels (remove annotations).