Hi,
We’ve recently had a new tenant on our OSS grafana service and their alerting rules go into pending and back into OK pretty often which has caused a massive increase in annotations.
We have just set the following:
[annotations]
cleanupjob_batchsize = 100000
[unified_alerting.state_history.annotations]
max_age = 1M
[annotations.dashboard]
max_age = 24M
[annotations.api]
max_age = 24M
However I am not seeing any decrease in annotations… I have set logging to DEBUG and currently checking the logs for clean but not seeing anything…
Any ideas?
Managed to find the log messages. It seems to see no annotations to delete:
logger=cleanup t=2024-10-09T12:35:10.024832021Z level=debug msg="Deleted excess annotations" annotationsaffected=0 annotationtagsaffected=0
Despite having 84571901 annotations in the system, the majority are alert annotations that are outside of a month in age.
Hitting /api/annotations?from=1691467229559&to=1698467229559&type=alert?orgId=15
Is showing me alerts from October 2023… so they are definitely there and outside of the 1 month period…
Ah… the setting I’m using for alert annotations was added in Grafana 11 and I’m currently on 10… will check with the old setting in [alerting] block
Setting max_annotation_age worked:
[alerting]
max_annotation_age = 1M
1 Like
