Clearing down over 3 million annotations

Grafana
,
1

Hi,
I user added a couple of alert rules that had 20,000 servers which were flapping for nearly 3 weeks. This has caused our annotation table to grow to 3 million rows, the database load is high due to the housekeeping, we’ve got batch size to 1000 but we’re hitting the timeout now and then, and obviously will take forever to clear.

Just wondering if it’s safe to truncate the table instead?

2

Yes, you can truncate the annotation table, but there is one important condition.

TRUNCATE will remove all historical annotation data from the table.

  • It will not delete alert rules.
  • It will not delete your servers.
  • It will not delete your Grafana dashboards.
  • However, all existing annotation history will be permanently removed.

Also, make sure the flapping alerts are fixed or disabled first. Otherwise, the alerts will continue generating new annotations:

3

Cheers @infofcc3 - great, that’s perfect

4

Just to update the thread with my steps.

  1. Scaled down the service to one node
  2. Stop grafana-server
  3. Connect to the database and executed the following:
    • TRUNCATE TABLE annotation_tag;
    • TRUNCATE TABLE annotation;
  4. Start grafana-server
  5. Scale service back to BAU