Understanding the consquences of alertingSaveStatePeriodic

Hi,

The docs are pretty sparse on information, I enabled the feature flag and saw the load come down on our database (perfect!), I then created an alert to trigger straight away and saw the annotation on the dashboard and history was immediately available, so what are the downsides to enabling this feature flag?

If Grafana crashes, or is force killed, then the last 5 minutes (unless using a different value for state_periodic_save_interval) of state data can be lost. This does not affect annotations or Loki for alert state history.

What this means is when Grafana starts up the UI may show incorrect state for some alerts until they are evaluated again. This also means that alerts that were firing before the crash can fire again. When this happens you may receive the notification again too.

1 Like

Perfect, thanks @georgerobinson

We just added this to the docs too!

1 Like