Detected stale state entry in logs

Grafana Alerting
1

Had a MySQL alert go off. This started a couple of issues:

  1. I have the following expressions:

image
image1224×264 19.6 KB

When it detects over 300 it alerts. I then have alert threshold for this:

{{ if gt $values.C.Value 600.0 }}Critical{{ else }}Warning{{ end }}

This is in my labels and my notification policies. It detected the correct label, however it sent an email to the Default Policy within Grafana and not to the appropriate contacts.

  1. I am seeing this in the logs file:

image
image1892×336 39.2 KB

I would like my thresholds to work correctly and I would like to know what all the stale entry log entries I am seeing. Can anyone help?

2

Hi there,

I’m also experiencing the same issue even though our log line is pretty new.

Grafana version: 9.5.5

image
image3556×348 21.5 KB

{“body”:"logger=ngalert.state.manager rule_uid=c1bd1c51-5163-4451-858a-a2dbdd842618 org_id=1 t=2023-09-27T20:22:33.089700987Z level=info msg="Detected stale state entry"

3

Hi there,

expierecing the same since we migrateted from sqlite (where we had database locked) to postgres. We are currently thinking its the backend source, but we are not realy sure where the issue lies. Have you guys found a workaround?

4

I migrated to MySQL and haven’t had the issue come up again