Getting "no value" alert although alert error state is "OK" - Issue with PostgreSQL queries

Grafana Alerting
1

Hello,

we have recently upgraded to Grafan 9.3 and now to 9.5.2.
In both versions, 9.3.x and 9.5.2 we see the same behaviour:

  • We repeatedly receive alerts containing [no value] and grafana_state_reason Error
  • We are provisioning all dashboards and alerts using file-configuration, for all alerts we changed the executionErrorState: OK (see snippet below)
  • But we still regularly receive the no-value alerts.

Alert snippet

            dashboardUid: 9jJFBRvmz
            panelId: 86
            noDataState: OK
            executionErrorState: OK
            annotations: 
              ...

alert_state_OK
alert_state_OK1326Ă—1202 72.1 KB

The database is a hosted prometheus and for configuration, we use a hosted PostgreSQL database.
But We expected that when “executionErrorState: OK” is set, these timeouts should be ignored.

But as is, we receive a flood of alerts which contain the no value messages (see screenshot)

no_value_alert
no_value_alert1372Ă—1026 75.4 KB

Since we had issues with the SQLite on the Kubernetes Persistent Volume, we also switched Grafana to a hosted PostgreSQL Database for configuration.

It seems, that not the Query to the Prometheus Database, but instead the Query to update the Alert state failed:

postgres_error
postgres_error2500Ă—256 45.9 KB

a little bit later we see Detected stale state entry
later_logs
later_logs2506Ă—212 51.4 KB

But nonetheless, this should not provoke an alert?!

How can I fix this, that a failed query to PostgreSQL does not trigger alerts?

1 Like
2

Hi! The log line in red looks like a Prometheus query that timed out, rather a SQL query to PostgreSQL? I see /prometheus/api/v1/query_range in the URL.

1 Like
3

Hi @georgerobinson
thank you, it seems I was blind on that eye.
But that does raise another question, why we keep seeing these no-value alerts although we configured

            noDataState: OK
            executionErrorState: OK

That might seem like a bug in Grafana then.
I just switched to another postgreSQL DB, trying to evaluate if that solves the issues.

4

I am facing same issue, I have configured grafana to handle nodata and execution error state although I am still receiving no-value alerts from grafana. Did anybody was able to resolve it?

1 Like