Dashboard alerts error out after update from 8.2.0 to 8.2.4

wkneewalden · November 17, 2021, 3:10pm

The Grafana version tested was v8.2.4. Checking dashboards after update from v8.2.0, found that all alerts were firing. The error was the following:

error:"request handler error: failed to query data: Post "http://***.com/api/v1/query_range": net/http: invalid header field name """

Expected all alerts to not fire. Found that Prometheus data sources needed to be manually replaced with identical versions in the console while using v8.2.4, in order for alerts to work properly. This solution was based on suggestions from here:

github.com/grafana/grafana

Alerts cannot connect to database since 8.1.0

opened 08:02AM - 06 Aug 21 UTC

closed 02:36PM - 19 Aug 21 UTC

jnguiot

datasource/InfluxDB area/backend

## Environment: * Grafana 8.1.0 * Influxdb 1.8.9-1 * Debian 9 ## Initial C…ondition: * Influxdb and Grafana are installed on the same server. * Influxdb has an authentication with a login and a password ## How to reproduce: * Create a dashboard * Create a graph panel. * Input any query related to your data into the query panel (eg `SELECT mean("percent_packet_loss") FROM "ping" WHERE ("url" = 'xxxxx') AND $timeFilter GROUP BY time($__interval) fill(null)` * In Alerting tab of the graph, input a threshold, * Save dashboard * Click on test rule in alert tab ## Observed behavior: * when testing rule, we have this response: ``` firing:true state:"alerting" conditionEvals:" = true" timeMs:"168.163ms" error:"request handler error: failed to query data: InfluxDB returned error status: 401 Unauthorized" logs:Array[1] 0:Object message:"Condition[0]: Query" data:Object from:1628236340096 queries:Array[1] 0:Object to:1628236640096 ``` ## Expected behavior: * no error should be displayed when testing. Alert should be fired if data requires it. ## Misc Please note: * all our alerts have suffered this issue when updating from 8.0.6 to 8.1.0 * no configuration has changed on influxdb, especially authentication * grafana still display data. Only alerting seems broken. * reverting back to 8.0.6 fixes our issue Thank you for Grafana which is an incredible piece of software. I will do my best to help you on this matter.

However, I should be able to use existing data sources without any issues to avoid replacing them after version update due to a large number of organizations/dashboards.

mattabrams · March 15, 2022, 11:43pm

Grafana’s transition from legacy alerting to the Unified Alerting platform represents a big step forward. But, there are many factors that can influence behavior, and it is often hard for the community to troubleshoot issues without a thorough understanding of your unique setup. Try to include the following info:

What is your Grafana version?
Are you using Grafana Cloud or self-hosted Grafana?
Are you using legacy alerting or Unified Alerting?
was the alert in question migrated from the legacy platform into Unified Alerting, or did you first create it inside the new platform?
Please list ALL configuration options related to alerting. You can find these in the Alerting and Unified Alerting sections of Grafana’s config file. If you are now using or have previously used the beta version of ngalert (released with Grafana 8), please note that too.
- you can use this table to better understand how configuration options can interact with each other
If this is a templating issue on Unified Alerting, check if your alert is using a multi-dimensional rule or not.
List the datasource associated with the alert
Increase the verbosity of the Grafana server logs to debug and note any errors. For printing to console, set the console logs to debug as well.
Search for open issues on GitHub that sound similar to your problem

wkneewalden · May 25, 2022, 9:33pm

Just saw the response. The issue was resolved by using a later version 8.3.6. It is still resolved using 8.5.3 and having the new alert system enabled, meaning the alerts tied to panels still appear and work properly alongside any new standalone Grafana alerts.