Debugging an Alert - 400 Bad Request, Data/Graph is fine

Grafana Time Series Panel
1

I’m busily configuring graphs and alerts for an Exchange server using Grafana. CPU, memory and other performance alerts are fine on this and other graphs; but the Exchange server is “different”. Instead of drive letters many of the volumes have paths instead. For example, traditionally you might have "C:", "D:" and "E:"; on this server I have "C:", “C:\Exchange\ExDB1” and “C:\Exchange\ExDB2”.

I already worked out that you need to use a double-backslash in the Telegraf configuration and the data is being stored.

image
image895×341 22.9 KB

The metrics are individual queries - all have the same structure, so here’s just one example:

image
image813×214 6.99 KB

[See first response to this Q for the remainder of the question - limited to 2 graphics as a new user].

2

The alert is defined based on a completely normal avg() query:

image

Which fails dismally:

image
image875×270 20.7 KB

I think it’s getting hung up on the instance names, which contain the backslash - but I have no clue how to test that or what I can do as a workaround. Suggestions welcome.

3

Gosh - no takers? No ideas on how I can investigate further?

4

I would guess that your thinking is correct and that the backslash in C:\ is not being escaped as http status code 400 indicates that the request is malformed.

The query for alerts is executed on the backend so any errors would appear in the Grafana server logs. Grafana currently has duplicate logic for executing queries on the frontend and backend so this could be a bug in the backend code. Have you tried escaping the slashes in the query (with a double backslash)?

Noticing that there are a lot of open issues in the InfluxDB repo about escaping backslashes but if it works on the frontend then it should work on the backend too.

5

Thanks to @daniellee and @bergquist for this fix which should land in Grafana 5.0.

1 Like