We have a datasource that can go into a “degraded” state. What we’d like to do is have panels that query that datasource display a warning message that the results might be degraded, perhaps similar to the way a warning is displayed when the query returns an error.
Is there any way to achieve this? I’m imagining something like a special http header in the response that grafana would turn into the notification and warning.
Thanks!
We use Stat panels to display OK or ALARM whenever the difference between the Actual and Setpoint value exceeds 0.05. See screenshot below.
but that must be based on the results of the actual query right? in this case, the query returns results, but they may not be complete, due to the degradation on the server side. this cannot be computed from the returned query data. when we’re showing the query results we’d like to additionally annotate that they may be degraded.
OK, so just to be clear, the degraded state is not known and cannot be computed from the query results. If the results returned by the query are “not complete” (whatever that means) due to degradation, you want a message appearing showing that it’s degraded.
If the above is correct, I think one would need to define what makes the data “not complete”, and thus what defines a degraded state. Maybe some examples of a regular and degraded state will help the forum understand?
we are using m3 (https://m3db.io/) which can be configured to query multiple backend database clusters at a single endpoint. (see Configuration file | M3 Documentation it’s not a great reference, but i don’t think they have better docs on it)
however, sometimes one of those backend clusters may be down/unreachable. we’d like to be able to indicate that the returned data does not include data from that cluster (or even just that the data may be incomplete)
Is the number of backend clusters a fixed amount that is known at the time of query?
it could be. are you imagining some way to return the number of clusters that was successfully queried in the result? i’m not clear how that could be done since the queries are for totally orthogonal data, but perhaps your idea is a different one 
