Monitor a Grafana datasource against data becoming stale to detect data loss or other dropout conditions

Grafana
1

Dear Grafana Community,

we always wanted to share a little tool we are running to check data staleness of various data acquisition channels we are feeding into InfluxDB within the Hiveeyes Project.

It observes overall data freshness through the whole data acquisition machinery by actually looking at the data source endpoint exposed by Grafana itself and checking the age of the most recent visible data point against a configurable staleness threshold.

This detects data loss conditions of any kind, either if data submission stops or if any part of the infrastructure breaks.

With kind regards,
Andreas.

P.S.: This tool has originally been conceived before Grafana received any alerting feature. We will be happy to learn how similar things might be able to be achieved natively in the meanwhile.

2

Saying that, doing it natively inside Grafana would possibly not yield the same level of confidence. As we might guess, alerting for data loss in Grafana probably sits on top of a properly configured and working data source. The external sensor on the other hand will monitor the infrastructure as a whole, which might be called black box monitoring.

Nevertheless, we would be interested what can be done within Grafana itself to let the user add notifications for individual acquisition channels on its own behalf. A quick forum search yielded some topics where people have been discussing similar things.