There is a lot of dashboards and we want to monitor each widget. Is widget working as expected or there is a problem like “Data is missing a time field” ? Dashboards are configured well but sometimes plugin updates or something broke widget and we need to investigate it. If we can setup any alerting for this usecase we can get response from grafana first not from users. That would be improve our experience.
I have single “check” dashboard where I used:
1.) all datasources used in the dashboards (or at least datasource types) to query some basic data
2.) all widget types used in the dashboards with some basic queries
This dashboard is my quick human view if something is broken seriously after Grafana/plugin upgrades (of course my plugin versions are pinned = plugin upgrades are managed).
Unfortunately, you can’t alert on many errors (e.g. plugin error after upgrade) only on signals in the Grafana. But you can still build something custom (based on Selenium, K6 browser, …), what will try to render all dashboards in the browser and check if everything is OK
Thanks jangaraj
This solution works but a lot of dashboard available and we cant keep one big master dashbord check for quick human check. Also I dont want to use human for this simple taks can be done with monitoring tools. I was thinking about monitoring tool solution but we need something doing client parsing to catch the error message.Simple http body check not works. Built in solution would be great. We can trigger alert mechanisim. Maybe Grafana Advisor add support for us
That’s not simple task. You can simply check API response, e.g. if datasource is responding. But if you want to check if plugin code render response properly, then you need to execute that plugin (JS) code. Your simple http body check will be OK, because it is not problem to receive broken plugin JS code, but user will be complaining that plugin is broken.
You must to use RUM - it will be “monitoring code” running in the user browser (but then it depends in the implementation what it can catch). E.g. Faro
As I said - render dashboard/panel (as you did here to generate screenshot) and check result. There is no easy way. Millions panels = millions renders + check it periodically = complex task.
Yes, that would be good if we can track this widgets with built in functions or advisor plugin. Creating alert will be best solution for this case.