Notification in case of panel errors

[hafeja]

Hey,

I want to get notified in case a panel of my Grafana instance throws an error like that:

A notification shall be created for all kinds of panels.

The notification channel can be email, RSS feed, HTTP post or something else.

As far as I read the docs, “Grafana Alerting” is not capable of doing this.

Any ideas how to setup such a notification?

Environment: Grafana v7.5 running inside Docker and soon v8.

[mattabrams]

Hi @hafeja

You could add your grafana-server logs to Loki, and then alert on loki queries that search for specific errors. If I understand your intent, then you’ll need to get the grafana-server logs into Grafana via some datasource. And if so, Loki makes the most sense to me.

Get logs into Loki

[hafeja]

Hi @mattabrams ,

thanks for your reply. I’ll keep “Loki” in mind for more complex use cases.

For the time being, it is “too much” for me to set up Loki and to create a further dashboard just to get Grafana error notifications. I was looking for a simpler, more leight-weigth solution.

Meanwhile, I solved it by setting up a cronjob/Jenkins-job which runs e.g. hourly and executes a docker logs <grafana-container> --since 1h. In case it contains a string like lvl=eror, I send a email notification.