How to configure alerts at grafana graph panels?

Grafana Configuration
1

Hi,
We have configured grafana.conf a /etc/grafana/ location at following sections:

[smtp]
enabled = true
host = localhost:25
user = my_gmail_id
# If the password contains # or ; you have to wrap it with trippel quotes. Ex """#password;"""
password = *********** - my gmail password
;cert_file =
;key_file =
;skip_verify = false
;from_address = admin@grafana.localhost
;from_name = Grafana

[emails]
welcome_email_on_sign_up = true
  • At grafana graph panels, we configured like below:

image
image763Ă—312 19.7 KB

  • When we click test rule, it shows like:

    image
    image710Ă—353 25 KB

  • At notification channels, we configured our email id.

  • We are not getting alert mails.

  • When we click send test it is showing Failed to send alert notifications.

  • Did I miss anything configuring? If yes, please help us to get alerts to our mails.

Thank you.

2

Hi,

Please check grafana server log for error details. If using Gmail and 2 factor Auth you need to generate specific password in Google accounts and use that as password in Grafana.

Marcus

3

Hi @mefraimsson

  • I am using gmail as my email-id. I am using my gmail account password in grafana.ini conf file.
  • I checked grafana logs, there I found the following logs when I checked with Test Rule:
t=2018-04-11T16:14:34+0200 lvl=info msg="Sending alert notification to" logger=alerting.notifier.email addresses=["my_email-id"]
t=2018-04-11T16:14:34+0200 lvl=eror msg="Failed to send alert notification email" logger=alerting.notifier.email error="535 5.7.0 authenticatio
n failed"
t=2018-04-11T16:14:34+0200 lvl=eror msg="Failed to send alert notifications" logger=context userId=1 orgId=1 uname=admin error="535 5.7.0 authe
ntication failed"
t=2018-04-11T16:14:34+0200 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/alert-notificatio
ns/test status=500 remote_addr=127.0.0.1 time_ms=6 size=48 referer=http://<my host>/alerting/notification/1/edit
  • Could you please help us with this issue ?.

Thank you!!!

4

Do you have 2 factor authentication activated for your gmail account?

5

No, I don’t have 2-factor authentication. Just password login based.
Thank you!!!

6

Hi @mefraimsson

  • I want to update some more details:
  • I am running grafana in linux server.
  • At root location, I have created a .mailrc file and added the below lines:
set smtp-use-starttls
set ssl-verify=ignore
set smtp=smtp://<some other email>:<some other port>
set smtp-auth=login
set smtp-auth-user='<auth-user>'
set smtp-auth-password='<auth password>'
set from="<from email addr>"
  • Will this details help to resolve this issue ?.

Question:

  • Do we need to configure the same specifications in grafana.ini and in .mailrc?

If not,
What are the ways to acheive the solution to resolve this?
Thank you!!!

7

I’ve never used a .mailrc configuration. Only configured Grafana’s smtp settings. Please refer to comment in Grafana email alerting error

8

Thanks a lot!!!
It worked for me…

Thank you very much @mefraimsson!!!

9

Hi @mefraimsson,

  • In my graph panel, I have 160+ servers CPU usage metric.

  • I have configured like below:
    image

  • But I got the mail in the following format:
    image

  • The alerts which I got in the mail are not clear to me.

  • I want to get mail like:

    • Which server is consuming more CPU usage. If 5 servers are taking more CPU usage, I should get the name of those 5 servers in the mail.
    • Whenever I got more CPU usage, I need to get email ASAP.
    • I am getting my-email-id as from and to mail. I need to get from mail as grafana.
      Could you please help me with this issue?.
      Thank you!!!
10

Hi,

That looks like a test email. Please read up on alerting and multiple series so that you’re aware of that there may be some limitations currently: http://docs.grafana.org/alerting/rules/#multiple-series

In general included panel screenshot in notification should show which servers are affected by an alert.

Marcus

11

Hi @mefraimsson,
Yeah, I tried, I got a mail. Thanks for the help!!!. But I am having some questions:

  • How can I get an email whenever the state goes to ALERT state.
  • I am getting my-email-id as from and to mail. I need to get from mail as grafana.
  • In the email, I got only 1 server, but we have more than 5 servers, which is occupying more CPU usage.

Could you please help me with these issues?
Thank you…

12

Hi,

No really sure what you mean. Maybe a screenshot of an email for a real alert would help here.

Marcus

13

Hi @mefraimsson,

image
image861Ă—503 71.9 KB

  • If we see the above image, we are getting only metric related to one host.
  • At that time, we are having more than 1 source, which is having high usage. Why those servers didn’t come in the email.
  • I need to get all the hosts which are having more CPU usages in the email ?.
  • Also I am seeing CPU alert(below image) for past 15 min but I didn’t get any mail. Why?
    image
    image1028Ă—174 13 KB
  • When I get a mail, there I am getting “FROM ADDRESS” as my-email-id, I need that “FROM ADDRESS” should come from grafana-server. How to configure that ?.

Please let me know if you didn’t understand the information.
Thank you…

14

As I wrote earlier and linked to documentation clearly states:

If a query returns multiple series then the aggregation function and threshold check will be evaluated for each series. What Grafana does not do currently is track alert rule state per series. This has implications that are detailed in the scenario below.

  • Alert condition with query that returns 2 series: server1 and server2
  • server1 series cause the alert rule to fire and switch to state Alerting
  • Notifications are sent out with message: load peaking (server1)
  • In a subsequence evaluation of the same alert rule the server2 series also cause the alert rule to fire
  • No new notifications are sent as the alert rule is already in state Alerting.

There are an open feature request issue for supporting what you want to have. Please upvote that by reacting with :+1: to the issue description - please do not add a new comment since that will trigger a notification to all the people who have subscribed to that issue.

See documentation for configuring smtp.

Marcus

2 Likes
15

Hey there,

We want to generate a mail for 100+ servers which are consuming more than 80% of CPU & RAM but we are getting mail regarding last one server on the dashboard. Can you please help us. Thank you!

16

Hi,

I tried to make it clear that we don’t support what you’re asking for. Therefore I referenced the open feature request issue. I cannot help you further regarding the alert emails.

Marcus

17

Hi @mefraimsson

We have created different ypes of dashboards, how can we show particular dashboard for certain user. Could you please help us. Thanks!

18

This topic now have a solution. Please create a new topic for specific questions. Thanks

19

@mefraimsson

Can you please check it once, already we have created a new topic. Thanks!