Alert Contact SMTP Timeouts

Grafana Alerting
1
  • Grafana v12.1.1 (df5de8219b)
  • Red Hat Enterprise Linux 8.10 (Ootpa)

Hello,

I am trying to switch alerting emails to a new secure company SMTP server, but when trying to send a test email for the contact point, I am getting a popup saying:

“Failed to send test alert.: failed to send email: read tcp
<unix host ip>:<port> → <smtp server ip>:587: i/o timeout”.

I enabled debug level logging, but do not see anything helpful
or related to the smtp connection attempts in the grafana log file.

There are a couple of older posts with similar issues here in the forums,
but no helpful solutions I could see.

I am suspecting some sort of TLS or authentication issue.

When using the option
startTLS_policy = NoStartTLS
the error box reads “proper authentication required”.

Here the configuration file smtp section I am using,
I tried all the options for startTLS_policy, to no avail.
setting skip_verify to false did not seem to make any difference:

[smtp]
enabled = true
host = securesmtp.mycompany.com:587
user = MailUser
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
password = "MailUserPassword"
cert_file =
key_file =
skip_verify = true
from_address = MailUser@mycompany.com
from_name = Grafana
ehlo_identity =
startTLS_policy = MandatoryStartTLS # --> timeout
#startTLS_policy = OpportunisticStartTLS --> timeout
#startTLS_policy = NoStartTLS # --> proper authentication required
#startTLS_policy = empty # --> timeout
enable_tracing = false

Using the deprecated unsecured company smtp host on port 25
without any necessary authentication and TLS/SSL,
emails are properly sent out.

On the same Unix host, with mutt and the configuration below in .muttrc I am able to send emails via that SMTP server no problem; so not a firewall/network problem:

set smtp_url = "smtp://MailUser:MailUserPassword@securesmtp.mycompany.com:587"
set ssl_force_tls = yes
set ssl_starttls = yes
set from = "MailUser@mycompany.com"
set use_from = yes

The requirements I was given to send emails via the server are as follows:

SMTP Server: securesmtp.mycompany.com
Port: 587
Sender: Valid Mail User Address
TLS: Required

Any hints as to where my problem is, or how to log the connection
attempts between grafana and the smtp server?

Thanks!

Jens

2

That indicates a problem with TCP connectivity. Firewall, sec. groups, routing,… ask your networking team.

3

tcpdumps for the Grafana connection attempts that are timing out show this,
the SMTP server seems to be not responding to the TLSv1 connection handshake:

image
image1920×613 102 KB

The SMTP relay is a ProofPoint relay, OS version 8.18.6, internally using sendmail.

Using mutt, on the same server and as the same user Grafana is running as, emails can be sent successfully.
mutt is sending a TLSv1.2 handshake, though, below.
(Note: The ProofPoint SMTP relays are on a round-robin DNS, hence the different IP address compared to above):

image
image1765×461 142 KB

Reading RFC 5246, and specifically appendix E (RFC 5246 - The Transport Layer Security (TLS) Protocol Version 1.2) sending a TLSv1 handshake is valid, and another version can subsequently be negotiated.
Our infrastructure team cannot tell me why the relay does not respond to the handshake.

That leaves me with these questions:

  1. Is it possible to configure Grafana to send a TLSv1.2 handshake?

  2. Our infrastructure team suggests configuring Grafana with the root and intermediate
    certificates of the SMTP relay. I do not see any option to do that, and this comment hints:
    cert_file config for SMTP does not provide reasonable errors · Issue #66263 · grafana/grafana · GitHub

    image
    image1026×380 126 KB

    Where is Grafana looking for these keystores? Is the path configurable, or hardcoded to /etc/ssl/certs/ per a later comment there?

Thanks!

SMTP TLS v1 Handshake Not Working - v1.2 Option?
4

That depends on Golang: