Enable unified / high availability alerting problem

Grafana Alerting
, , ,
1

Hi!

I would like to use Grafana in High-Availability mode. I installed it on two separate machines and configured them to use the same remote database. This works perfectly.

The hosts:
grafana-node1 - 192.168.14.31
grafana-node2 - 192.168.14.32

OS: Almalinux 8.7
Grafana: 9.3.0

Both nodes names and IP addresses are in the /etc/hosts files.
The network devices name is ens192 and the IPV6 is disabled.

I would also like to set up high availability (unified) alerting, based on this:

This is how I modified the [unified_alerting] section in the /etc/grafana/grafana.ini file:

enabled = true ha_peers=192.168.14.31:9094,192.168.14.32:9094

After I restarted the grafana-server.service on both nodes, they don’t start. The logs show this:

logger=settings t=2022-12-01T17:39:40.708484842+01:00 level=info msg="Starting Grafana" version=9.3.0 commit=e9cb2a313e branch=HEAD compiled=2022-11-29T17:03:52+01:00 logger=settings t=2022-12-01T17:39:40.708739951+01:00 level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini logger=settings t=2022-12-01T17:39:40.708788777+01:00 level=info msg="Config loaded from" file=/etc/grafana/grafana.ini logger=settings t=2022-12-01T17:39:40.708814558+01:00 level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" logger=settings t=2022-12-01T17:39:40.708838387+01:00 level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" logger=settings t=2022-12-01T17:39:40.708864332+01:00 level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" logger=settings t=2022-12-01T17:39:40.708893657+01:00 level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" logger=settings t=2022-12-01T17:39:40.708920477+01:00 level=info msg="Path Home" path=/usr/share/grafana logger=settings t=2022-12-01T17:39:40.708945747+01:00 level=info msg="Path Data" path=/var/lib/grafana logger=settings t=2022-12-01T17:39:40.708969656+01:00 level=info msg="Path Logs" path=/var/log/grafana logger=settings t=2022-12-01T17:39:40.708993471+01:00 level=info msg="Path Plugins" path=/var/lib/grafana/plugins logger=settings t=2022-12-01T17:39:40.709026793+01:00 level=info msg="Path Provisioning" path=/etc/grafana/provisioning logger=settings t=2022-12-01T17:39:40.709059036+01:00 level=info msg="App mode production" logger=sqlstore t=2022-12-01T17:39:40.709144102+01:00 level=info msg="Connecting to DB" dbtype=postgres logger=migrator t=2022-12-01T17:39:40.748527558+01:00 level=info msg="Starting DB migrations" logger=migrator t=2022-12-01T17:39:40.755413175+01:00 level=info msg="migrations completed" performed=0 skipped=464 duration=701.135µs logger=plugin.loader t=2022-12-01T17:39:40.846841639+01:00 level=info msg="Plugin registered" pluginID=input logger=plugin.finder t=2022-12-01T17:39:40.846936511+01:00 level=warn msg="Skipping finding plugins as directory does not exist" path=/var/lib/grafana/plugins logger=secrets t=2022-12-01T17:39:40.847270886+01:00 level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 logger=query_data t=2022-12-01T17:39:40.850291928+01:00 level=info msg="Query Service initialization" logger=live.push_http t=2022-12-01T17:39:40.854378736+01:00 level=info msg="Live Push Gateway initialization" logger=ngalert.multiorg.alertmanager component=cluster t=2022-12-01T17:39:40.854611924+01:00 **level=warn err="couldn't deduce an advertise address: failed to get private IP: route ip+net: netlinkrib: address family not supported by protocol"**

What is the problem? What can I do?

2

This error sounds like you’re missing the setting ha_advertise_address

I’m looking at the docs and it seems to be missing in the ones you referenced, even though it is shown in the kubernetes example in step 5:

[unified_alerting]
enabled = true
ha_listen_address = "${POD_IP}:9094"
ha_peers = "grafana-alerting.grafana:9094"
ha_advertise_address = "${POD_IP}:9094"

I’ve forwarded this feedback to our docs team. Let me know if that was the setting you needed to get things working.

3

Dear Melori,

You’re right.
I set up these two lines on both Grafana nodes:

ha_listen_address = "0.0.0.0:9094"
ha_advertise_address = "0.0.0.0:9094"

And Grafana started on both nodes, there is no trace of the previous error message. I see that the alert manager starts up nicely about 20 seconds after starting Grafana:

logger=ngalert.multiorg.alertmanager component=cluster t=2023-01-02T14:25:14.705646445+01:00 level=info msg="gossip not settled" polls=0 before=0 now=3 elapsed=2.000836297s
logger=ngalert.multiorg.alertmanager component=cluster t=2023-01-02T14:25:18.70586677+01:00 level=info msg="gossip not settled" polls=2 before=3 now=1 elapsed=6.001057536s
logger=ngalert.multiorg.alertmanager component=cluster t=2023-01-02T14:25:24.708080941+01:00 level=info msg="gossip not settled" polls=5 before=1 now=2 elapsed=12.003269418s
logger=ngalert.multiorg.alertmanager component=cluster t=2023-01-02T14:25:32.709637536+01:00 level=info msg="gossip settled; proceeding" elapsed=20.004829172s

Thank you very much for your help!
I would never have realized this by myself, because in the comments of both parameters in the configuration file it is written that:

The default value is `0.0.0.0:9094`

Maybe they don’t take the default value? After all, I gave the same ones, like this:

# Listen address/hostname and port to receive unified alerting messages for other Grafana instances. The port is used for both TCP and UDP. It is assumed other Grafana instances are also running on the same port. The default value is `0.0.0.0:9094`.
;ha_listen_address = "0.0.0.0:9094"
ha_listen_address = "0.0.0.0:9094"

# Listen address/hostname and port to receive unified alerting messages for other Grafana instances. The port is used for both TCP and UDP. It is assumed other Grafana instances are also running on the same port. The default value is `0.0.0.0:9094`.
;ha_advertise_address = ""
ha_advertise_address = "0.0.0.0:9094"

Once again, thank you very much for your help!
And I almost forgot the most important thing: I wish you a new year rich in health, happiness, love and wealth! :slight_smile: