Grafana Alerting HA Notification Deduplication Implementation

Grafana Alerting
, ,
1

Hi Grafana Community :wave:,

I am exploring HA notification deduplication (memberlist, Redis) and looking for guidance on the following questions:

  1. What message content is passed between HA cluster peers? I ran the Redis HA docker-compose test setup and observed a timestamp published by each peer, incremented at 5s intervals (e.g. 1708994146). Curious if I’m missing something, or if that’s the only data required to sync the peers and dedupe the notifications.
  2. Where is the code implementation which deduplicates the notification? Is there a leader elected somewhere? I understand the mechanism for building and syncing the peer list, however I can’t seem to locate the code which actually deduplicates the notification.

Thanks!
Kyle

2

Update Mar 1 2024

I believe I’ve narrowed down the notification deduplication to usage of nflog in the grafana/alerting package here:

Package nflog implements a garbage-collected and snapshottable append-only log of active/resolved notifications. Each log entry stores the active/resolved state, the notified receiver, and a hash digest of the notification’s identifying contents. The log can be queried along different parameters.

Listing the redis channels we can see jp-test:notificationlog:1:

127.0.0.1:6379> pubsub channels *
1) "jp-test:silences:1"
2) "jp-test:full_state:request"
3) "jp-test:full_state"
4) "jp-test:notificationlog:1"

However subscribing to this channel, I do not see any messages come though. Only pattern subscribing to all * do I receive a message:

psubscribe *

Each time an alert is sent, I can see a published message

1) "pmessage"
2) "*"
3) "jp-test:full_state"
4) "\n\x13\n\x11notificationlog:1\n\x0c\n\nsilences:1"

Question

How is the notification log synced between HA peers?