How to Send Alerts to External Alertmanager

Grafana Alerting
1

Grafana Version: v9.2.4

I have added a Prometheus-Alertmanager datasource. And I can see my alertmanager in Alerting in the Grafana.

image

Then I created an alert with following steps:

  1. In the Grafana menu, click the Alerting (bell) icon to open the Alerting page listing existing alerts.
  2. Click New alert rule. The new alerting rule page opens where the Grafana managed alerts option is selected by default.
    …

After the alert was created and firing, it is sent to the alertmanager of Grafana.
I want my external alertmanager to receive this alert. What should I do?

2

Hi! To send alerts to an external Alertmanager go to the Alerting page, then click on the Admin tab. At the end of the page there is Send alerts to where its possible to change between Internal Alertmanager, External Alertmanager, or both.

1 Like
3

Thank you for the solution. It works!

Is it possible to make this config as code?

4

I’m also searching to configure this as code.

By searching in the grafana code, I found this PR where it has been merged

state.unifiedAlerting.externalAlertmanagers.alertmanagerConfig.result?.alertmanagersChoice
It seems there is a alertmanagersChoice config that can be “external” but I don’t know how to make it work/where it should be located.

5

@elio any update you have? I am also looking for a config option where I can set “alertmanagersChoice” during provisioning. @georgerobinson maybe you know it?

6

+1 with aahmadzada. I cant find this config as code. Im using the grafana helm chart

7

+1 on config as code

8

I was looking for the same configuration option and unfortunately, it is not yet present.
I did implement it via a workaround by adding an extra container that executes Grafana API.

containers:
            # Temporary workaround until grafana  add support of unifiedAlerting.externalAlertmanagers.alertmanagerConfig parameter
            {{- if or (eq .Values.alertmanagerEngine "external") (eq .Values.alertmanagerEngine "all") }}
            - name: setup-alertmanager-engine
              image: {{ .Values.image }}
              imagePullPolicy: {{ .Values.imagePullPolicy }}
              resources:
                limits:
                  memory: 100Mi
                requests:
                  cpu: 10m
                  memory: 100Mi
              env:
                - name: GF_SECURITY_ADMIN_USER
                  valueFrom:
                    secretKeyRef:
                      key: username
                      name: grafana-auth-secrets
                - name: GF_SECURITY_ADMIN_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      key: password
                      name: grafana-auth-secrets
              command:
              - "/bin/sh"
              - "-c"
              - |
                while true; do
                  curl -sSf -X POST -H "Authorization: Basic $(echo -n "$GF_SECURITY_ADMIN_USER:$GF_SECURITY_ADMIN_PASSWORD" | base64 | tr -d '\n')" -H "Content-Type: application/json" -d '{"alertmanagersChoice": "{{ .Values.alertmanagerEngine }}"}' localhost:3000/api/v1/ngalert/admin_config
                  if [ $? -eq 0 ]; then
                    echo ""
                    echo "Alertmanager engine settings validated. Next validation in 5 minutes."
                    sleep 300
                  else
                    echo "Waiting for grafana to be fully initialized."
                    sleep 10
                  fi
                done
            {{- end }}

In reality, the same command can be added via readinessProbe, but I am using grafana operator and due to the bug can’t utilize it.
As for the image I used the same one as grafana container.

Workaround a bit dirty, but it work.