How to Send Alerts to External Alertmanager

Grafana Version: v9.2.4

I have added a Prometheus-Alertmanager datasource. And I can see my alertmanager in Alerting in the Grafana.


Then I created an alert with following steps:

  1. In the Grafana menu, click the Alerting (bell) icon to open the Alerting page listing existing alerts.
  2. Click New alert rule. The new alerting rule page opens where the Grafana managed alerts option is selected by default.

After the alert was created and firing, it is sent to the alertmanager of Grafana.
I want my external alertmanager to receive this alert. What should I do?

Hi! To send alerts to an external Alertmanager go to the Alerting page, then click on the Admin tab. At the end of the page there is Send alerts to where its possible to change between Internal Alertmanager, External Alertmanager, or both.

1 Like

Thank you for the solution. It works!

Is it possible to make this config as code?

I’m also searching to configure this as code.

By searching in the grafana code, I found this PR where it has been merged

It seems there is a alertmanagersChoice config that can be “external” but I don’t know how to make it work/where it should be located.

@elio any update you have? I am also looking for a config option where I can set “alertmanagersChoice” during provisioning. @georgerobinson maybe you know it?

+1 with aahmadzada. I cant find this config as code. Im using the grafana helm chart

+1 on config as code

I was looking for the same configuration option and unfortunately, it is not yet present.
I did implement it via a workaround by adding an extra container that executes Grafana API.

            # Temporary workaround until grafana  add support of unifiedAlerting.externalAlertmanagers.alertmanagerConfig parameter
            {{- if or (eq .Values.alertmanagerEngine "external") (eq .Values.alertmanagerEngine "all") }}
            - name: setup-alertmanager-engine
              image: {{ .Values.image }}
              imagePullPolicy: {{ .Values.imagePullPolicy }}
                  memory: 100Mi
                  cpu: 10m
                  memory: 100Mi
                - name: GF_SECURITY_ADMIN_USER
                      key: username
                      name: grafana-auth-secrets
                - name: GF_SECURITY_ADMIN_PASSWORD
                      key: password
                      name: grafana-auth-secrets
              - "/bin/sh"
              - "-c"
              - |
                while true; do
                  curl -sSf -X POST -H "Authorization: Basic $(echo -n "$GF_SECURITY_ADMIN_USER:$GF_SECURITY_ADMIN_PASSWORD" | base64 | tr -d '\n')" -H "Content-Type: application/json" -d '{"alertmanagersChoice": "{{ .Values.alertmanagerEngine }}"}' localhost:3000/api/v1/ngalert/admin_config
                  if [ $? -eq 0 ]; then
                    echo ""
                    echo "Alertmanager engine settings validated. Next validation in 5 minutes."
                    sleep 300
                    echo "Waiting for grafana to be fully initialized."
                    sleep 10
            {{- end }}

In reality, the same command can be added via readinessProbe, but I am using grafana operator and due to the bug can’t utilize it.
As for the image I used the same one as grafana container.

Workaround a bit dirty, but it work.