Contact Points Missing in Notification Policy UI (Grafana in GKE via kube-prometheus-stack)

Grafana Alerting
1

Hello everyone,

I’m currently facing a puzzling issue with Grafana’s alerting configuration and I’d really appreciate your insight.

:brick: Deployment Setup:

  • Platform: Google Kubernetes Engine (GKE)

  • Deployment Method: Helm

  • Chart: kube-prometheus-stack

  • Chart Source: grafana/helm-charts

  • Chart Version: Latest (as of the date of this post)

  • Tooling: Using HelmRelease from FluxCD to manage deployment

  • Grafana SMTP config: Provided via grafana.ini in values.yaml as documented in official Grafana sources.

:white_check_mark: What Works:

  • I successfully configured SMTP (via grafana.ini) and restarted the Grafana pod to apply the changes.

  • Created an Email contact point via the Grafana UI.

  • Used the “Send Test” function — the test message was delivered successfully to my mailbox.

  • This confirms that:

    • SMTP config is correct,

    • Email contact point is valid,

    • Outbound connectivity is fine.

:cross_mark: What’s Broken:

After configuring SMTP and restarting Grafana, I tried to create a Notification Policy via the UI. However:

  • When clicking “Add child policy”, the Contact Points dropdown is empty.

  • This is unexpected — we already have existing contact points:

    • Slack

    • Microsoft Teams

    • Several Email contact points

  • Additionally, I created a new alert rule as a test, and in Step 5 (Configure notifications), no contact points are listed either.

Previously (as of last week), all contact points were shown in both dropdowns. Today — nothing.

:test_tube: What I’ve Tried:

  1. Reverted changes: Removed SMTP config and Email contact points → restarted pod → issue persists.

  2. Rollback: Downgraded to the last 5 chart versions → no effect.

  3. Restored config: Re-applied SMTP config and re-created Email contact points → still no contact points listed.

  4. Checked logs: No errors or warnings in Grafana logs (stdout or container logs).

  5. Validation: Existing alert rules still work and are correctly sending to Teams and Slack contact points that were created and associated before this issue began.

:light_bulb: Questions:

  • Could this be a UI regression or bug in the latest version of Grafana or the chart?

  • Is there any known caching or internal state management that might need to be cleared?

  • Could the updated contact points be stored in a different secret/configmap that isn’t being reloaded properly?

  • Any other debugging steps or configuration refreshes I could try?

:wrench: Additional Info:

  • Grafana is running as part of the kube-prometheus-stack Helm release — no external DB (default SQLite).

  • No RBAC restrictions — I have full admin access via the Grafana UI.

  • Contact points were all created via UI, not provisioned via YAML files.

2

That source doesn’t contain that chart.

First specify Grafana version (not helm chart version, “latest” is not version). Clean browser cache (e.g. CRTL+F5) - especially when you are mixing with different Grafana versions. Use browser console and verify that requests to API to list contact points are not blocked or which kind of error response are they returning.

3

Hi, Grafana v12.1.0 (ccd7b6ce7e) using latest helm chart version, sorry I should have specified that yep.
I tried to login to grafana’s UI through the incognito window, tried cleaning browser cache it didn’t work.
Just to be sure, I used 3 different browsers (Edge, Chrome, Brave) and issue persists.
When verifying API requests as you suggested: (I am not BE developer .. DevOps)
In receivers API call return’s 200 listing contact points. But as you see on the picture UI shows no contact points but API call returns them :confused:

image
image1899×928 85 KB

image
image1920×748 79.8 KB

[
    {
        "active": true,
        "integrations": [
            {
                "lastNotifyAttempt": "0001-01-01T00:00:00.000Z",
                "lastNotifyAttemptDuration": "0s",
                "name": "email",
                "sendResolved": true
            }
        ],
        "name": "Mail-ContactPoint"
    },
    {
        "active": true,
        "integrations": [
            {
                "lastNotifyAttempt": "0001-01-01T00:00:00.000Z",
                "lastNotifyAttemptDuration": "0s",
                "name": "email",
                "sendResolved": true
            }
        ],
        "name": "grafana-default-email"
    },
    {
        "active": true,
        "integrations": [
            {
                "lastNotifyAttempt": "0001-01-01T00:00:00.000Z",
                "lastNotifyAttemptDuration": "0s",
                "name": "teams",
                "sendResolved": true
            }
        ],
        "name": "MS-Teams-Contact-point"
    },
    {
        "active": true,
        "integrations": [
            {
                "lastNotifyAttempt": "0001-01-01T00:00:00.000Z",
                "lastNotifyAttemptDuration": "0s",
                "name": "email",
                "sendResolved": true
            }
        ],
        "name": "CEM-Mail-Group"
    }
]
4
5

Hi, Any update on this?

6

Please read/watch linked GitHub issue