Upgrade 10.2.3 issues

Grafana Developers & API
1

  • What Grafana version and what operating system are you using?
    Grafana 10.2.3, Ubuntu 16.04.3 LTS

  • What are you trying to achieve?
    I just upgraded to Grafana 10.2.3 but I am unable to pull relevant alert information using the API call. Before upgrading to 10.2.3, I was using the API keys to successfully pull the real time alert data. But after the upgrade, I am getting stale information. For example, the information I am getting back using the API call shows that the alert is “Paused”, however, in reality, the alert is in “ok” state.

  • How are you trying to achieve it?
    curl -H “Authorization: Bearer secretpassword1234=” http://10.6.1.1:3000/api/alerts | jq ‘.’

  • What happened?
    I get the information back but it seems like it is stale(old information)

  • What did you expect to happen?
    I expected to get real time alert data.

  • Can you copy/paste the configuration(s) that you are having problems with?
    curl -H “Authorization: Bearer secretpassword1234=” http://10.64.111.179:3000/api/alerts | jq ‘.’

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    No

  • Did you follow any online instructions? If so, what is the URL?
    No

Thanks in advance for the help!

2

Hi @hah2930,

Thanks for opening this post.

Let me also try to reproduce this on my machine and will get back to you.

3

Hi Usman,

Thank you very much for looking into this!

Regards.

5

Hi @usman.ahmad ,

If I change the alert name on the UI and compare that with output with the using the API call, I see the discrepancy.

Following is the screenshot after I updated the alert name from “Internal P0” to “Internal Alert P0”

image
image1040×88 6.9 KB

Following is the API call output:

 "id": 172,
    "dashboardId": 4,
    "dashboardUid": "SiFv1M6zz",
    "dashboardSlug": "bandwidth-general",
    "panelId": 63,
    "name": "[Internal P0] Core switch CPU > 45%",
    "state": "ok",
    "newStateDate": "2024-01-10T15:43:32-05:00",
    "evalDate": "0001-01-01T00:00:00Z",
    "evalData": {},
    "executionError": "",
    "url": "/d/SiFv1M6zz/bandwidth-general"

As you can see, the name of the alert in the screenshot is “Internal Alert P0”, whereas, on the API call output, it still shows as “Internal P0”

The bigger issue that I am seeing is that in the alarm condition, the alert state on the API call output does not get updated whereas the UI shows that the alert is being fired.

Do we know if this is a software bug? I am interested in finding out since I was using the API output and integrating that with another system, but ever since upgrading to Grafana 10.2.3, I seem to have broken that functionality.

Any help on this would be highly appreciated.

Thank you.

6

Hi @hah2930

I’m afraid that the API endpoint that you are using is deprecated since v.9 was available.

Currently there is no APi in the documentation that will return the current state of the alert. However there are a couple alternatives. They aren’t pretty but that’s all I could find. Otherwise, feel free to submit a feature request.

/api/prometheus/grafana/api/v1/rules/

This endpoint will return the state of each (Grafana-managed only - I think ) alert

/api/annotations?alertId={id}

And this other endpoint will tell you what is the current state (see newState) based on the annotations

I tested the current -supported - API

/api/v1/provisioning/alert-rules/{UID}

And info such as title matches the title of the alert rule in the UI