Unable to fetch alert rule by its uid via http api when alert rule is created from UI. Same works for a migrated alert from previous grafana version somehow(after upgrade).

Hi , I have recently created an alert rule(grafana 9.1.4) for a panel(in a dashboard) manually from UI. Now I have done the following to fetch the alert rule using HTTP API.
Used /api/prometheus/grafana/api/v1/alerts HTTP API to fetch “all” the alert rules(had to use this API due to this issue). Below is the snippet of the alert config from the output that i have configured manually as i mentioned above.

                "labels": {
                    "alertname": "canada-shards",
                    "grafana_folder": "mongo-volume-iops",
                    "rule_uid": "canada_mongo_disk"
                "annotations": {
                    "__dashboardUid__": "Kh7igLA4z",
                    "__panelId__": "2",
                    "description": "canada mongo vol iopps > 90%"
                "state": "Normal",
                "activeAt": "0001-01-01T00:00:00Z",
                "value": ""

Now I got the “rule_uid” of the configured alert rule from step 1 and ran /api/v1/provisioning/alert-rules/canada_mongo_disk which gives me this error below :

    "message": "could not find alert rule",
    "traceID": "00000000000000000000000000000000"

I don’t understand why I keep getting this error though I am using the correct api to fetch the alert by it’s rule_uid.
BTW, When I try the same steps above for migrated dashboards and alerts(from previous grafana version). it works fine. I am able to fetch the alert by it’s rule_uid.
I am seeing this issue only for newly created alerts in current grafana version(9.1.4).
Below is the alert rule_uid I configured as a label from UI. Is this the rule_uid that it is referring to ? Anything I am missing here.
To me this feels like a bug. Please confirm if this is a bug indeed or please let me know of a work around. I am currently blocked.

Also how is alert “rule_uid” is created when an alert is manually configured from UI. I just gave a label “rule_uid” as key and a value for it. I use this key value pair to map it to notification policy.

Try using /api/v1/provisioning/alert-rules, and then you will get the list of the alert rules with the actual uid. In you request example you are getting the list of alert instances where you can see the label rule_uid but not the uid for the alert rule that is actually what the /api/v1/provisioning/alert-rules/{uid} request is expecting.