I have posted this problem on Github (404 Errors loading rules when viewing AMP rules (alert/recording) in Grafana UI · Issue #43301 · grafana/grafana · GitHub) and was asked to post the problem in the community instead. So cross posting here as well.
What happened:
I imported an Amazon Managed service for Prometheus (AMP) as a Prometheus datasource on latest v8.3.3 (30bb7a93c). While the alert tab shows the list of the rules I had in AMP, it will always show the following error when it try to do GET call to the http://localhost:3000/api/ruler/1/api/v1/rules:
What you expected to happen:
Expected to render this page without an error nor 404 failure.
How to reproduce it (as minimally and precisely as possible):
- Add an Amazon Managed Service for Prometheus as a datasource
- Navigate to Alert tab
Anything else we need to know?:
FYI: All the other requests were successfully made, including
http://localhost:3000/api/prometheus/1/api/v1/rules
http://localhost:3000/api/prometheus/grafana/api/v1/rules
http://localhost:3000/api/ruler/grafana/api/v1/rules
From the debugging log
DBUG[12-17|12:10:13] Applying default URL parsing for this data source type logger=datasource type=prometheus url=<MY_AMP_URL>
INFO[12-17|12:10:13] Proxying incoming request logger=data-proxy-log userid=1 orgid=1 username=admin datasource=prometheus uri= method=GET body=
DBUG[12-17|12:10:13] User does not have access to any namespaces logger=ngalert.api
DBUG[12-17|12:10:13] User has no access to any namespaces logger=ngalert.api
DBUG[12-17|12:10:13] Requesting logger=data-proxy-log url=<MY_AMP_URL>/rules
INFO[12-17|12:10:13] Request Completed logger=context userId=1 orgId=1 uname=admin method=GET path=/api/ruler/1/api/v1/rules status=404 remote_addr=[::1] time_ms=157 size=32 referer=http://localhost:3000/alerting/list
On top of the problems,
I have couples questions based on this situation:
- from the doc, it seems like grafana is expected to provide legacy prometheus api? I know AMP endpoint ultimately use
prometheusprefix instead of/api/prom. I am curious if this is the reason why I have seeing such error? - From the error message, it seems like I can enable ruler api? Curious if there is a way to define ruler api by myself?
- The error log also mentioned
User has no access to any namespaces, is this a permission issue that I need to configure both my AMP workspace and grafana instance?
Environment:
- Grafana version: v8.3.3 (30bb7a93c).
- Data source type & version: Amazon Managed Service for Prometheus
- OS Grafana is installed on: Mac OS
- User OS & Browser:Mac OS , Chrome
- Grafana plugins: Amazon Managed Service for Prometheus
- Others: