404 Errors loading rules when viewing AMP rules (alert/recording) in Grafana UI

Grafana Prometheus
#1

I have posted this problem on Github (404 Errors loading rules when viewing AMP rules (alert/recording) in Grafana UI · Issue #43301 · grafana/grafana · GitHub) and was asked to post the problem in the community instead. So cross posting here as well.

What happened:
I imported an Amazon Managed service for Prometheus (AMP) as a Prometheus datasource on latest v8.3.3 (30bb7a93c). While the alert tab shows the list of the rules I had in AMP, it will always show the following error when it try to do GET call to the http://localhost:3000/api/ruler/1/api/v1/rules:

Screen Shot 2021-12-17 at 11 34 14 AM

What you expected to happen:
Expected to render this page without an error nor 404 failure.

How to reproduce it (as minimally and precisely as possible):

  1. Add an Amazon Managed Service for Prometheus as a datasource
  2. Navigate to Alert tab

Anything else we need to know?:
FYI: All the other requests were successfully made, including
http://localhost:3000/api/prometheus/1/api/v1/rules
http://localhost:3000/api/prometheus/grafana/api/v1/rules
http://localhost:3000/api/ruler/grafana/api/v1/rules

From the debugging log

DBUG[12-17|12:10:13] Applying default URL parsing for this data source type logger=datasource type=prometheus url=<MY_AMP_URL>
INFO[12-17|12:10:13] Proxying incoming request                logger=data-proxy-log userid=1 orgid=1 username=admin datasource=prometheus uri= method=GET body=
DBUG[12-17|12:10:13] User does not have access to any namespaces logger=ngalert.api
DBUG[12-17|12:10:13] User has no access to any namespaces     logger=ngalert.api
DBUG[12-17|12:10:13] Requesting                               logger=data-proxy-log url=<MY_AMP_URL>/rules
INFO[12-17|12:10:13] Request Completed                        logger=context userId=1 orgId=1 uname=admin method=GET path=/api/ruler/1/api/v1/rules status=404 remote_addr=[::1] time_ms=157 size=32 referer=http://localhost:3000/alerting/list

On top of the problems,
I have couples questions based on this situation:

  1. from the doc, it seems like grafana is expected to provide legacy prometheus api? I know AMP endpoint ultimately use prometheus prefix instead of /api/prom. I am curious if this is the reason why I have seeing such error?
  2. From the error message, it seems like I can enable ruler api? Curious if there is a way to define ruler api by myself?
  3. The error log also mentioned User has no access to any namespaces, is this a permission issue that I need to configure both my AMP workspace and grafana instance?

Environment:

  • Grafana version: v8.3.3 (30bb7a93c).
  • Data source type & version: Amazon Managed Service for Prometheus
  • OS Grafana is installed on: Mac OS
  • User OS & Browser:Mac OS , Chrome
  • Grafana plugins: Amazon Managed Service for Prometheus
  • Others: