404 Errors loading rules when viewing AMP rules (alert/recording) in Grafana UI

I have posted this problem on Github (404 Errors loading rules when viewing AMP rules (alert/recording) in Grafana UI · Issue #43301 · grafana/grafana · GitHub) and was asked to post the problem in the community instead. So cross posting here as well.

What happened:
I imported an Amazon Managed service for Prometheus (AMP) as a Prometheus datasource on latest v8.3.3 (30bb7a93c). While the alert tab shows the list of the rules I had in AMP, it will always show the following error when it try to do GET call to the http://localhost:3000/api/ruler/1/api/v1/rules:

Screen Shot 2021-12-17 at 11 34 14 AM

What you expected to happen:
Expected to render this page without an error nor 404 failure.

How to reproduce it (as minimally and precisely as possible):

  1. Add an Amazon Managed Service for Prometheus as a datasource
  2. Navigate to Alert tab

Anything else we need to know?:
FYI: All the other requests were successfully made, including
http://localhost:3000/api/prometheus/1/api/v1/rules
http://localhost:3000/api/prometheus/grafana/api/v1/rules
http://localhost:3000/api/ruler/grafana/api/v1/rules

From the debugging log

DBUG[12-17|12:10:13] Applying default URL parsing for this data source type logger=datasource type=prometheus url=<MY_AMP_URL>
INFO[12-17|12:10:13] Proxying incoming request                logger=data-proxy-log userid=1 orgid=1 username=admin datasource=prometheus uri= method=GET body=
DBUG[12-17|12:10:13] User does not have access to any namespaces logger=ngalert.api
DBUG[12-17|12:10:13] User has no access to any namespaces     logger=ngalert.api
DBUG[12-17|12:10:13] Requesting                               logger=data-proxy-log url=<MY_AMP_URL>/rules
INFO[12-17|12:10:13] Request Completed                        logger=context userId=1 orgId=1 uname=admin method=GET path=/api/ruler/1/api/v1/rules status=404 remote_addr=[::1] time_ms=157 size=32 referer=http://localhost:3000/alerting/list

On top of the problems,
I have couples questions based on this situation:

  1. from the doc, it seems like grafana is expected to provide legacy prometheus api? I know AMP endpoint ultimately use prometheus prefix instead of /api/prom. I am curious if this is the reason why I have seeing such error?
  2. From the error message, it seems like I can enable ruler api? Curious if there is a way to define ruler api by myself?
  3. The error log also mentioned User has no access to any namespaces, is this a permission issue that I need to configure both my AMP workspace and grafana instance?

Environment:

  • Grafana version: v8.3.3 (30bb7a93c).
  • Data source type & version: Amazon Managed Service for Prometheus
  • OS Grafana is installed on: Mac OS
  • User OS & Browser:Mac OS , Chrome
  • Grafana plugins: Amazon Managed Service for Prometheus
  • Others:

Grafana’s transition from legacy alerting to the Unified Alerting platform represents a big step forward. But, there are many factors that can influence behavior, and it is often hard for the community to troubleshoot issues without a thorough understanding of your unique setup. Try to include the following info:

  • What is your Grafana version?
  • Are you using Grafana Cloud or self-hosted Grafana?
  • Are you using legacy alerting or Unified Alerting?
  • was the alert in question migrated from the legacy platform into Unified Alerting, or did you first create it inside the new platform?
  • Please list ALL configuration options related to alerting. You can find these in the Alerting and Unified Alerting sections of Grafana’s config file. If you are now using or have previously used the beta version of ngalert (released with Grafana 8), please note that too.
    • you can use this table to better understand how configuration options can interact with each other
  • If this is a templating issue on Unified Alerting, check if your alert is using a multi-dimensional rule or not.
  • List the datasource associated with the alert
  • Increase the verbosity of the Grafana server logs to debug and note any errors. For printing to console, set the console logs to debug as well.
  • Search for open issues on GitHub that sound similar to your problem