Cannot sent metrics to AWS Managed Prometheus

Grafana Alloy
1

We try to use Grafana Agent Flow with AWS Managed Prometheus from a Kubernetes cluster which runs directly on a EC2 instance in the same AWS account.

For that, we created an AMP workspace and configured Grafana Agent Flow to send metrics to AMP:

    prometheus.remote_write "mimir" {
      endpoint {
        url = "https://aps-workspaces.eu-central-1.amazonaws.com/workspaces/ws-....1/api/v1/remote_write"
        headers = {"X-Scope-OrgID" = "clustername1"}
        sigv4 {
          region = "eu-central-1"
          access_key = "AKIAxxxxxxxxxxxxxxxxxXP"
          secret_key = "b...........................................................9"          } 
      }
    }

Grafana Agent Flow: 0.37.2
All EC2 nodes has the policy AmazonPrometheusRemoteWriteAccess applied.

The error message is:

server returned HTTP status 403 Forbidden: {\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method.

Do we miss something ? E.g. is a sigv4-proxy needed, or is this already integrated in GAF 0.37.2

2

I don’t understand why you need access/secret key, when you already have EC2 instance profile with AmazonPrometheusRemoteWriteAccess.

3

It was a try because without the ACCESSKEY/SECRETKEY I also get an error message

4

Off topic: I used OTEL collector to send metric to AMP in the past and it was working fine. It may be usefull for you to switch to ADOT if you don’t need to be dependant on GA, so you will be fully AWS compatible/supported.

5

I found the problem

The reason was the additional and not necessary line:
headers = {“X-Scope-OrgID” = “clustername1”}

This configuration was used for Mimir as a backend and does not work for “AWS Managed Prometheus”

This minimal configuration is working:

prometheus.remote_write "mimir" {
      endpoint {
        url = "https://aps-workspaces.eu-central-1.amazonaws.com/workspaces/ws-....1/api/v1/remote_write"
        sigv4 {
          region = "eu-central-1"
        }
      }
    }