"Invalid API key" error when setting up remote_write in prometheus

Grafana Cloud
#1

When I run
curl -H "Authorization: Bearer <my api key>" https://xxxxxx.grafana.net/api/dashboards/home
It works perfectly, no “unauthenticated” errors.

But when I set up prometheus’ remote_write like this:

remote_write:
      - url: <url that I found using docs https://grafana.com/docs/grafana-cloud/metrics-prometheus/>
        basic_auth:
          username: <username that I found using docs https://grafana.com/docs/grafana-cloud/metrics-prometheus/>
          password: <my api key>

It results in authentication error: invalid API key in logs.

I’ve tried replacing the basic_auth with authorization:

authorization:
  credentials: <my api key>

But it showed legacy auth cannot be upgraded because the host is not found in logs.

What am I doing wrong? :confused:

#2

Hello!

The API key you’re using in that first command calls to your grafana.net instance which requires the kind of API key generated from within the your grafana.net instance under Configurations > API Keys (https://<instance_slug>.grafana.net/org/apikeys) . You likely generated the API key from there and therefore it works to authenticate with that Grafana instance.

However the kind of API key you need for remote_write to a Grafana Cloud metrics endpoint (Prometheus) requires a Grafana Cloud account-level API key which is found in your account’s Cloud Portal (https://www.grafana.com/<account_slug>/api-keys). You’ll need at MetricsPublisher level API key for remote_write metrics pushing.

My assumption here is that you are using a grafana.net API key for the remote_write run and that this is why it is failing – it is the wrong kind of API key. Please generate a MetricsPublisher key from your Cloud Portal and try again. This should work.

1 Like
#3

Thanks, that solved the problem.

1 Like
#4

This is confusing UX. One would expect to be able to generate both types of API keys in one place. Instead, there are two completely different places to create API keys.

Is there anything that can be done to make this better? Maybe make this more clear in Monitoring a Linux host using Prometheus and node_exporter | Grafana Cloud documentation ?

As an aside, the documentation section that says " You can find the /api/prom/push URL, username, and password for your metrics endpoint by clicking on Details in the Prometheus card of the Cloud Portal." is not clear. Please consider showing an example screenshot of where to find the URL. (I eventually figured it out but it required way too much poking around - I must have tried 10 different variations on that URL.)