Prometheus remote write fail in grafana cloud

Grafana k6
1

hi

I have a grafana cloud setup with a prometheus receiving in this url https://prometheus-prod-22-prod-eu-west-3.grafana.net/prom/push
I am trying to send the k6 metrics as well to prometheus to remove statsD and i am using -o experimental-prometheus-rw with the latest k6 version ( k6 v0.47.0 (commit/5ceb210056, go1.21.2, linux/amd64)

I have tried as well building a new binary with the extension in its latest version as well with same results.

K6_PROMETHEUS_RW_SERVER_URL=https://prometheus-prod-22-prod-eu-west-3.grafana.net/api/v1/write
K6_PROMETHEUS_RW_USERNAME=xxxx
K6_PROMETHEUS_RW_PASSWORD=xxxxxxx
k6 run -o experimental-prometheus-rw test.js

I have tried with 2 enpoints

/api/v1/write the one in the documentation and in the local environment
/prom/push the one we are using in our environment to push metrics

both with same results.

ERRO[0010] Failed to send the time series data to the endpoint error=“got status code: 404 instead expected a 2xx successful status code” output=“Prometheus remote write”

In local i can correctly send the metrics to localhost:3000/api/v1/write
with the flag --web.enable-remote-write-receiver, that is as well in the documentation of k6, but i am failing to find it to enable it in grafana cloud.

Not sure if it is only problem I have in the process, but at the moment i am stuck not being able to enable it in our grafana cloud and not being sure whether it is or not enabled

Any idea or reference?

Thanks

3

Hi @mrsdaehin,

The way you have worded the question and after some googling it seems you are asking how to enable remote write in grafana cloud.

Given this docs it is possible you have not configured the user to have metrics:write access although getting 404 is strange :thinking:

Can you confirm that the user and password you are using have this set up?

4

Hi @mrsdaehin!

I think /api/prom/push is the correct endpoint.

1 Like
5

Sorry for bad wording i will reformulate the question.

Is there a way to send k6 metrics to grafana cloud prometheus service directly?

In local i can enable --web.enable-remote-write-receiver in the prometheus service BUT i cannot find the option in the prometheus configured in grafana cloud.

Any idea how to set --web.enable-remote-write-receiver for the prometheus in grafana cloud?

Thanks

6

@mrsdaehin this likely isn’t for the “grafana k6” category, but even more importantly - it seems to be enabled by default. And @bandorko did also found the endpoint is slightly different than what it is locally.

Can you try with taht endpoint and check in the cloud as per the link I gave you if the user and password has the correct permissions?

1 Like
7

@mrsdaehin : I am able to send k6 metrics to grafana cloud prometheus service from my local machine with the /api/prom/push endpoint and using a token with metrics:write scope as a password. If you get 404 then I am sure, that the endpoint you are using is wrong.

8

Thanks! it was the typo in the endpoint- Thanks @bandorko Bandorko!

2 Likes