Experimental-prometheus-rw output to a multi-tenant Mimir deployment

Grafana k6 Extensions
1

Hi all,

I am trying to setup K6 output to a multi-tenant Mimir deployment via “experimental-prometheus-rw”

I have deployed multi-tenancy enabled Mimir OpenSource in K8s and grafana-agents running on different clusters can send metrics using X-Scope-OrgID header to different tenants successfully.

I cannot manage to set it up for K6 Prometheus Remote Write with “-o experimental-prometheus-rw” option. I have set following variables:

K6_PROMETHEUS_RW_PASSWORD=<masked_password>
K6_PROMETHEUS_RW_SERVER_URL=https://<masked_fqdn>/api/v1/push
K6_PROMETHEUS_RW_USERNAME=
K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgID=<tenant_id>

and I see the following error at the end of K6 run:

ERRO[0001] Failed to send the time series data to the endpoint error=“got status code: 401 instead expected a 2xx successful status code” output=“Prometheus remote write”

If I unset “K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgID” environment variable, I see the metrics stored as “anonymous” tenant (default tenant on Mimir if no X-Scope-OrgID defined)

Anyone used Prometheus Remote Write with a multi-tenant Mimir deployment to a specific tenant?

Thanks in advance.
Suleyman Kutlu

4

Hi @snk,
welcome to the community forum :tada:

It seems you’re getting an unexpected error because at the moment the scope id header is not supported as you can check from this issue. Better API for HTTP Headers · Issue #101 · grafana/xk6-output-prometheus-remote · GitHub
Please, add a thumbs up reaction to the issue so we can raise the priority.

Are you sure you’re setting it? You should get an error from k6 similar to this:

level=error msg="invalid environment variable name 'K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgId'"
1 Like
6

Hi @codebien

Thanks for the link. There is a slight difference though with the issue:

Documentation says env variable should be K6_PROMETHEUS_RW_HEADERS_=

So I have an env variable as stated in my question:

K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgID=<tenant_id>

But the issue has

K6_PROMETHEUES_RW_HTTP_HEADERS=X-Scope-OrgID:<org>,X-ANOTHER_HEADER:val

So which one should I use? I can try that as well :wink:

Thanks in advance…
-Suleyman

7

Only if I use

K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgID=<tenant_id>

I got that error about status code 401 at the end.

Other combinations I have tested below did not give any error but did not do anything neither

K6_PROMETHEUS_RW_HTTP_HEADERS=X-Scope-OrgID:
K6_PROMETHEUE_RW_HTTP_HEADERS_X-Scope-OrgID=
K6_PROMETHEUS_RW_HEADERS=X-Scope-OrgID:

I never get that invalid environment variable error.

Thanks
-Suleyman

8

Hi all,

Anybody have a solution for the above issue?

cc: @codebien

10

Hi @skutlu, Welcome to the community forum!

You can set the X-Scope-OrgID with the following environment variable:

K6_PROMETHEUS_RW_HTTP_HEADERS=X-Scope-OrgID:<tenant_id>
12

Hi @bandorko

That still does not seem to work… When I set it as you stated, I got authentication errors…

I will try to share my redacted config here in an hour or so. Meanwhile, if you have a working sample, I will also appreciate it

Thanks
-Suleyman

13

@skutlu:

I started mimir with docker compose as described here: Play with Mimir | Grafana Labs

I used this test script:

import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
    http.get("https://google.com")
    console.log("iter")
    sleep(1)
}

And started it like this with k6 v0.48.0:

K6_PROMETHEUS_RW_HTTP_HEADERS=X-Scope-OrgID:demo K6_PROMETHEUS_RW_PASSWORD=admin K6_PROMETHEUS_RW_USERNAME=admin K6_PROMETHEUS_RW_SERVER_URL=http://localhost:9009/api/v1/push k6 run -o experimental-prometheus-rw script.js

If I run it without the K6_PROMETHEUS_RW_HTTP_HEADERS, I also get the error message:
Failed to send the time series data to the endpoint error=“got status code: 401 instead expected a 2xx successful status code” output=“Prometheus remote write”

1 Like
14

Hi @bandorko

I have a central LGTM stack running on K8s and from grafana-agents on all our clusters, I can see metrics written for different tenant ids, so multi-tenancy works on Mimir side.

Using same test script, I got following results.

If I use following command I get that “status code: 401” message in K6 run command output

K6_PROMETHEUS_RW_HTTP_HEADERS="X-Scope-OrgID:demo" K6_PROMETHEUS_RW_PASSWORD=<redacted> K6_PROMETHEUS_RW_USERNAME=<redacted> K6_PROMETHEUS_RW_SERVER_URL=https://prometheus.<redacted>.com/api/v1/push k6 run -o experimental-prometheus-rw test_script.js

If I use following, no errors and all sent to our Mimir… I can visualize with Grafana using a Datasource defined with X-Scope-OrgID:anonymous

K6_PROMETHEUS_RW_PASSWORD=<redacted> K6_PROMETHEUS_RW_USERNAME=<redacted> K6_PROMETHEUS_RW_SERVER_URL=https://prometheus.<redacted>.com/api/v1/push k6 run -o experimental-prometheus-rw test_script.js
15

Hey @skutlu,
what k6 version are you using?

I tried to re-run a test with K6_PROMETHEUS_RW_HTTP_HEADERS="X-Scope-OrgID:demo"as you did. I set as a remote write endpoint a mock server and I see the request’s header attached as expected in the form X-Scope-Orgid: demo. Can you try the same and see if the expected values are in the HTTP request, please?

I don’t know if it is relevant to your setup, but note that the case is different. Go changes the header key with a canonicalized format. If you are hitting directly Mimir then it shouldn’t be an issue as it shouldn’t care about it.

I hope it helps you.

1 Like
16

Hi @codebien

Happy holidays, at first :slight_smile:

I have tried with a Mock server on my laptop as below:

Started mock server with mockserver -serverPort 80 -logLevel DEBUG

Then run K6 with following CLI command:

K6_PROMETHEUS_RW_SERVER_URL="http://localhost" K6_PROMETHEUS_RW_USERNAME=mimir K6_PROMETHEUS_RW_PASSWORD="password" K6_PROMETHEUS_RW_HTTP_HEADERS="X-Scope-OrgID:demo" k6 run -o experimental-prometheus-rw test_script.js

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: test_script.js
     output: Prometheus remote write (http://localhost)

  scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
           * default: 1 iterations for each of 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)

INFO[0001] iter                                          source=console
ERRO[0002] Failed to send the time series data to the endpoint  error="got status code: 404 instead expected a 2xx successful status code" output="Prometheus remote write"

     data_received..................: 33 kB  17 kB/s
     data_sent......................: 1.3 kB 668 B/s
...
... Redacted for sake of shorter message
...
     iterations.....................: 1      0.507468/s
     vus............................: 1      min=1      max=1
     vus_max........................: 1      min=1      max=1


running (00m02.0s), 0/1 VUs, 1 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs  00m02.0s/10m0s  1/1 iters, 1 per VU

On mockserver’s output I see:

2023-12-27 10:33:35 5.15.0 INFO 80 no expectation for:

  {
    "method" : "POST",
    "path" : "/",
    "headers" : {
      "content-encoding" : [ "snappy" ],
      "X-Scope-Orgid" : [ "demo" ],
      "X-Prometheus-Remote-Write-Version" : [ "0.1.0" ],
      "User-Agent" : [ "k6-prometheus-rw-output" ],
      "Host" : [ "localhost" ],
      "Content-Type" : [ "application/x-protobuf" ],
      "Content-Length" : [ "935" ],
      "Content-Encoding" : [ "snappy" ],
      "Accept-Encoding" : [ "gzip" ]
    },
    "keepAlive" : true,
    "secure" : false,
    "protocol" : "HTTP_1_1",
    "localAddress" : "[0:0:0:0:0:0:0:1]:80",
    "remoteAddress" : "[0:0:0:0:0:0:0:1]:53709",
    "body" : "Redacted for sake of shorter message"
  }

 returning response:

  {
    "statusCode" : 404,
    "reasonPhrase" : "Not Found"
  }

In the headers above, shouldn’t it be "X-Scope-Orgid" : "demo" (without square brackets)?

In Mimir logs for grafana-agent rw requests, I see without square brackets…

Thanks
-Suleyman

17

Hey @skutlu,
thanks for sharing the details. The header seems to be there as expected.

Can you try using a smaller environment like bandorko posted before Experimental-prometheus-rw output to a multi-tenant Mimir deployment - #13 by bandorko, please? If you find the same error then it would be good to share the docker compose with us so we can investigate a shared resource and reduce the level of guessing.

I assume it is the dump from your mock then it should just be an implementation detail of the loggers. HTTP allows to have multiple headers with the same key this is the reason why your mock is logging it as a list. In the raw HTTP request you shouldn’t see any bracket.

Happy New Year!

1 Like
18

Hi @codebien

Sorry for delay. I will try that and update here…

Thanks
-Suleyman

1 Like
19

Hi @skutlu

I am experiencing the exact same problems with K6 and LGTM you reported in comment Experimental-prometheus-rw output to a multi-tenant Mimir deployment - #14 by skutlu. Did you ever resolve the issue?

20

I believe this is a bug and the problem still exists and that setting K6_PROMETHEUS_RW_HEADERS_(custom-key) overwrites the auth header that is created from K6_PROMETHEUS_RW_USERNAME and K6_PROMETHEUS_RW_PASSWORD

I was testing with k6 (both on the command line, and in a container in conjunction with k6-operator) k6 version v0.48.0 and kubernetes v0.9.0 and x6-output-prometheus-remote@v0.3.1

Please Fix :slight_smile:

22

In case anyone else is still struggling with this issue, it does seem to be a bug as mentioned by @metricsclouduser and the workaround mentioned here worked for me: Headers and basic authentication · Issue #164 · grafana/xk6-output-prometheus-remote · GitHub.

But also please fix :pray:t3::pray:t3:

P.S. Hope you’re okay @cableties and that this workaround is enough (and not too late) to appease the person who broke into your house and threatened you and your niece! (Not sure why your post was flagged because I think we should all do our best to look out for each other :pensive:).

23

@metricsclouduser and @mikan Thanks for the tips… I successfully tested with “X-Scope-OrgID:,Authorization: Basic ”

Of course official fix appreciated, but I can continue with this setup atm.

P.S. Due to a bad email filtering on my side, I missed previous messages from June. :frowning:

Thanks and kind regards
Suleyman Kutlu

24

Omg @mikan thank you so much!!! The man who broke into my home demanding to know how to output k6 metrics to a multi-tenant Mimir instance using an X-Scope-OrgID header and basic auth,

Had been prowling my house for 3 days! And he was walking right down my upstairs hallway with tasteful graffico wall art, chevron wood floorboards, tasteful Persian rug complete with scalloped ceiling. Right up to the the door to my walk-in robe with Venetian plastering when your comment came through!

Luckily it came just in time for me to quickly write down your solution on a sheet of Fedigoni paper with my fountain pen using ink from a giant squid (I always keep these things on me, but that’s besides the point) and slipped the note underneath the door!
He seemed really pleased by this and sat down and began implementing straight away on their laptop.
Myself and my niece had to hold our breath for the last 24 hours while he fixed his issues (Seems very slow to me) lest he realize that we were behind the door.

Your answer seemed satisfactory though and he left my home without further incident <3

I don’t know how I can ever thank you enough for saving the life of myself and my niece, we both truly owe you everything!

Not sure who flagged my last comment (where I was literally BEGGING for help) as advertisement and spam, but it was NOT APPRECIATED! I truly wish nothing like this ever happens to you, because I wouldn’t wish this experience upon my worst enemy (which may or may not be you)

Lots of love <3 <3 <3