Handling OAuth token expiration in Custom Resource Definition (CRD)

I use the xk6 prometheus remote write extension for pushing my metrics and logs. I use a custom resource definition (CRD) file and use env definition to pass the values ex K6_PROMETHEUS_RW_HEADERS_AUTHORIZATION and K6_LOG_OUTPUT where the authorization tokens are being passed. But when I run longer duration tests, these tokens that are passed gets expired in between as these are valid only for 1 hr (by the provider) and I am looking for options on how to pass the new tokens to my CRD. There is no option to extend the duration of the token from the provider side,

Looking for solutions if anyone in the group has faced similar issues.

Hi @akp

Welcome to the community forum :wave:

In order to better help you out we’ll need a bit more context:

  • You are using the extension, GitHub - szkiba/xk6-prometheus: Prometheus HTTP exporter for k6, not Prometheus remote write?
  • Can you share your deployment details? Is this deployed on Kubernetes, using the k6 operator?
    If will help if you can share your (sanitized) deployment details, and how you execute k6.
    • If running on Kubernetes, are you restricted to mounting the authentication details via a custom file CRD (no configmaps, vaults, etc.)? How do you update the file CRD in other applications using Prometheus in your infra, which seems to require updating the token every hour? Are all using a file CRD and the pods update the new token, while the k6 extension doesn’t?

Thanks in advance for the additional information.

Thank you for your help.

We are using GitHub - grafana/xk6-output-prometheus-remote: k6 extension to output real-time test metrics using Prometheus Remote Write.

We are trying to run distributed load-testing using K6 and xK6 extenstions in a kubernetes environment. Our docker image has K6 and xk6-output-prometheus-remote applied right. Our cloud provider is Azure. Later, we generate a CRD file and applies this to the kubernetes cluster. For metrics, we use K6_PROMETHEUS_RW_HEADERS_AUTHORIZATION (out of the box option) and the value contain the bearer token that we generate during the process. Our test scripts are located on a PVC. Now by default, our Oauth tokens are valid for 1 hr. So when the tests run for a duration > 1hr, the metrics fails to be written to prometheus after an hr. I am trying to understand if there are any hooks or ideas to update this authorization header (environment variable of the Pod) out of the box? Or is generating tokens with greater expiration token the only option - for which we have security concerns. Let me know if you need any further details.

Hey @akp,
sorry for the late reply.

I agree that at the moment there isn’t any way for supporting your use case. The unique solution I can think of is to implement the OAuth2 flow as described in the Prometheus doc with client_id and client_secret.

Could it be a solution for you? Does your provider accept the OAuth2 authentication flow?

1 Like