Grafana cloud synthetics test - http-debug is true by default and dashboard logs by default may show sensitive auth header data

Grafana k6
1

By default it appears that standard http calls such as the simple example below log the Authorization header and are visible in the logs section of the dashboard.
The logs include source=http-debug.

This appears to be the default behaviour.

Is there any way of turning off http debugging specifically in the synthetic monitor scripts.

Using

export const options =  {

  httpDebug: ‘none’

}

does not work. The documentation implies httpDebug:’full’ is not the default however it does appear to me to be.in the context of synthetics.

Note - I appreciate in the example below you could use k6/secrets - however that does not resolve the problem as in subsequent calls the auth token generated also appears in the logs.for subsequent requests.

Thanks

const authTokenResponse = http.post(ctp_auth_url, `grant_type=client_credentials&scope=${ctp_scopes}`, {

    headers: {

      'Content-Type': 'application/x-www-form-urlencoded',

      'Authorization': `Basic ${encoding.b64encode(`${ctp_client_id}:${ctp_client_secret}`)}`,

    },

  });

(subsequent request - can’t use k6/secrets - authorization header appears in logs)

const req = http.get(url,{

    headers: { 

      'Content-Type': 'application/json',

      'Authorization': `Bearer ${access_token}`,

    }

  })
2

Hi @michaelodriscoll , thanks for flagging this. We’ve received similar reports from others and have a a voice-of-customer issue open for it. I don’t have an ETA on a fix, but we are looking at

  • making the `httpDebug` option configurable in the scripts (it is currently hardcoded to httpdebug=”headers”)
  • updating the secrets module to allow masking values that are not stored as secrets (similar to how github actions allows values to be masked)

It’s possible to workaround the issue with a private probe, where you can set the K6_HTTP_DEBUG environment variable, which overrides the hardcoded CLI flag in the probe agent. For example with docker:

docker run -e K6_HTTP_DEBUG='' grafana/synthetic-monitoring-agent --api-server-address=${API_SERVER} --api-token=${API_TOKEN} --verbose=true
3

Many thanks @peterschretlen1 for confirming current behavour and potential private probe option, we’ll look into that. cheers.

4

Hi!

I’ve just hit this issue too. Any news on a solution that doesn’t require running private probes? I know ETAs can be tricky but it would be great to know if this is being actively worked on now.

Thanks

5

Hi @jimcroft there is issue open to add support for dynamic secrets and/or turning off verbose logging: https://grafana.ideas.aha.io/ideas/IDEASVOC-I-9352. This is on the team’s radar, but we don’t have an ETA for it unfortunately.

If you have generated auth tokens showing up in headers, how about an adaptive logging drop rule? It might be worth trying in the meantime as a workaround.

For example I set this rule up to drop any synthetics logs that contain a basic auth header:

Image

6

Hi @peterschretlen1

When I want to go to that issue Github gives me 404. And I can not find the synthetic-monitoring repo at all. Can it be an internal repo?

7

@snk my apologies, you’re correct. https://grafana.ideas.aha.io/ideas/IDEASVOC-I-9352 is the public facing version of the github issue. I’ve updated the original post.