Sensitive info masking in Synthetic Monitoring

Grafana
1

Hi

I have been assisting a company to get Synthetic monitoring solutions in place.
They have initially made this work by just supplying a API Key as a header value, but then discovered that the API key is actually logged and visible in Grafana when you Explore Synthetic Monitoring.

image
image1856Ă—457 69.2 KB

This is seen as a security risk.

  1. Is it possible to disable this logging in Grafana or at least suppress the logging of headers?
  2. If they decide to try a multi-step HTTP check with auth handled in the request body, will that also be logged?

Thanks

2

Are you providing that key in the dashboard’s datasource itself or in the datasource config when creating the datasource in connections

6

Hi @mornelsd thanks for raising this. It is a known issue, I’ve referenced a few related github issues below that others have flagged. The team has a “secrets management” feature for synthetic monitoring in active development. The feature includes redacting the secret from the logs, and secrets aren’t visible when looking at the synthetic monitoring datasource.

I don’t have an ETA for the release of the feature, but we’re working towards a public preview as soon as we can.

As for workarounds, the most secure workaround at the moment is to use k6 scripted checks with a private probe that is configured with the secrets as environment variables. That way the secrets doesn’t exist in the datasource at all. It’s admittedly an inconvenient workaround, but can share more details if it’s of interest. Some users have had success with this approach. Even then, care has to be taken not to leak the secret via logging.

1 Like