In K6, if I use custom tags, they end up as labels in Prometheus. This is an issue as my custom tag value contains dynamic data and thus always different. this causes issue with Grafana alerts as it will threat each different label value as a new alert as according to: Labels and annotations | Grafana documentation. Is there a way to send a custom tag from K6 but not have it end up as a label and instead as an annotation? thanks!
Hello @wjc,
Welcome back to the community forums! And apologies for the delayed answer
To be honest, I think I get what your current situation is and the problem you’re experiencing with cardinality, but as far as I know, there is no way to use custom tags as annotations.
The only, a bit crazy, and non-trivial solution that comes to my mind for that is to:
- Write a log line for every event that you consider “an annotation”.
- Capture logs and send them to Loki, or any other DS supported by Grafana.
- Use that DS as the source for displaying annotations in your dashboards.
I know it’s far from ideal, but it is what it is for now. I’m sorry for not being able to bring a definitive solution. That said, bear in mind that you can always submit a new feature request either in the main k6 repository or in one of the output extensions.
I hope it helps, thanks!
Hey @wjc,
Here’s a quick update:
You may want to take a look at this issue, I think it’s kinda related with what you’re trying to accomplish and there’s an interesting discussion there, as well as a possible workaround.
Have you tried with options.systemTags
?
Thanks!