Unable to set "all values" filter to filter traces

Grafana
1

I have a dashboards with grafana/tempo with 2 variables defined: service name and operation (span name). For both, I need an “ALL values” option.

For simplicity, let’s assume I have 2 service names: products-api, availability-api. I want my dashboard table widget + traces widget to rely on these variables (obviously). And here comes the problem.

When I choose a single service (not all), e.g. products, all works fine:
{resource.service.name="products-api"}- this returns results as expected :ok_hand:

But when I click “ALL” then I get following query: {resource.service.name="products-api\\|availability-api"} which never returns what it should.

I’ve tried regexes, manually defining values (in the query), I tried “custom ALL value“ in variable definition and I failed miserably with each of them and I’ve run out of ideas. And I hope this should be doable, as it seems such an important feature to have…

Any help would be appreciated, thank you.

2

Generally, define custom All value, as wild char regexp (blind guess .*) and then write regexp query, which will use it, e.g.

{resource.service.name=~"$servicenamevariable"}

This is not copy/paste example, so test it/improve it (syntax issues, regex issue, tempo issues, variable formatting, …) for your desired state.

3

I have already tried that - and the problem I encountered is that .* gets escaped by grafana and it doesn’t work as expected :frowning:

This is what gets executed - after variable gets applied (more or less, maybe not 100% accurate, but fails to work as expected)

{resource.service.name=~"\\.*\\"}

A follow up USECASE. Even if showing traces for all services (service names) might be questionable, nevermind - I have a need to configure a dashboard variable “operation”, which filters by span name, example values are: “GET / something”, “specific instrumentaion operation X” etc. And in this case I really need to show ALL traces, even within one service.

All grafana resources I’ve seen seem to use “expore” tab for traces and haven’t found a trace-specific dashboard. Could it be that hardly anyone configures trace-specific grafana dashboards (and so a rather obvious scenario is unsupported, as people are happy with “explore” tab)?

4

Use advance variable and use raw format so there is no special magic/escapes:

{resource.service.name=~"${servicenamevariable:raw}"}