Unable to set Tempo storage parameters with helm chart during installation

[ragamx]

Hi
I am trying to automate the installation and configuration of Tempo. For that I am using Task and helm charts. By default Tempo uses local storage for the tracing but we want to use Azure blobs for that. Azure requires storage_account_name and storage_account_key and are injected during installation as they are secrets and can’t be in a file. The problem is that the parameters passed during installation are not taken by tempo, it falls back to local storage. So far I have tried different combinations, for example:

helm install tempo grafana/tempo-distributed -n tempo \
--set storage.trace.backend=azure \
--set storage.trace.azure.container-name="mybucket" \
--set storage.trace.azure.endpoint-suffix=blob.core.windows.net \
--set storage.trace.azure.storage-account-name="myaccname" \
--set storage.trace.azure.storage-account-key="mykey"

Also different paths
–set config.storage.trace.azure.container-name=" "
…
–set tempo.storage.trace.azure.container-name=" "
…
–set data.tempo.storage.trace.azure.container-name=" "
…

But none of them work. The question is if it is possible to configure storage params during installation and what is the right way to do it.

Thanks

[joeelliott]

It appears that the config is loaded as a single variable:

grafana/helm-charts

Perhaps we could look at adding some common configuration options as helm vars? I don’t know a ton about helm charts.

[ragamx]

Hi

Yeah, it would be great to have those helm vars available so Tempo can be more production friendly. As I mentioned before having secrets in files is frown upon for obvious security reasons.

Any idea how long it might take?

Thanks.

[joeelliott]

Apologies, but unfortunately I do not know. Generally the community has been maintaining the helm chart.

I also have noticed that the loki-distributed chat requires the config file as a whole:

grafana/helm-charts - Example configuration using memberlist, boltdb-shipper, and S3 for storage

It looks like based on that we can parameterize specific fields in the config if we want to? If you have the time to submit a PR I would gladly review/merge.

[ragamx]

Yeah, it looks that Loki has the option to pass a yaml config file using

--set-file loki.config=/path/to/config.yaml

Tempo doesn’t seem to have that. As a work around and also very similar to that option, would it be possible to have a config.yaml file with proper storage parameters and just apply it to the Tempo running instance with kubectl?

kubectl apply -f config.yaml

That would help in the short term. But not sure how to get the yaml file with proper tempo distributed parameters?

PS I am also new to helm charts and I will try to spend some time helping with this missing helm vars option but not sure how long it will take me.

[joeelliott]

It looks like for us the parameter would be

--set-file config=/path/to/config.yaml

Is this what you tried?

[ragamx]

I didn’t try that before as I didn’t see it in the documentation. However it seems to work when using it along with the helm install command. The only drawback is that you need to include the configuration for the whole application and not only for the storage. If you include only the storage configuration the application won’t start as it looks for the rest of the parameters. But it helps thanks for that.

I have another unrelated issue not sure if I should open a new Topic, (also I know that I have been bugging you a lot, sorry!. I am very close to have a solid POC).
When I configure in Loki Derived fields as follow

Then go to Loki query , look for a TraceID in the logs click on the link, the system takes me to this page:

if I use Jaeger url http://localhost:16686/trace/${__value.raw} works but it takes me to Jaeger not Tempo. Also this one works:

http://grafana.staged-by-discourse.com/explore?orgId=1&left=["now-1h","now","Tempo",{"query":"${__value.raw}"}]

Thanks

[joeelliott]

I’m not 100% sure about your Loki Derived fields issue.

It looks like you’ve provided a value for the Query field. I think if you leave this blank then Grafana will “do the right thing”. Right now I think it’s just slapping the trace id on its root path b/c of the value you provided. Try clearing this field and see if it works.

[joeelliott]

Hmm, I take that back. Here’s our json config for that datasource:

              {
                matcherRegex: '(?:traceID|trace_id|tid)=(\\w+)',
                name: 'TraceID',
                url: '$${__value.raw}',
                datasourceUid: 'ops-tools1-tempo',
              },

I think that $$ is there to escape the $ sign. It appears in Grafana like:

I’m honestly not sure. If you can’t get it working perhaps we should file an issue in Grafana’s repo.