Tempo: Add environment in Grafana Cloud UI Search

Grafana Tempo
1

Hi,

I am new to tracing and I am wondering why there does not seem to be an environment field to search for in Tempo. I added these two variables to my traces:

const otel = new NodeSDK({
    resource: new Resource({
        [SemanticResourceAttributes.SERVICE_NAME]: APP_API_NAME,
        [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: DEPLOYMENT_TIER,
    }),

I can not add the field or at least see the label “DEPLOYMENT_ENVIRONMENT”. As we have the name of the service for apps on 3 stages, it is hard to find the right one.

What is best practice for you here?

image
image855×488 31.7 KB

2

Do you have own tempo for all stages? Or how do you distinguish between?

3

Those variables look like resource attributes, so filter by resource tag (SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT should be deployment.environment) in the UI:

image
image2434×1393 240 KB

1 Like
4

ok it does… thanks @jangaraj. Maybe it needed some more time that the value was proposed.

The documentation sais we can extend the search but going to the settings in the datasource it sais “This data source was added by config and cannot be modified using the UI” So I can not modify it. This means I can also not add the rules to search logs from traces. Is this also an API thing here? Or is it simply not possible to modify datasource settings for Tempo in Grafana Cloud? That would be sad, I could never configure it to find logs from traces.

5

Just to clarify: you are using Grafana Cloud = someone is hosting “everything” for you (Grafana, Loki, Mimir, Tempo, …). That someone also precofigured Grafana for you - e.g. datasources to all provided services.
But you can misconfigure them as a standard curious rookie users. So safe options is have those preconfigured datasources in “read only”, so you can’t break them.

You can still add (it can be limited, depends on used cloud plan) more datasources. Just add new Tempo datasource, which will be pointed to the same Tempo instance (you can find/generate url, username, password/token in you Grafana cloud administration). This your new datasource will be editable, so you can configure what you need (of course you can break it, so follow documentation).

6

@jangaraj

oh I am sure that Grafana Labs can host all these things way better than I do. And it is perfectly fine for me, that I am not a root user :slight_smile:

I just want to change some settings here so that it fits to my needs.

image
image1158×682 46.1 KB

But the solution, that I can add Tempo again, I am fine. I did not think about that this might be possible. Thanks for your patient help.

7

@jangaraj

Just wanted to say thank you again. It worked great to add the data source again. So I think you need to see the default created data source in Grafana as a generic one and you can create another one for each use case like

app1 needs custom query to find logs from traces
app2 needs filter by spanID etc.