Securing OTLP input

Grafana Tempo
1

At this point I fear I may not have understood the architecture behind the LGTM stack.

I’m tring to set up an LGTM stack, and after making some progress, I am now trying to configure authentication and multi-tenancy for the LTM components. I found basic auth is supported in both Mimir and Loki’s HTTP endpoints, but I am so far unable to find how to secure Tempo’s OTLP endpoints, neither HTTP or GRPC.

The closest I was able to find is in the Distributor docs, where they mention more advanced settings in open-telemmetry’s receiver docs. In there, I was able to find this basicauth extension example, but it doesn’t seem to be supported by Tempo. Neither within distributor…

failed parsing config: failed to parse configFile /etc/tempo/config.yaml: yaml: unmarshal errors:
2023-10-09T03:13:23.758773354Z line 12: field extensions not found in type distributor.Config

…or at the top level…

failed parsing config: failed to parse configFile /etc/tempo/config.yaml: yaml: unmarshal errors:
2023-10-09T03:11:40.182179824Z line 11: field extensions not found in type app.Config

So, my questions at this point are:

  1. Is it even possible to secure the OTLP endpoints?
  2. Should I instead rely on just the HTTP endpoint behind nginx for authentication?
  3. Does it even make sense what I’m trying to do? (maybe this is not how Tempo is supposed to be used at all)
  4. Should I instead focus on securing everything using mutual TLS?

Thank you in advance for any pointers,

Carlos

2

Hi @crypticmind. As you’re guessing, Tempo doesn’t support basic auth. Would you open an issue describing your request? That’ll help us prioritise work.

As I said above, unfortunately no.

That’s what we’ve recommended so far, yes. If you’re using the helm-charts, in the tempo-distributed chart there is gateway component, which is just that.

Sure. It makes sense.

I think you’re on the right path with basic auth.

3

Hi @mariorodriguez ,

Thank you very much for such a prompt response!

I’ll proceed with the HTTP + reverse proxy option then.

As suggested, I’ve created a feature request in the hope that this will be supported in the future.

Thank you again,
Carlos

4

Is there any reverse proxy with basic auth support for grpc?

5

AFAIK, GRPC lets you define protocols with an implied transport where there isn’t any means for intermediate hops to add middleware like security. If there’s a way, I’d be happy to learn from it.