Grafana Tempo multi-cluster connection

Grafana Tempo
1

Hi,
I’d like to ask you for the multi-cluster connection availability to the Grafana Tempo.

We have one main AKS cluster which plays as main monitorng cluster. There are deployed via helm charts:

  • Grafana via grafana/grafana,
  • Grafana Loki via grafana/loki-distributed,
  • Grafana Mimir via grafana/mimir-distributed,
  • Grafana Tempo via grafana/tempo-distributed.

At others AKS cluster we have only deployed Grafana Agent which collects metrics, logs and traces and send them to our main monitoring cluster.
At the main monitoring cluster we have configured ingress-nginx with host grafana.test.com for the Grafana and Grafana Mimir connection.

Is it possible to configure ingress also for the Tempo? We have only found gateway configuration in the helm chart. Is it neccessary to create specific ingress object for that purpose manually?

Thanks!

1 Like
2

Hi,

I have the same setup here: one k8s cluster with LGTM and others 2 k8s clusters (app).

If you check tempo-distributed default helm values, you have the gateway entry:

In my case, I’m running at Azure. I do not use an ingress. The cluster is connected to each other via vNet peering, and I’ve setup the gateway with a service (type = loadbalance) using an internal load balance.

3

Thanks for you reply! Is there a different way? We have one ingress controller with 3 ingresses: mimir, grafana and loki. I was wondering if it isn’t there some connectivity between loki and ingress when both of them are on a same port 3100.
That just went through my mind. Maybe it does not make sense.

4

If you have an ingress, recheck the values in Loki, Tempo, Mimir and Grafana. Did you configured each one to use your ingress (specifying the ingressClassName value)?

Loki: helm-charts/charts/loki-distributed/values.yaml at e9237f7060c7be4e04f5e5427cdf61db8ee76a64 · grafana/helm-charts · GitHub (gateway)

Mimir: mimir/operations/helm/charts/mimir-distributed/values.yaml at be486e1ed2747a8bad70e966c9b928d0baf53716 · grafana/mimir · GitHub (gateway)

Tempo:
helm-charts/charts/tempo-distributed/values.yaml at e9237f7060c7be4e04f5e5427cdf61db8ee76a64 · grafana/helm-charts · GitHub (gateway)

5

Each k8s service listen to port 3100, but using different IP. It is not a problem