Grafana tempo with helm-distributed always returns different results if I query in grafana

Grafana Tempo
1

hello,

I try to use grafana tempo with the helm-distributed chart.
however when I make the same query multiple times, the result set in grafana will change. is there something that I miss?

this is our tempo chart:

storage:
  trace:
    backend: gcs
    gcs:
      bucket_name: '.....'
memcached:
  enabled: false
gateway:
  enabled: true
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
    hosts:
    - host: ....
      paths:
      - path: /
        pathType: Prefix
    tls:
      - secretName: ....
        hosts:
        - ....
  basicAuth:
    enabled: true
    username: "......"
    password: "......"
  # service:
  #  type: Service
  #  loadBalancerIP: ""
minio:
  enabled: false
traces:
  otlp:
    grpc:
      enabled: true
    http:
      enabled: true
  zipkin:
    enabled: false
  jaeger:
    thriftHttp:
      enabled: false
  opencensus:
    enabled: false
multitenancyEnabled: true

my querier always fails with cancelled_requests=12 or so, when I have a range of 2 days.

Edit: nvm the problem was my understanding of grafana-tempo when it reaches the limit of 20 traces it will just abort, so I raised the number of traces that get returned

1 Like
2

yes, tempo returns the first N (N=limit) results that match the query in the selected time range, search results are not sorted.

see similar discussion on this thread

1 Like
3

yeah it would be great if grafana itself could have a bigger default than 20 results for tempo, since it doesn’t even slow down with like 1000 traces or so…