Grafana kubernetes monitoring

Grafana Tempo
, ,
1

Hey guys, please I need help with getting traces to grafana cloud, I already deployed grafana monitoring to the cluster, I’m getting logs and I think to some extent metrics but I’m not getting traces at all even after instrumenting my app. I tested the traces locally and it’s working when I used my local endpoint, but when I deployed the app to cluster and used the endpoint provided by Grafana during configuration, it doesn’t work. Thanks

2

Can you share the config you used with your deployment? I can try and determine which endpoint you want to point your applications to.

3

Observility .env

OTEL_EXPORTER_OTLP_ENDPOINT="http://grafana-k8s-monitoring-grafana-agent.default.svc.cluster.local:4318"

OTEL_EXPORTER_OTLP_PROTOCOL="http"

OTEL_TRACES_EXPORTER="otlp"

OTEL_NODE_RESOURCE_DETECTOR="env,host,os"

cluster:
  name: "cluster-alpha-us-central1"
externalServices:
  prometheus:
    host: https://prometheus-prod-13-prod-us-east-0.grafana.net
    basicAuth:
      username: "1518595"
      password: password
  loki:
    host: https://logs-prod-006.grafana.net
    basicAuth:
      username: "860032"
      password: password
  tempo:
    host: https://tempo-prod-04-prod-us-east-0.grafana.net:443
    basicAuth:
      username: "854348"
      password: password
metrics:
  enabled: true
  cost:
    enabled: true
  node-exporter:
    enabled: true
logs:
  enabled: true
  pod_logs:
    enabled: true
  cluster_events:
    enabled: true
traces:
  enabled: true
  receiver:
    filters:
      span:
        - attributes["http.route"] == "/live"
        - attributes["http.route"] == "/healthy"
        - attributes["http.route"] == "/ready"
    transforms:
      resource:
        - limit(attributes, 100, [])
        - truncate_all(attributes, 4096)
      span:
        - limit(attributes, 100, [])
        - truncate_all(attributes, 4096)
receivers:
  grpc:
    enabled: true
  http:
    enabled: true
  zipkin:
    enabled: true
  jaeger:
    grpc:
      enabled: true
    thriftBinary:
      enabled: true
    thriftCompact:
      enabled: true
    thriftHttp:
      enabled: true
opencost:
  enabled: true
  opencost:
    exporter:
      defaultClusterId: " cluster-alpha-us-central1"
    prometheus:
      external:
        url: https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom
kube-state-metrics:
  enabled: true
prometheus-node-exporter:
  enabled: true
prometheus-operator-crds:
  enabled: true
grafana-agent: {}
grafana-agent-logs: {}
4

That all seems ok. Are you seeing any errors in the pod logs for your app or on the alloy pods?