Hey guys, please I need help with getting traces to grafana cloud, I already deployed grafana monitoring to the cluster, I’m getting logs and I think to some extent metrics but I’m not getting traces at all even after instrumenting my app. I tested the traces locally and it’s working when I used my local endpoint, but when I deployed the app to cluster and used the endpoint provided by Grafana during configuration, it doesn’t work. Thanks
Can you share the config you used with your deployment? I can try and determine which endpoint you want to point your applications to.
Observility .env
OTEL_EXPORTER_OTLP_ENDPOINT="http://grafana-k8s-monitoring-grafana-agent.default.svc.cluster.local:4318"
OTEL_EXPORTER_OTLP_PROTOCOL="http"
OTEL_TRACES_EXPORTER="otlp"
OTEL_NODE_RESOURCE_DETECTOR="env,host,os"
cluster:
name: "cluster-alpha-us-central1"
externalServices:
prometheus:
host: https://prometheus-prod-13-prod-us-east-0.grafana.net
basicAuth:
username: "1518595"
password: password
loki:
host: https://logs-prod-006.grafana.net
basicAuth:
username: "860032"
password: password
tempo:
host: https://tempo-prod-04-prod-us-east-0.grafana.net:443
basicAuth:
username: "854348"
password: password
metrics:
enabled: true
cost:
enabled: true
node-exporter:
enabled: true
logs:
enabled: true
pod_logs:
enabled: true
cluster_events:
enabled: true
traces:
enabled: true
receiver:
filters:
span:
- attributes["http.route"] == "/live"
- attributes["http.route"] == "/healthy"
- attributes["http.route"] == "/ready"
transforms:
resource:
- limit(attributes, 100, [])
- truncate_all(attributes, 4096)
span:
- limit(attributes, 100, [])
- truncate_all(attributes, 4096)
receivers:
grpc:
enabled: true
http:
enabled: true
zipkin:
enabled: true
jaeger:
grpc:
enabled: true
thriftBinary:
enabled: true
thriftCompact:
enabled: true
thriftHttp:
enabled: true
opencost:
enabled: true
opencost:
exporter:
defaultClusterId: " cluster-alpha-us-central1"
prometheus:
external:
url: https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom
kube-state-metrics:
enabled: true
prometheus-node-exporter:
enabled: true
prometheus-operator-crds:
enabled: true
grafana-agent: {}
grafana-agent-logs: {}
That all seems ok. Are you seeing any errors in the pod logs for your app or on the alloy pods?