Hi all,
I’m trying to build the monitoring infra for my web service.
I’ve verified the following architecture works:
loki
/ \
backend ---> otel-collector ----- tempo ------> grafana
\ /
mimir
And now I want to evaluate the Grafana Cloud because I don’t want to maintain the 4 Grafana services (loki+tempo+mimir+grafana) by myself but I couldn’t quite figure it out.
Below is my otel-collector config:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
otlphttp/grafana_cloud:
endpoint: https://otlp-gateway-prod-ap-northeast-0.grafana.net/otlp
auth:
authenticator: basicauth/grafana_cloud
extensions:
basicauth/grafana_cloud:
client_auth:
username: <instnace_id>
password: <token>
service:
extensions: [basicauth/grafana_cloud]
pipelines:
logs:
receivers: [otlp]
#processors: []
exporters: [otlphttp/grafana_cloud]
traces:
receivers: [otlp]
#processors: []
exporters: [otlphttp/grafana_cloud]
metrics:
receivers: [otlp]
#processors: []
exporters: [otlphttp/grafana_cloud]
I only set the env OTEL_SERVICE_NAME: <name>
and pass the collector endpoint from config file.
And here’s the error when the collector trying to export data:
2024-07-23T06:15:43.661Z info exporterhelper/retry_sender.go:118 Exporting failed. Will retry the request after interval. {“kind”: “exporter”, “data_type”: “logs”, “name”: “otlphttp/grafana_cloud”, “error”: “Throttle (0s), error: rpc error: code = Unavailable desc = error exporting items, request to https://otlp-gateway-prod-ap-northeast-0.grafana.net/otlp/v1/logs responded with HTTP Status Code 502”, “interval”: “34.082688279s”}
Not sure if I missed anything, but if anyone know how to fix it please let me know.
Thanks for the help!