502 when trying to export data to Grafana Cloud

Grafana Cloud
, ,
1

Hi all,

I’m trying to build the monitoring infra for my web service.
I’ve verified the following architecture works:

                                  loki
                            /               \
backend ---> otel-collector ----- tempo ------> grafana
                            \               /
                                  mimir

And now I want to evaluate the Grafana Cloud because I don’t want to maintain the 4 Grafana services (loki+tempo+mimir+grafana) by myself but I couldn’t quite figure it out.
Below is my otel-collector config:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlphttp/grafana_cloud:
    endpoint: https://otlp-gateway-prod-ap-northeast-0.grafana.net/otlp
    auth:
      authenticator: basicauth/grafana_cloud

extensions:
  basicauth/grafana_cloud:
    client_auth:
      username: <instnace_id>
      password: <token>
service:
  extensions: [basicauth/grafana_cloud]
  pipelines:
    logs:
      receivers: [otlp]
      #processors: []
      exporters: [otlphttp/grafana_cloud]
    traces:
      receivers: [otlp]
      #processors: []
      exporters: [otlphttp/grafana_cloud]
    metrics:
      receivers: [otlp]
      #processors: []
      exporters: [otlphttp/grafana_cloud]

I only set the env OTEL_SERVICE_NAME: <name> and pass the collector endpoint from config file.

And here’s the error when the collector trying to export data:

2024-07-23T06:15:43.661Z info exporterhelper/retry_sender.go:118 Exporting failed. Will retry the request after interval. {“kind”: “exporter”, “data_type”: “logs”, “name”: “otlphttp/grafana_cloud”, “error”: “Throttle (0s), error: rpc error: code = Unavailable desc = error exporting items, request to https://otlp-gateway-prod-ap-northeast-0.grafana.net/otlp/v1/logs responded with HTTP Status Code 502”, “interval”: “34.082688279s”}

Not sure if I missed anything, but if anyone know how to fix it please let me know.
Thanks for the help!

1 Like
2

502 is server error (not your client error). So if you followed doc Send data using OpenTelemetry Protocol (OTLP) | Grafana Cloud documentation properly, then contact Grafana Cloud support (it seems to be their problem). Eventually, you can try to use dedicated endpoints for each signal and not more convenient single OTLP endpoint.

3

The issue was that the default access policy region is us (not sure if it’s configurable though) and my grafana stack was on japan.
After deleting the stack and recreated a new one on us-west everything worked correctly.