Description:
We are experiencing connectivity issues with Grafana Alloy’s OTLP gRPC exporter when trying to connect to Grafana Cloud’s unified OTLP endpoint. Despite following documentation and testing multiple configurations, the gRPC exporter consistently fails with resolver errors.
Environment Details:
- Grafana Alloy Version: v1.4.2
- Deployment: Kubernetes cluster (DigitalOcean)
- Grafana Cloud Instance ID: 1367037
- OTLP Endpoint: <
https://otlp-gateway-prod-us-east-2.grafana.net/otlp> - Authentication: Basic auth with instance-id:api-token (168-character glc_ token)
Issue Summary:
Grafana Alloy’s otelcol.exporter.otlp component cannot establish gRPC connections to Grafana Cloud’s unified OTLP endpoint, consistently producing “produced zero addresses” errors from the gRPC resolver.
Network Connectivity Analysis:
We performed comprehensive network testing using nmap from within our Kubernetes cluster:
# Port 4317 (standard OTLP gRPC)
PORT STATE SERVICE
4317/tcp filtered opentelemetry
# Port 443 (HTTPS/unified endpoint)
PORT STATE SERVICE
443/tcp open https
Key Finding: Port 4317 is filtered/blocked by network infrastructure, while port 443 is accessible.
Configuration Attempts and Results:
-
Original HTTPS URL (as per documentation):
endpoint =https://otlp-gateway-prod-us-east-2.grafana.net/otlp
Result: “rpc error: code = Unavailable desc = last resolver error: produced zero addresses” -
Hostname with port 443:
endpoint =otlp-gateway-prod-us-east-2.grafana.net:443
Result: “error reading server preface: http2: frame too large”
This indicates successful connection but protocol mismatch -
Hostname with port 443 and path:
endpoint =otlp-gateway-prod-us-east-2.grafana.net:443/otlp
Result: “dial tcp: lookup tcp/443/otlp: unknown port”
gRPC resolver cannot parse this format -
TLS Configuration:
tls {
insecure = false
}
Result: No improvement in any endpoint format
- Explicit Protocol Parameter:
protocol = “grpc”
Result: Segmentation fault in Grafana Alloy (panic: runtime error)
Working Configuration (Partial):
Our logging exporter works perfectly, confirming Alloy is correctly:
- Receiving OTLP data on ports 4317 (gRPC) and 4318 (HTTP)
- Processing metrics, traces, and logs (currently handling 8 log records, 3 spans)
- Authentication credentials are valid
Root Cause Analysis:
The issue appears to be that Grafana Alloy’s gRPC resolver cannot properly parse HTTPS URLs with paths. The unified OTLP endpoint format https://hostname/path is incompatible with how Alloy’s gRPC exporter expects
endpoint URLs.
Questions:
- Endpoint Format: What is the correct endpoint format for gRPC connections to the unified OTLP gateway? Should we use:
https://otlp-gateway-prod-us-east-2.grafana.net/otlpotlp-gateway-prod-us-east-2.grafana.net:443- A different format entirely?
- Protocol Support: Does the unified OTLP endpoint support native gRPC on port 443, or only HTTP/2 over HTTPS?
- Path Requirement: If using hostname:port format, how should the /otlp path be specified for gRPC requests?
- Alternative Ports: Are there dedicated gRPC ports available that bypass the URL parsing issue?
- Known Issues: Is this a known limitation with Grafana Alloy’s gRPC exporter and the unified endpoint?
