K6 Failed to push trace samples to the cloud

Grafana k6
1

I am attempting to enable traces in our k6 cloud run tests following the documentation here: Correlate results with Grafana Cloud Traces | Grafana Cloud documentation

The service endpoint I am calling in the K6 test has otel traces enabled (confirmed this as I can see the test traffic in Grafana Cloud Traces when running the script locally).

My k6 script includes the following:
import tempo from ‘https://jslib.k6.io/http-instrumentation-tempo/1.0.0/index.js’;

tempo.instrumentHTTP({
// possible values: “w3c”, “jaeger”
propagator: ‘w3c’,
});

And I confirmed that the test request has the Traceparent request header.

However, when I attempt to run the k6 script with the following:
k6 cloud run --local-execution .\DocMgmt_svc.js -e scenario=smoke_test -e K6_CLOUD_TRACES_ENABLED=true

I see the following output in the terminal, and no trace data is available in the Grafana Cloud test run:
ERRO[0012] Failed to push trace samples to the cloud error=“failed to ingest request metadatas batch: code=Unavailable, msg=connection error: desc = "error reading server preface: http2: frame too large"” output=cloudv2 t=2.7082311s

2

Thanks for sharing the issue! For future reference and to help others who might run into the same problem: the error message was quite misleading. The root cause was that the URL https://grpc-k6-api-prod-prod-us-east-0.grafana.net:443, which is required for this feature, was blocked by a proxy. This URL is used to send metadata about HTTP requests and tracing, allowing k6 cloud backend to correlate them with the service under test tracing.

2 Likes
3

In addition to the note above, we were running into a second issue with the error:

ERRO[0111] Failed to push trace samples to the cloud error=“failed to ingest request metadatas batch: code=Unavailable, msg=connection error: desc = "transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property"” output=cloudv2 t=2.8764ck peer: missing selected ALPN property"" output=cloudv2 t=2.8764534s

We ended up being able to resolve this by enabling SSL_Bypass on our web proxy. Apparently the decryption/inspection was causing issues, and disabling that for the traffic going to https://grpc-k6-api-prod-prod-us-east-0.grafana.net:443 resolved it.