Failed to push trace samples to the cloud

Grafana k6
1

Following the Correlate results with Grafana Cloud Traces documentation, I’m attempting to enable tracing for my k6 tests.

import tempo from 'https://jslib.k6.io/http-instrumentation-tempo/1.0.0/index.js';

tempo.instrumentHTTP({
    propagator: 'w3c'
});

However, I’m getting the following errors when running my tests:
ERRO[0018] Failed to push trace samples to the cloud error="failed to ingest request metadatas batch: code=Unavailable, msg=connection error: desc = \"error reading server preface: http2: frame too large\"" output=cloudv2 t=2.7847398s

I discovered the K6_CLOUD_TRACES_ENABLED environment variable via a related forums post, but setting that to true didn’t seem to have any effect (curious that this isn’t in the document I linked above?). I’m also not aware of any proxy settings that would be blocking the endpoint referenced in the aforementioned forum post.

It’s my understanding that calling instrumentHTTP just adds the necessary headers to the outbound HTTP requests, so I’m not sure what is attempting to be pushed to the cloud.

I’m running my tests locally, making requests to an auto-instrumented .NET service. In Grafana Cloud, I can see the traces, but the “Service” is <root span not yet received>. My traces are being collected by a Grafana Alloy instance deployed in cluster (managed by the k8s-monitoring Helm chart) and forwarded to Grafana Cloud.

image
image3158×388 78.4 KB

2

Okay so I was able to make some good progress on this. The original error was because I was running a <1.0 version of k6 locally (strange error to get from running an old version?). After upgrading to the latest version, I started getting the following:

ERRO[0020] Failed to push trace samples to the cloud error="failed to ingest request metadatas batch: code=Unavailable, msg=connection error: desc = \"transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property. If you upgraded from a grpc-go version earlier than 1.67, your TLS connections may have stopped working due to ALPN enforcement. For more details, see: https://github.com/grpc/grpc-go/issues/434\"" output=cloudv2 t=3.0583456s

This appears to be some cert validation issue. Running the tests in Docker with the k6 base image resolved that issue as well. As of now, I can run the tests without error and view the traces from “Traces” tab in the Grafana Cloud k6 UI. However, I’m still seeing <root span not yet received> for the “Service” when exploring the traces. Anybody have an idea what could be causing this?

4

Hi @cameronwaterman,

I’m glad to see that you managed to make some progress and resolve the issues you encountered along the process.

Now, regarding the <root span not yet received> you still see, I think that’s “expected”, as k6 is the trace initiator, but it is not sending the trace data to Tempo, as it just adds the HTTP headers that would subsequently cause downstream instrumented services to participate in the trace.

Tempo assumes it’ll receive the root span at some point, which would then also allow it to work out which service initiated the trace, and so it’ll display that message until that happens, which it never will in this case.

I hope that helps you understanding this confusing situation! :person_bowing:

Cheers!