I noticed that the last batch of metrics fails to upload to an Otel Collector.
Both gRPC and HTTP protocols have the same behavior.
To replicate set the export interval to 1 second, K6_OTEL_EXPORT_INTERVAL=1. (The default 10s interval doesn’t log if the test is completed, but all metrics from the final batch are not sent.)
I believe, in that case, k6 doesn’t control the metrics upload. It delegates it entirely to the OTEL SDK. By checking it, I see that there is a configuration option:
// WithTimeout configures the time a PeriodicReader waits for an export to
// complete before canceling it. This includes an export which occurs as part
// of Shutdown or ForceFlush if the user passed context does not have a
// deadline. If the user passed context does have a deadline, it will be used
// instead.
//
// This option overrides any value set for the
// OTEL_METRIC_EXPORT_TIMEOUT environment variable.
//
// If this option is not used or d is less than or equal to zero, 30 seconds
// is used as the default.
Right now, we don’t provide the overwriting it on the k6 side, so you could try directly using OTEL_METRIC_EXPORT_TIMEOUT.
I tried using OTEL_METRIC_EXPORT_TIMEOUT, but it had no impact. Even just testing 1 VU with 100 iterations, the report summary will confirm the completion of the 100 iterations, but the OTLP receivers will miss 1-2% of them, and you get this log at the end:
After looking again at your original message, I believe the issue is that you don’t specify the unit for the export interval K6_OTEL_EXPORT_INTERVAL=1. This resolves to not 1s (second) but 1ms (millisecond), which is too short.
Could you please try using K6_OTEL_EXPORT_INTERVAL=1s.
Ah yes, you’re right it needs the unit.
I tried passing K6_OTEL_EXPORT_INTERVAL=1s unfortunately more metrics were not sent (~80 of 100). I also tested using the default (10s) but lost more metrics (~60 of 100).
It seems that the increased duration of the K6_OTEL_EXPORT_INTERVAL results in more metrics potentially being lost at the end.
I’m guessing that the k6 exporter ends as soon as the main function exits, thus missing to push the remaining metrics to OTLP. Thoughts?
I tried adding the option --linger to see if it completes the push, but did not work.
I tried passing K6_OTEL_EXPORT_INTERVAL=1s unfortunately more metrics were not sent (~80 of 100). I also tested using the default (10s) but lost more metrics (~60 of 100).
It seems that the increased duration of the K6_OTEL_EXPORT_INTERVAL results in more metrics potentially being lost at the end.
Is there a way to get the script that can be used to reproduce this?
To reproduce, I need to understand what I should reproduce. After fixing the export interval, the error is gone, which I see on my end, and you just confirmed that, but if the issue remains, I need other details like the script and the exact issue.
Like:
I tried passing K6_OTEL_EXPORT_INTERVAL=1s unfortunately more metrics were not sent (~80 of 100). I also tested using the default (10s) but lost more metrics (~60 of 100).
What are these metrics you lost when detecting? How do you measure the loss? All that said, in order to help you, I need as many details as you can provide.