K6-sample pod shows logs for few seconds, but later gives connection error

Grafana k6 k6-operator
1

I have 2 test scripts -

  1. http.js - to test the kube-api service https://kubernetes.default.svc/api
  2. create-pod.js - to create pods with high cpu load referenced from xk6-kubernetes/examples/pod_operations.js at main · grafana/xk6-kubernetes · GitHub

for the http.js script, the initializer, starter and sample pods gets created.

kubectl get pods
NAME READY STATUS
pod/k6-sample-1-89pvh 1/1 Running
pod/k6-sample-initializer-nn5bw 0/1 Completed
pod/k6-sample-starter-spl7r 0/1 Completed

the sample pods, shows logs for few seconds, later it gives connection issues or immediately goes in containerstatusunknown state

kubectl logs pod/k6-sample-1-89pvh
time=“2024-01-11T19:57:51Z” level=warning msg=“Request Failed” error=“Get "https://kubernetes.default.svc/api\”: tls: failed to verify certificate: x509: certificate signed by unknown authority"
time=“2024-01-11T19:57:51Z” level=info msg=“Status Code: 0” source=console
time=“2024-01-11T19:57:51Z” level=info msg=“Response Body: null” source=console

after 2s :

kubectl logs pod/k6-sample-1-89pvh
Error from server: Get “https://<>:10250/containerLogs/k6-demo/k6-sample-1-89pvh/k6”: dial tcp <>:10250: connect: connection refused

kubectl get pods
NAME READY STATUS
pod/k6-sample-1-89pvh 0/1 ContainerStatusUnknown
pod/k6-sample-initializer-nn5bw 0/1 Completed
pod/k6-sample-starter-spl7r 0/1 Completed

  1. for the create-pod.js, it does not show any logs
    in few cases, the describe pod shows readiness probe getting failed

kubectl describe pod/k6-sample-1-89pvh

Events:
Type Reason Age From Message

Normal Created 102s kubelet Created container k6
Normal Started 102s kubelet Started container k6
Warning Unhealthy 100s (x2 over 101s) kubelet Readiness probe failed: Get “http://<>:6565/v1/status”: dial tcp <>:6565: connect: connection refused
Warning FailedMount 48s kubelet MountVolume.SetUp failed for volume “k6-test-volume” : object “k6-demo”/“k6-test” not registered
Warning FailedMount 47s kubelet MountVolume.SetUp failed for volume “kube-api-access-75vxj” : object “k6-demo”/“kube-root-ca.crt” not registered

Any suggestions to resolve both the issues is appreciated.
Thanks

3

I face same issue as you, Anyone help us pleas

6

Hi @gunjanvmirchandani,
Your listings show errors related to TLS setup:

time=“2024-01-11T19:57:51Z” level=warning msg=“Request Failed” error=“Get "https://kubernetes.default.svc/api\”: tls: failed to verify certificate: x509: certificate signed by unknown authority"
...
Warning FailedMount 47s kubelet MountVolume.SetUp failed for volume “kube-api-access-75vxj” : object “k6-demo”/“kube-root-ca.crt” not registered

These errors are related to your Kubernetes setup; it’s not about k6 or k6-operator. How do you deploy your cluster? I’d suggest to double-check how the cluster is created, where is root CA certificate, how TLS is configured, with what tooling and steps, etc. Some resources:

It makes sense to also check the version of Kubernetes (and / or cloud service if you’re using one) and double-check if there are any issues with it.

1 Like