K6-operator failed to send metrics to prometheus

Grafana k6 k6-operator
1

Hi, I was testing to send test metrics to Prometheus. Somehow the k6 pods failed to send metrics with error in the log. Any suggestion?

 demo-1-rdg95 time="2024-02-02T07:13:30Z" level=error msg="Failed to send the time series data to the endpoint" error="HTTP POST request failed: Post \"http://prometheus-kube-prometheus-prometheus:9090/api/v1/write\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" output="Prometheus remote write"                                                    │
│ demo-1-rdg95 time="2024-02-02T07:13:30Z" level=warning msg="Successful flushed time series to remote write endpoint but it took 5.002130252s while flush period is 5s. Some samples may be dropped." nts=15 output="Prometheus remote write"

I installed k6-operator and prometheus on a local minikube k8s. I used kube-prometheus-stack-56.6.0 helm chart for deploying Prometheus.
The remove write receiver was enabled on Prometheus side. And I was able to verify that it worked by port-forwarding the Prometheus pod and triggering a test from my local. The metrics were reporting to Prometheus correctly…

alertmanager:
  enabled: false

prometheus:
  enabled: true
  prometheusSpec:
    ## enable --web.enable-remote-write-receiver flag on prometheus-server
    enableRemoteWriteReceiver: true

    # EnableFeatures API enables access to Prometheus disabled features.
    # ref: https://prometheus.io/docs/prometheus/latest/disabled_features/
    enableFeatures:
      - native-histograms

However, the same test failed to send metrics to Prometheus when I triggered it via k6-operator. In the k6 executor pod, I see error above. Below is the yaml file:

apiVersion: k6.io/v1alpha1
kind: K6
metadata:
  name: demo
spec:
  parallelism: 1
  cleanup: post
  arguments: -o experimental-prometheus-rw --tag testid=demo_test
  script:
    configMap:
      name: demo
      file: test.js
  runner:
    env:
      - name: K6_PROMETHEUS_RW_SERVER_URL
        value: "http://prometheus-kube-prometheus-prometheus:9090/api/v1/write"
      - name: K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM
        value: "true"

I also tested it the connection from another pod on the same k8s:

prometheus-grafana-9c98f646b-7h2mg:/usr/share/grafana$ curl -v -X POST http://prometheus-kube-prometheus-prometheus:9090/api/v1/write
* Host prometheus-kube-prometheus-prometheus:9090 was resolved.
* IPv6: (none)
* IPv4: 10.104.33.165
*   Trying 10.104.33.165:9090...
* Connected to prometheus-kube-prometheus-prometheus (10.104.33.165) port 9090
> POST /api/v1/write HTTP/1.1
> Host: prometheus-kube-prometheus-prometheus:9090
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 02 Feb 2024 07:26:56 GMT
< Content-Length: 22
<
snappy: corrupt input
3

Hi @zzhao2022 !

I assume, that the prometheus-kube-prometheus-prometheus service and the prometheus-grafana-9c98f646b-7h2mg pod are in the same namespace, but
are the k6 pod (demo-1-rdg95) and the prometheus-kube-prometheus-prometheus service in the same namespace? If not, you should try prometheus-kube-prometheus-prometheus..svc.cluster.local instead.

I hope it helps

1 Like
4

@bandorko Thank you for your quick response. Yes, turned out it was the fix. I see metrics are reporting to prometheus after I updated the endpoint with namespace.svc.cluster.local.

One more question regarding the prometheus.enabled value in the k6-operator chart. What does it do? Does it have to be enabled for metrics reporting to Prometheus?