# K6-operator failed to send metrics to prometheus

**URL:** <https://community.grafana.com/t/k6-operator-failed-to-send-metrics-to-prometheus/113460>\
**Category:** k6-operator\
**Created:** [February 1, 2024, 7:54pm UTC](https://community.grafana.com/t/k6-operator-failed-to-send-metrics-to-prometheus/113460 "2024-02-01T19:54:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![zzhao2022](https://avatars.discourse-cdn.com/v4/letter/z/439d5e/32.png) [@zzhao2022](https://community.grafana.com/u/zzhao2022)\
**Post date:** [February 1, 2024, 7:54pm UTC](https://community.grafana.com/t/k6-operator-failed-to-send-metrics-to-prometheus/113460/1 "2024-02-01T19:54:07Z")

</div>

Hi, I was testing to send test metrics to Prometheus. Somehow the k6 pods failed to send metrics with error in the log. Any suggestion?

```auto
 demo-1-rdg95 time="2024-02-02T07:13:30Z" level=error msg="Failed to send the time series data to the endpoint" error="HTTP POST request failed: Post \"http://prometheus-kube-prometheus-prometheus:9090/api/v1/write\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" output="Prometheus remote write" │
│ demo-1-rdg95 time="2024-02-02T07:13:30Z" level=warning msg="Successful flushed time series to remote write endpoint but it took 5.002130252s while flush period is 5s. Some samples may be dropped." nts=15 output="Prometheus remote write"

```

I installed k6-operator and prometheus on a local minikube k8s. I used kube-prometheus-stack-56.6.0 helm chart for deploying Prometheus.  
The remove write receiver was enabled on Prometheus side. And I was able to verify that it worked by port-forwarding the Prometheus pod and triggering a test from my local. The metrics were reporting to Prometheus correctly…

```auto
alertmanager:
  enabled: false

prometheus:
  enabled: true
  prometheusSpec:
    ## enable --web.enable-remote-write-receiver flag on prometheus-server
    enableRemoteWriteReceiver: true

    # EnableFeatures API enables access to Prometheus disabled features.
    # ref: https://prometheus.io/docs/prometheus/latest/disabled_features/
    enableFeatures:
      - native-histograms

```

However, the same test failed to send metrics to Prometheus when I triggered it via k6-operator. In the k6 executor pod, I see error above. Below is the yaml file:

```auto
apiVersion: k6.io/v1alpha1
kind: K6
metadata:
  name: demo
spec:
  parallelism: 1
  cleanup: post
  arguments: -o experimental-prometheus-rw --tag testid=demo_test
  script:
    configMap:
      name: demo
      file: test.js
  runner:
    env:
      - name: K6_PROMETHEUS_RW_SERVER_URL
        value: "http://prometheus-kube-prometheus-prometheus:9090/api/v1/write"
      - name: K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM
        value: "true"

```

I also tested it the connection from another pod on the same k8s:

```auto
prometheus-grafana-9c98f646b-7h2mg:/usr/share/grafana$ curl -v -X POST http://prometheus-kube-prometheus-prometheus:9090/api/v1/write
* Host prometheus-kube-prometheus-prometheus:9090 was resolved.
* IPv6: (none)
* IPv4: 10.104.33.165
* Trying 10.104.33.165:9090...
* Connected to prometheus-kube-prometheus-prometheus (10.104.33.165) port 9090
> POST /api/v1/write HTTP/1.1
> Host: prometheus-kube-prometheus-prometheus:9090
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 02 Feb 2024 07:26:56 GMT
< Content-Length: 22
<
snappy: corrupt input

```

---

<div class="post-metadata">

**Author:** ![bandorko](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/bandorko/32/38733_2.png) [@bandorko](https://community.grafana.com/u/bandorko)\
**Post date:** [February 2, 2024, 8:45am UTC](https://community.grafana.com/t/k6-operator-failed-to-send-metrics-to-prometheus/113460/3 "2024-02-02T08:45:53Z")

</div>

Hi @zzhao2022 !

I assume, that the prometheus-kube-prometheus-prometheus service and the prometheus-grafana-9c98f646b-7h2mg pod are in the same namespace, but  
are the k6 pod (demo-1-rdg95) and the prometheus-kube-prometheus-prometheus service in the same namespace? If not, you should try prometheus-kube-prometheus-prometheus..svc.cluster.local instead.

I hope it helps

---

<div class="post-metadata">

**Author:** ![zzhao2022](https://avatars.discourse-cdn.com/v4/letter/z/439d5e/32.png) [@zzhao2022](https://community.grafana.com/u/zzhao2022)\
**Post date:** [February 2, 2024, 9:33am UTC](https://community.grafana.com/t/k6-operator-failed-to-send-metrics-to-prometheus/113460/4 "2024-02-02T09:33:50Z")

</div>

@bandorko Thank you for your quick response. Yes, turned out it was the fix. I see metrics are reporting to prometheus after I updated the endpoint with namespace.svc.cluster.local.

One more question regarding the `prometheus.enabled` value in the [k6-operator chart](https://github.com/grafana/k6-operator/blob/main/charts/k6-operator/README.md). What does it do? Does it have to be enabled for metrics reporting to Prometheus?
