Unable to send the metric to prometheus

Hello folks,
I am trying to send my k6 metrics to prometheus for that i have done the following thing.

  1. i have installed prometheus and it is up and running at port 9090.
  2. after that i am using command “k6 run -o experimental-prometheus-rw sampleScript.js”

I am getting following error :
ERRO[0003] Failed to send the time series data to the endpoint error=“got status code: 404 instead expected a 2xx successful status code” output=“Prometheus remote write”

Do i have to make any configuration in prometheus , or m i missing somethings , please help!!

Hi @ajay

Welcome to the community forum :wave:

I understand you are running experimental prometheus remote write output. It does seem that prometheus’ remote endpoint write API is not “located” at the default K6_PROMETHEUS_RW_SERVER_URL, which is http://localhost:9090/api/v1/write.

I would suspect your Prometheus does not have remote write receiver enabled, which is usually done with --web.enable-remote-write-receiver, as documented in Prometheus remote write.

You can check this with

curl -X POST http://localhost:9090/api/v1/write

Which will return remote write receiver needs to be enabled with --enable-feature=remote-write-receiver. Note that the flag has been deprecated and is now --web.enable-remote-write-receiver.

If this is the case, running curl -v -X POST http://localhost:9090/api/v1/write, in verbose, you would also see that your Prometheus is returning a HTTP/1.1 404 Not Found. Same as the errors you see got status code: 404.

Let me know if that helps.

Cheers!

Yes it helps , thanks a lot @eyeveebe

1 Like

How did you enable --enable-feature=remote-write-receive

Can you guide me through it!

1 Like

Hi @Mitul

Welcome to the community forum :wave:

Does this page in the docs help: Prometheus remote write? To enable the remote write flag in Prometheus you have to follow the Prometheus docs. Or are you stuck at another step?

Cheers!

Where do we enable the remote write flag. I have hosted prometheus server locally on localhost:9090 and i am trying to send a custom k6 metrics to the server.

import { Counter } from 'k6/metrics';
import http from 'k6/http';

const myCounter = new Counter('my_counter');

export default function () {
  myCounter.add(1);
  myCounter.add(2, { tag1: 'myValue', tag2: 'myValue2' });
}

export function handleSummary(data) {
  console.log('Preparing the end-of-test summary...');

  const metricsPayload = `
    # TYPE my_counter counter
    # HELP my_counter My Counter
    my_counter{label1="value1", label2="value2"} ${data.metrics.my_counter.values.count}
  `;

  const url = 'http://localhost:9090/api/v1/writex';
  const params = {
    headers: { 'Content-Type': 'text/plain' },
    body: metricsPayload,
  };

  const response = http.post(url, params);

  if (response.status === 200) {
    console.log('Metrics sent successfully');
  } else {
    console.error('Failed to send metrics:', response.body);
  }
}

This is the YAML file:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

remote_write:
  - url: "http://localhost:9090/api/v1/write"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "k6"
    static_configs:
      - targets: ["localhost:8082"]

Is there any issue with the above code?
on running the k6 script im getting the error - ERRO[0000] Failed to send metrics: remote write receiver needs to be enabled with --web.enable-remote-write-receiver source=console

Hi @Mitul

Apologies for the late reply. This is in the realm of Prometheus configuration, so you would have to see how to add this to your Prometheus configuration. Not on the k6 side.

If you don’t manage the Prometheus instance, ask for help in following the Prometheus docs for your concrete installation: Feature flags | Prometheus

Cheers!

Yeah this doesn’t work for me either. I have added it to the systemd config and the prometheus.yaml:

curl -X POST http://localhost:9090/api/v1/write
remote write receiver needs to be enabled with --web.enable-remote-write-receiver

@eyeveebee I did enable the required flags on Prometheus’ side:

❯ cat prom-stack-values.yaml
alertmanager:
  enabled: false
prometheus:
  prometheusSpec:
    enableRemoteWriteReceiver: true
    enableFeatures:
      - remote-write-receiver
      - web.enable-remote-write-receiver
      - enable-feature=native-histograms

However when I run the curl command to verify, I received… Any idea?

❯ curl -v -X POST http://localhost:9090/api/v1/write
*   Trying [::1]:9090...
* Connected to localhost (::1) port 9090
> POST /api/v1/write HTTP/1.1
> Host: localhost:9090
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 02 Feb 2024 00:08:08 GMT
< Content-Length: 22
<
snappy: corrupt input
* Connection #0 to host localhost left intact

Here is my systemd startup script:

[Unit]
 Description=Prometheus
 Documentation=https://prometheus.io/docs/introduction/overview/
 Wants=network-online.target
 After=network-online.target

 [Service]
 Type=simple
 Environment="GOMAXPROCS=8"
 User=prometheus
 Group=prometheus
 ExecReload=/bin/kill -HUP $MAINPID
 ExecStart=/usr/local/bin/prometheus \
--enable-feature=remote-write-receiver \
 --config.file=/etc/prometheus/prometheus.yml \
 --storage.tsdb.path=/var/lib/prometheus \
 --storage.tsdb.retention.time=24h \
 --web.console.templates=/etc/prometheus/consoles \
 --web.console.libraries=/etc/prometheus/console_libraries \
 --web.listen-address=0.0.0.0:9090 \
 --web.external-url=

 SyslogIdentifier=prometheus
 Restart=always

 [Install]
 WantedBy=multi-user.target

And my Prometheus config:

global:
  scrape_interval: 1m # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 1m # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

storage:
  tsdb:
    out_of_order_time_window: 1d

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"
    honor_timestamps: false

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]