The flush operation took higher than the expected set push interval

Grafana k6
1

Hi,

I am using K6 in a Docker with influx db and Grafana dashboard.

I am using rampingArrivalRate as shown below.

When I run the tests, in the containers I see lots of the following warnings. Anyone know what this means and how I can fix it please?

time="2021-12-02T10:48:56Z" level=warning msg="The flush operation took higher than the expected set push interval. If you see this message multiple times then the setup or configuration need to be adjusted to achieve a sustainable rate." output=InfluxDBv1 t=1.67979745s

Here are the options I am using in the test:

export let options = {
  scenarios: {
    successfulLogin_rampingArrivalRate: {
      executor: 'ramping-arrival-rate',
      startRate: 280,
      timeUnit: '5m',
      preAllocatedVUs: 2,
      maxVUs: 8,
      stages: [
        { target: 280, duration: '1m' }, //Ramp up
        { target: 280, duration: '5m' }, //Stay at same level
        { target: 0, duration: '30s' }, // Ramp down
      ],
    },
  },
3

Hi @MattK6,
the requests for flushing metrics to InfluxDB are taking longer (1.6s in your reported case) of the set flush interval (default to 1s) so it means that k6 is probably flushing more metrics than InfluxDB can really handle.
Do you see a high value of CPU and/or RAM for your InfluxDB instance? Are you using the same machine for running k6 and InfluxDB or are they separated?

I list here some steps to try for getting a better response time from InfluxDBv1:

  • You could set only the SystemTags that you really need for your analysis. This can help for reducing the number of metrics that k6 is generating and InfluxDB have to ingest.
  • Tune the options of the k6 output for InfluxDB:
    • You could consider excluding some tags to be indexed K6_INFLUXDB_TAGS_AS_FIELDS that should speed up the ingestion phase for InfluxDB. Note that not indexed tags could be slow when you need to query them.
    • Reduce the K6_INFLUXDB_PUSH_INTERVAL and increase the K6_INFLUXDB_CONCURRENT_WRITES options for flushing batches with a smaller number of metrics. You probably will need to check the logs with the –verbose option for setting the right values.
  • Try to use Telegraf for aggregating some metrics before sending them to InfluxDB.

If none of the previous suggestions works for you then you could try the new InfluxDBv2 extension and see if it’s able to ingest your metrics faster.

2 Likes
When performing extensive read and write operations, InfluxDB encounters delays in real-time data writing
5

Hey man!

Would you please be so kind and explain (an example) how do manipulate these:

K6_INFLUXDB_PUSH_INTERVAL
K6_INFLUXDB_CONCURRENT_WRITES

Is it via the command line? Like that:

$ k6 run K6_INFLUXDB_PUSH_INTERVAL=blah script.js
6

Hi @fatkobra,
you can find documentation about how to use the options at the doc page Using options.

So, as the example mention, in the case of shell you can use:

$ K6_INFLUXDB_PUSH_INTERVAL=1s k6 run script.js
1 Like
7

Thank you so much @codebien for your input.

I was able to successfully use K6_INFLUXDB_PUSH_INTERVAL=2s and see the positive changes.

K6_INFLUXDB_PUSH_INTERVAL=2s k6 run --out influxdb=https://influxdb.blah.blahblah:portNo/somedb script.js --quiet
1 Like
8

Where do I do this?

My K6 version is, k6 v0.56.0 (commit/50afb99947, go1.23.4, linux/amd64) and influx-db version is 1.11.8.

I was running K6 script from bash like this, k6 run --out influxdb=http://localhost:8086/db_name sample_tests.js. Do I need to specify the the tags that need to be pushed to InfluxDB in a .config file somewhere? Can you point me to docs for this please, preferrably with an example. I couldn’t figure out much from the SystemTags that you had linked above, as I was concerned primarily with sending/pushing tags to InfluxDB.

Thanks.

9

Hey @naveenmarthala1,
here there is the up to date link for Options.SystemTags. Please, check it and let we know if it works for you.

1 Like
10

Thanks for the reply. In the section that opens from the link attached, I see proto ,subproto ,status ,method ,url ,name ,group ,check ,error ,error_code ,tls_version ,scenario ,service ,expected_response, however, to InfluxDB, metrics like http_req_duration, http_req_connecting are what I’m expecting to push and show up in my grafana dashboard. Where do I find the list of tags or metrics that are sent to influxDB via the flag --out (of k6 run) and how do I restrict that list of metrics? I’m sure I’m missing something here.