How to generate variable load using multiple scenarios and by changing stream size and not number of iterations (requests/second)

I am using K6 with “k6/x/loki” extension to test Loki so that I can configure it to meet my requirements. Loki is installed in K8S cluster. The tests are running in same cluster.

I am trying to create a test with constant requests/second and variable stream size between 1MB and 10 MB. I have tried various approaches for this and so far, haven’t achieved the expected results.

Please note that I am not trying to increase load by increasing number of requests per second here. I am increasing load by increasing stream size. That’s why I have kept rate constant in each scenario.

Given below is K6 custom resource configuration:

Name:         k6-test-run-6343683
Namespace:    k6-test-run-6343683
Labels:       <none>
Annotations:  <none>
API Version:  k6.io/v1alpha1
Kind:         TestRun
Spec:
  Arguments:  --log-output=none
  Initializer:
    Image:  k6-image
    Node Selector:
      some-node-selector:  true
    Tolerations:
      Effect:    NoSchedule
      Key:       some-toleration-key
      Operator:  Equal
      Value:     true
  Parallelism:   10
  Runner:
    Image:    k6-image
    Node Selector:
      some-nodeselector:  true
    Tolerations:
      Effect:    NoSchedule
      Key:       some-toleration-key
      Operator:  Equal
      Value:     true
  Script:
    Config Map:
      File:  script.js
      Name:  k6-test-run-6343683
  Separate:  true

Here are the important parts from test script:
Scenarios:

scenarios: {
    averageLoad1: {
      startTime: "0s",
      executor: "constant-arrival-rate",
      exec: "writePath",
      rate: 100,
      timeUnit: "1s",
      preallocatedVUs: 1000,
      maxVUs: 1100,
      duration: "10m"
    },
    spike3MB: {
      startTime: "10m",
      executor: "constant-arrival-rate",
      exec: "writePath3",
      rate: 100,
      timeUnit: "1s",
      preallocatedVUs: 1000,
      maxVUs: 1100,
      duration: "10m"
    },
    averageLoad2: {
      startTime: "20m",
      executor: "constant-arrival-rate",
      exec: "writePath",
      rate: 100,
      timeUnit: "1s",
      preallocatedVUs: 1000,
      maxVUs: 1100,
      duration: "10m"
    },
    spike5MB: {
      startTime: "30m",
      executor: "constant-arrival-rate",
      exec: "writePath5",
      rate: 100,
      timeUnit: "1s",
      preallocatedVUs: 1000,
      maxVUs: 1100,
      duration: "10m"
    },
    averageLoad3: {
      startTime: "40m",
      executor: "constant-arrival-rate",
      exec: "writePath",
      rate: 100,
      timeUnit: "1s",
      preallocatedVUs: 1000,
      maxVUs: 1100,
      duration: "10m"
    },
    spike7MB: {
      startTime: "50m",
      executor: "constant-arrival-rate",
      exec: "writePath7",
      rate: 100,
      timeUnit: "1s",
      preallocatedVUs: 1000,
      maxVUs: 1100,
      duration: "10m"
    },
    averageLoad4: {
      startTime: "60m",
      executor: "constant-arrival-rate",
      exec: "writePath",
      rate: 100,
      timeUnit: "1s",
      preallocatedVUs: 1000,
      maxVUs: 1100,
      duration: "10m"
    },
    spike10MB: {
      startTime: "70m",
      executor: "constant-arrival-rate",
      exec: "writePath10",
      rate: 100,
      timeUnit: "1s",
      preallocatedVUs: 1000,
      maxVUs: 1100,
      duration: "10m"
    },
    averageLoad5: {
      startTime: "80m",
      executor: "constant-arrival-rate",
      exec: "writePath",
      rate: 100,
      timeUnit: "1s",
      preallocatedVUs: 1000,
      maxVUs: 1100,
      duration: "10m"
    }  }

Test Functions:

const size1 = (1 * 1024 * 1024);

const size3 = (3 * 1024 * 1024);

const size5 = (5 * 1024 * 1024);

const size7 = (7 * 1024 * 1024);

const size10 = (10 * 1024 * 1024);

export function writePath() {
  const res = lokiDistributorClient.pushParameterized(1, size1, (size1 + 1));

  WriteOK.add(res.status === 204);
}

export function writePath3() {
  const res3 = lokiDistributorClient.pushParameterized(1, size3, (size3 + 1));

  WriteOK.add(res3.status === 204);
}

export function writePath5() {
  const res5 = lokiDistributorClient.pushParameterized(1, size5, (size5 + 1));

  WriteOK.add(res5.status === 204);
}

export function writePath7() {
  const res7 = lokiDistributorClient.pushParameterized(1, size7, (size7 + 1));

  WriteOK.add(res7.status === 204);
}

export function writePath10() {
  const res10 = lokiDistributorClient.pushParameterized(1, size10, (size10 + 1));

  WriteOK.add(res10.status === 204);
}

Expected result:
I was expecting with above test design, the Distributor Received bytes / sec graph will be somewhat like:

Actual result:
But the Distributor Received bytes / sec graph came up like:

Can anybody please help me to figure out what am I doing incorrectly here?

@taraktrivedi

I don’t know what has happened here. The average load should be 100 MB/sec (but it is 25 MB/sec), the spike3MB is 300MB/sec (I think it is correct), but the 5MB ,7MB and 10MB seem not running at all. Maybe are there some error in the logs?

1 Like

Thanks @bandorko !

You are correct. Loki Gateway is throwing an error

client intended to send too large body

To solve this error I will have to either break the requests in to multiple small requests or increase value of max_body_size of Loki Gateway. As I want to test Loki with large size streams, I am going to try second option.

The problem I am facing right now is, I am using loki-distributed and it does not expose max_body_size parameter. So, I am looking into how can I set it even if it is not exposed by loki-distributed.