K6-operator error, not spawning starter and basic resources

Grafana k6 k6-operator
1

Hi!
I am having some issues when I try to use env variables that we store on Vault.
I see the initializer resource but starter and basic never start .

This is the error I see on k6-operator manager

"error": "invalid character 'i' in literal true (expecting 'r')"

time="2023-12-26T12:38:42Z" level=info msg="spawning process: [sh -c mkdir -p $(dirname /tmp/k6_script.js.archived.tar) && k6 archive /test/k6_script.js -O /tmp/k6_script.js.archived.tar  2> /tmp/k6logs && k6 inspect --execution-requirements /tmp/k6_script.js.archived.tar 2> /tmp/k6logs ; ! cat /tmp/k6logs | grep 'level=error']" app=vault-env
{
..test data..
}`	{"namespace": "k6-test-te", "name": "k6-basic", "reconcileID": "11bc84a2-80be-47cf-9ae5-b7b65862d5d5", "error": "invalid character 'i' in literal true (expecting 'r')"}
github.com/grafana/k6-operator/controllers.inspectTestRun
	/workspace/controllers/common.go:94
github.com/grafana/k6-operator/controllers.RunValidations
	/workspace/controllers/k6_initialize.go:53
github.com/grafana/k6-operator/controllers.(*K6Reconciler).Reconcile
	/workspace/controllers/k6_controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235

this is the test manifest where we set the env var

apiVersion: k6.io/v1alpha1
kind: K6
metadata:
  name: k6-basic
spec:
  parallelism: 1
  script:
    configMap:
      name: test-scripts
      file: k6_script.js
  runner:
    image: docker-registry.com/mostafamoradian/xk6-kafka:0.20.0
    metadata:
      annotations: 
        stuart.com/ship-slug: ship
    env:
      - name: KAFKA_TOPIC
        value: "test_topic"
      - name: K6_TEST_ENV
        value: vault:kv/data/path#K6_TEST_ENV

If I delete this two lines starter and basic resources are deployed and I don’t see any error log there.

      - name: K6_TEST_ENV
        value: vault:kv/data/path#K6_TEST_ENV

Any clue on this error ?
Thanks in advance!

3

Hi @hermesmenezesjunior, this error usually means that there’s something off with the k6 script itself and arguments / env vars passed to it. Is it possible to share the script? It’s also highly recommended to try to run the script separately with k6 run; in this case, while passing the same env vars.

1 Like
5

hi @olhayevtushenko,
its a really simple script where we just test if the variable has a value inside. As I mention if I execute the same script but I deleting this:

      - name: K6_TEST_ENV
        value: vault:kv/data/shiptesteng/k6-performance-tests#K6_TEST_ENV

from manifest, everything works well

this is the script in any case, but its just printing the value of the env value:

import { createMsgSendWebhookV1 } from "../kafka/messages/sendWebhookV1.js";
import { KafkaMessageProducer } from "../kafka/KafkaMessageProducer.js";


export const options = {
  discardResponseBodies: true,
  scenarios: {
    my_second_scenario: {
      executor: "per-vu-iterations",
      vus: 1,
      iterations: 1,
      maxDuration: "5m",
    },
  },
};


const broker = __ENV.KAFKA_BROKER_URL || "kafka:9092";
const topic = __ENV.KAFKA_TOPIC || "webhooks.commands.send_webhook";

const testVar = __ENV.K6_TEST_ENV || "Nothing";

const kafkaProducer = new KafkaMessageProducer([broker], topic);


export default function () {
  const message = createMsgSendWebhookV1();
  console.log("Here it is the env var value: ", testVar)
  console.log(message);
  const messages = [
    {
      keyData: {},
      valueData: message,
      headers: {},
    },
  ];
  kafkaProducer.produceMessages(messages);
}

export function teardown(data) {
  kafkaProducer.close();
}

Do you have any idea where the problem could be :thinking: ? If you need any other information just tell me

EDIT: Related question, is there any other way to handle secrets with k6?

7

@hernanmateikastuart

I tried very hard, but I can’t reproduce this error message. Does it occur even if you try with simpler script like:

export const options = {
  discardResponseBodies: true,
  scenarios: {
    my_second_scenario: {
      executor: "per-vu-iterations",
      vus: 1,
      iterations: 1,
      maxDuration: "5m",
    },
  },
};

const testVar = __ENV.K6_TEST_ENV || "Nothing";

export default function () {
  console.log("Here it is the env var value: ", testVar)
}
8

hi @bandorko, thanks for your reply!

Tested just in case, exactly same behaviour:

Pasting full log just in case

2024-01-05T09:09:34Z	ERROR	controllers.K6	unable to marshal: `time="2024-01-05T08:58:13Z" level=info msg="received new Vault token" addr= app=vault-env path=general-xxx role=k8s-test-role
time="2024-01-05T08:58:13Z" level=info msg="initial Vault token arrived" app=vault-env
time="2024-01-05T08:58:13Z" level=info msg="spawning process: [sh -c mkdir -p $(dirname /tmp/k6_script.js.archived.tar) && k6 archive /test/k6_script.js -O /tmp/k6_script.js.archived.tar  2> /tmp/k6logs && k6 inspect --execution-requirements /tmp/k6_script.js.archived.tar 2> /tmp/k6logs ; ! cat /tmp/k6logs | grep 'level=error']" app=vault-env
{
  "paused": null,
  "vus": null,
  "duration": null,
  "iterations": null,
  "stages": null,
  "scenarios": {
    "my_second_scenario": {
      "executor": "per-vu-iterations",
      "startTime": null,
      "gracefulStop": null,
      "env": null,
      "exec": null,
      "tags": null,
      "vus": 1,
      "iterations": 1,
      "maxDuration": "5m0s"
    }
  },
  "executionSegment": null,
  "executionSegmentSequence": null,
  "noSetup": null,
  "setupTimeout": null,
  "noTeardown": null,
  "teardownTimeout": null,
  "rps": null,
  "dns": {
    "ttl": null,
    "select": null,
    "policy": null
  },
  "maxRedirects": null,
  "userAgent": null,
  "batch": null,
  "batchPerHost": null,
  "httpDebug": null,
  "insecureSkipTLSVerify": null,
  "tlsCipherSuites": null,
  "tlsVersion": null,
  "tlsAuth": null,
  "throw": null,
  "thresholds": null,
  "blacklistIPs": null,
  "blockHostnames": null,
  "hosts": null,
  "noConnectionReuse": null,
  "noVUConnectionReuse": null,
  "minIterationDuration": null,
  "ext": null,
  "summaryTrendStats": [
    "avg",
    "min",
    "med",
    "max",
    "p(90)",
    "p(95)"
  ],
  "summaryTimeUnit": null,
  "systemTags": [
    "check",
    "error",
    "error_code",
    "expected_response",
    "group",
    "method",
    "name",
    "proto",
    "scenario",
    "service",
    "status",
    "subproto",
    "tls_version",
    "url"
  ],
  "tags": null,
  "metricSamplesBufferSize": null,
  "noCookiesReset": null,
  "discardResponseBodies": true,
  "totalDuration": "5m30s",
  "maxVUs": 1
}`	{"namespace": "k6-test-te", "name": "k6-basic", "reconcileID": "612ff40f-fa34-4c6e-8b1f-3677a3e00f83", "error": "invalid character 'i' in literal true (expecting 'r')"}

If I use a simple string for that value in test manifest it works smoothly so the error has to be related to how it uses the variable to retrieve the Vault secret :thinking:

      - name: K6_TEST_ENV
        value: "simpleString"

Anything else worth trying?

9

@hermesmenezesjunior, it looks like function KafkaMessageProducer prints out logging lines into the console:

level=info msg="received new Vault token" addr= app=vault-env path=general-xxx role=k8s-test-role
time="2024-01-05T08:58:13Z" level=info msg="initial Vault token arrived" app=vault-env

Can this log be removed? It doesn’t allow to have correct JSON, thus producing an error from initializer.

Btw, the easiest way to check this is to run standalone k6 first:

K6_TEST_ENV=... k6 run scripts.js
K6_TEST_ENV=... k6 inspect scripts.js
1 Like
11

hi @olhayevtushenko!
this latest log relates to this script:

export const options = {
  discardResponseBodies: true,
  scenarios: {
    my_second_scenario: {
      executor: "per-vu-iterations",
      vus: 1,
      iterations: 1,
      maxDuration: "5m",
    },
  },
};

const testVar = __ENV.K6_TEST_ENV || "Nothing";

export default function () {
  console.log("Here it is the env var value: ", testVar)
}

So im not logging anything in the script itself. This log must come from the resolution of the Vault secrets (for this reason it fails when I add that variable and works when I remove it), this log should be displayed here? :thinking:
Any way I can avoid this? Since what causes this log is only the resolution of the secret that has been put in the manifest, there is nothing related in the script

13

@hernanmateikastuart, are you using Banzai Cloud platform?
After some googling around, I see that the likely culprit is not Vault but this platform.

14

@olhayevtushenko

Oh yes, you hit the nail, we are using Banzai Cloud vault injection :cry: .
There is something we can do?

15

@hernanmateikastuart

As I see, the problem is, that bank-vaults (You use it, right?) injects vault-env container as an init container into the k6 initializer pod if there is environment variable, which contains a vault path. This vault-env init container logs various messages into stdout. After the run of the k6 initializer pod, k6-operator tries to parse the pod output as a json, but because of the vault-env container’s log messages it fails.

Could you please try what happens if you add this two annotations to the runner section (after the stuart.com/ship-plug annotation)?

        vault.security.banzaicloud.io/vault-env-image: "ghcr.io/bank-vaults/vault-env:v1.21.0"
        vault.security.banzaicloud.io/log-level: "fatal"
16

@bandorko
Oh that did the trick, since its not logging anything now it works flawlessly.
You rock guys, thanks for taking the time to take a look at this! :heart_decoration:

17

@hernanmateikastuart
Maybe you can set the log level to anything above info (e.g. warning). The vault-env image version can’t be higher because they changed the logging library in v1.21.1, and unfortunatelly current implementation doesn’t support the setting of the log-level.

And maybe it would worth not relying on the log format in k6-operator

2 Likes
18

I agree that even if this is a workaround that works, k6-operator should ignore these logs to avoid breaking execution. Maybe I can create an issue in k6-operator repo? WDYT @bandorko @olhayevtushenko ?

19

@hernanmateikastuart, if this article is correct, you should be able to configure your env vars (secrets) injection via vault-env by using standard vault’s config:

So setting log level of vault-env to err should be sufficient to bypass the whole issue.

I consider this use case rather unusual as Banzai Cloud uses intrusive logic: it is rewriting the entrypoints of containers that are in the ownership of another application. Apparently, they do it to decrease an attack surface, but whether it’s a good design is quite debatable, IMHO.

Since Banzai Cloud can work with k6-operator with only minimum amount of configuration, I don’t think there’s a high-priority issue that should be solved on k6-operator side. But @hernanmateikastuart, if you’d like to add your observations from using Banzai Cloud together with k6-operator, there’s is already an issue over here:

Please feel free to comment there.

2 Likes