Panic mode under Load test

Grafana k6 OSS Support
1

Hi,

I’m running k6 v0.26.1 (dev build, go1.13, windows/amd64) and I seem to run into an issue.
Whether I use 10, 100 or 350Vus does not change a thing. It does work sometimes with 350 and can also crash with 10.

export let options = {
	stages: [
		{ duration: "5m", target: 10}, .
		{ duration: "10m", target: 10},
		{ duration: "5m", target: 0 }, 
	]
};

The test starts and after a while stops with the panic log under.
Does anyone has any idea what can cause it? I did manage to run the exact same test with over 300 VUs.
I’m running K6 locally on a dedicated test server. CPU, memory and disk usage are all under 50%.

Running [===================panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x30 pc=0xa48c2c]f
2

Hi @Airchtit,

that is very strange indeed. Can you post your entire script, or relevant parts of it that reproduce the issue? Can you also test your script on a Linux or macOS machine to pinpoint if this only affects Windows?

3

Sure. Here is the ultra simplified version of the script that still fails.
Note this is a Postman export but it worked well earlier.
if someone could explain to me the postman[Post].pop(); line while we’re at it, I’d be very grateful.

// Auto-generated by the Load Impact converter

import "./libs/shim/core.js";
import "./libs/shim/expect.js";
import "./libs/shim/urijs.js";
import { group } from "k6";

export let options = {
	stages: [
		{ duration: "5m", target: 5 }, // simulate ramp-up of traffic from 1 to 100 users over 5 minutes.
		{ duration: "10m", target: 5 }, // stay at 50 users for 5 minutes
		{ duration: "5m", target: 0 }, // ramp-down to 0 users
	]
};

const Post = Symbol.for("post");
const Request = Symbol.for("request");
postman[Symbol.for("initial")]({
	options,
	environment: {
		username: "username",
		password: "password",
		"base-url": "URL",
		token:"FGJWH8PMgN55T489Mw3120sgcP_asVeDYxeYUYTAR7jQ4GTlKixTTD725DNVfK7wF9xutMnPY3IKdbCC-CXjxFaQQtLMewTILY-2OnJdHcOUPvoPGON60X3AibwR7tT6"
	}
});

export default function() {

	group("Auth", function() {
		postman[Request]({
			name: "GenerateToken",
			id: "b01d5cb4-4c98-4dde-8ae7-b4f5dfed7a88",
			method: "POST",
			address: "{{base-url}}/arcgis/tokens/generateToken",
			data: {
				username: "{{username}}",
				password: "{{password}}",
				referer: "{{base-url}}/arcgis/tokens/",
				client: "referer",
				expiration: "100000",
				f: "json"
			},
			headers: {
				accept: "application/json",
				Referer: "{{base-url}}"
			}
		});
	});

	postman[Post].pop();
}
4

Out of curiosity, i tried replacing the options with the following and this gave me no error (for the last 10 minutes at least).

export let options = { 
	maxRedirects: 4, 
	vus: 350,
	iterations: 1000000
};

Am I doing something wrong with the ramping up? Am I poorly using the stage option?

5

Thanks. While I’m trying to reproduce your issue, can you try using the latest k6 binary from the official releases page? Its k6 version should be k6 v0.26.1 (2020-02-24T08:10:13+0000/v0.26.1-0-g4c49f9a3, go1.13.8, windows/amd64). This is to discard any issues with your local k6 build.

And please test the same script on Linux or macOS if you can, to see if the issue is generic or specific to Windows.

Also, please post the full stack trace when the error happens, not just the panic message. It should have a list of function calls, filenames, etc.

6

Am I doing something wrong with the ramping up? Am I poorly using the stage option?

No, your stages configuration is OK.

I just ran a variation of your script (with the URL changed to test against https://httpbin.test.loadimpact.com) and I didn’t notice any issues. The test ran for 20m and finished without errors. I used the v0.26.1 Windows version I posted above.

So I’m inclined to think the problem is with your dev k6 version or your Windows environment. Please try a binary from the releases page and let us know.

if someone could explain to me the postman[Post].pop(); line while we’re at it, I’d be very grateful.

I’m not familiar with the inner-workings of the postman-to-k6 converter, but it seems that postman[Post] is an array of functions that process the k6 response, so that call to pop() would remove the last one added. It seems that you are able to define these handlers at the global level and at the request level, but I don’t think you need that pop() call if you’re not using them. Someone more familiar with the converter can correct me.

1 Like
7

Hi and thank you for taking the time to answer.
It looks like switching to the release version fixed my issue.

I haven’t been at work and have not had the opportunity to test on other environments than Windows but I guess this should be investigated.

8

I get the same issue. Using version: k6-v0.26.1 installed in windows machine with 4 CPU and 14 GB memory.

When a test with ramp up of 10 Virtual users every minute for 10 minutes to 100 virtual users. I get

 panic: runtime error: invalid memory address or nil pointer dereference [recovered]0s
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x30 pc=0xa48c2c]
goroutine 13153 [running]:
github.com/loadimpact/k6/vendor/github.com/dop251/goja.AssertFunction.func1.1(0xc01d6efcd8)
    github.com/loadimpact/k6/vendor/github.com/dop251/goja/runtime.go:1456 +0x9f
panic(0xd7a400, 0x1947e90)
    runtime/panic.go:679 +0x1c0
github.com/loadimpact/k6/vendor/github.com/dop251/goja.(*vm).try.func1(0xc007843110, 0x0, 0xc01d6efba8, 0x0, 0x0, 0x0, 0xc01d6efc00)
    github.com/loadimpact/k6/vendor/github.com/dop251/goja/vm.go:373 +0x3b7
panic(0xd7a400, 0x1947e90)
    runtime/panic.go:679 +0x1c0

Note that the test goes fine if 100 virtual users are run at same time for 10 minutes without ramp up.

9

Hi @jeevananthank,
can you provide more of the stacktrace as this doesn’t show where it starts … at least the whole first gorotuine stack trace (there will be new lines between them and goroutine <number> ... as the next line). Although the more you can give us the better :wink: - you can put them in a gist as to not make some huge post ( I also think discord will not like it :smiley: ).

Do you get the panic early on ? maybe your options are wrong and we haven’t handled it … well :frowning: . If you can post them or at least the stages.

Can you post k6 version output just in case ? although I don’t think it will be the same panic as this one.

Also anything else that you think relevant will be a good idea. And to be honest this is probably issue worthy :wink:

10

Hi @mstoykov. Thanks for the quick reply.
added the complete log here: panic error · GitHub

export const options = {
thresholds:{
    "errors": ["count<1"]
},
vus: 100,
stages: [
    { duration: '1m', target: 10},
    { duration: '1m', target: 20},
    { duration: '1m', target: 30},
    { duration: '1m', target: 40},
    { duration: '1m', target: 50},
    { duration: '1m', target: 60},
    { duration: '1m', target: 70},
    { duration: '1m', target: 80},
    { duration: '1m', target: 90},
    { duration: '1m', target: 100}
  ]
};

I got the panic error after around 3 minutes of test run, very difficult to find out when it happens as k6 progress just hangs up after sometime.

image
image691×23 668 Bytes

k6 version:
image

Hope this helps. Let me know if there is more info required.

11

I don’t know why your progressbar hangs … that sounds like a bug - does it happen with no ramping as well?

You are running v0.26.1 which (as you can see there is compiled with go1.13) and there was a bug in golang itself which lead to this panic , which is fixed in golang 1.13.3.

Due to some CI problems and configurations the windows build used the old version which is why we released v0.26.2, which only changes the version the windows binary is build with. Can you try with it and see if it fixes it for you as well? For the record ramping up and down should have nothing to do with it, but maybe it exaggerates it :slight_smile: .

The progressbar might be related, but if it happens constantly please open an bug report

1 Like
12

Sure thing, i will update the version.
progress bar hangs up when there is good amout of virtual users run. for example, it hanged with 100 vus without any ramp up. I had a good amount of available CPU and memory in my machine, so don’t think that would be a problem.
When it hangs up, with a key press (enter) it continues. so strange.