How to fail a functional test inside docker-container

[rakeshsukla53]

Hi,

I am using k6 to run my functional and load api tests. I am using docker containers to run my tests.

I have tried bunch of approaches but I can’t make a test fail inside the docker container. It’s always returning with code 0. So to give you an example

    docker-compose up
    Recreating platform-api-tests_test_1 ... done
    Attaching to platform-api-tests_test_1
    test_1  |
    test_1  |           /\      |‾‾|  /‾‾/  /‾/
    test_1  |      /\  /  \     |  |_/  /  / /
    test_1  |     /  \/    \    |      |  /  ‾‾\
    test_1  |    /          \   |  |‾\  \ | (_) |
    test_1  |   / __________ \  |__|  \__\ \___/ .io
    test_1  |
      execution: local--------------------------------------------------]   servertorunner
    test_1  |      output: -
    test_1  |      script: start.js
    test_1  |
    test_1  |     duration: -, iterations: 1
    test_1  |          vus: 1, max: 1
    test_1  |
    time="2020-09-24T00:40:48Z" level=error msg="GoError: GoError: test\n\tat native\n\tat file:///test/api/threshold.js:26:16(25)\n\tat file:///test/api/threshold.js:23:20(14)\n\tat file:///test/start.js:15:27(4)\n"
    test_1  | time="2020-09-24T00:40:48Z" level=info msg="Test finished" i=1 t=1.1576ms
    test_1  |
    test_1  |     data_received........: 0 B     0 B/s
    test_1  |     data_sent............: 0 B     0 B/s
    test_1  |     iteration_duration...: avg=849.3µs min=849.3µs med=849.3µs max=849.3µs p(90)=849.3µs p(95)=849.3µs
    test_1  |     iterations...........: 1       863.856254/s
    test_1  |     rt_errors............: 100.00% ✓ 1   ✗ 0
    test_1  |     vus..................: 1       min=1 max=1
    test_1  |     vus_max..............: 1       min=1 max=1
    test_1  |
    platform-api-tests_test_1 exited with code 0

Even a simple example like

import { fail } from "k6";

export default function() {
	fail("failing")
};

If I run this inside a container it will pass result in pass

docker-compose up
Recreating platform-api-tests_test_1 ... done
Attaching to platform-api-tests_test_1
test_1  |
test_1  |           /\      |‾‾|  /‾‾/  /‾/
test_1  |      /\  /  \     |  |_/  /  / /
test_1  |     /  \/    \    |      |  /  ‾‾\
test_1  |    /          \   |  |‾\  \ | (_) |
test_1  |   / __________ \  |__|  \__\ \___/ .io
test_1  |
  execution: local--------------------------------------------------]   servertorunner
test_1  |      output: -
test_1  |      script: start.js
test_1  |
test_1  |     duration: -, iterations: 1
test_1  |          vus: 1, max: 1
test_1  |
time="2020-09-24T00:54:19Z" level=error msg="GoError: failing\n\tat native\n\tat file:///test/api/threshold.js:5:15(4)\n\tat file:///test/start.js:15:27(4)\n"
test_1  | time="2020-09-24T00:54:19Z" level=info msg="Test finished" i=1 t=0s
test_1  |
test_1  |     data_received........: 0 B 0 B/s
test_1  |     data_sent............: 0 B 0 B/s
test_1  |     iteration_duration...: avg=79.3µs min=79.3µs med=79.3µs max=79.3µs p(90)=79.3µs p(95)=79.3µs
test_1  |     iterations...........: 1   0/s
test_1  |     vus..................: 1   min=1 max=1
test_1  |     vus_max..............: 1   min=1 max=1
test_1  |

As you can see there is an error code but still the container exited with code 0. I have tired thresholds, fail and other ways to tackle this issue but still the code is returning 0. Can you advise please ? How to make the docker container exit with an error code.

[pawel]

Hi @rakeshsukla53 !

the fail() function is quite poorly named in k6 it doesn’t fail the test. It aborts the current iteration of the main function. Calling this function won’t produce a non-0 exit code.

If you would like to fail the test based on some condition, you need to use thresholds.
Here’s a simple example

import http from 'k6/http';
import {Counter} from 'k6/metrics'

let failedTestCases = new Counter('failedTestCases');

export let options = {
  thresholds: {
    failedTestCases: [{threshold: 'count===0'}], 
  }
};

export default function() {
  let res = http.get("https://test-api.k6.io/public/crocodiles/404");
  failedTestCases.add(res.status !== 200);  
};

And here’s how it looks like when you execute it. Note the exit code.

If you are serious about functional testing, here’s a more extensive example you can build on.

import http from 'k6/http'
import {Counter} from 'k6/metrics'
import {sleep, check as loadTestingCheck, group} from "k6";
import {randomIntBetween} from "https://jslib.k6.io/k6-utils/1.0.0/index.js";

let assert = function (result, name) {
  loadTestingCheck(null, {[name]: result}); // to record a check
  failedTestCases.add(!result);
  return result;
};

let failedTestCases = new Counter('failedTestCases');

export let options = {
  thresholds: {
    failedTestCases: [{threshold: 'count===0', abortOnFail: true}], 
  },
  scenarios: {
    runFunctionalTests: {
      executor: 'per-vu-iterations',
      vus: 1,
      iterations: 1,
    },
  },
};

const USERNAME = `${randomIntBetween(10000, 100000)}@example.com`;
const PASSWORD = 'superCroc2019';
const BASE_URL = 'https://test-api.k6.io';

function simpleFunctionalTest() {
  group("Simple smoke test", ()=> {
    let res = http.get(`${BASE_URL}/public/crocodiles/`);
    
    let api_ok = assert(res.status === 200, "API is working");
    let crocs_ok = assert(api_ok && res.json("#") === 8, "got 8 crocodiles");
    let bert_ok = assert(crocs_ok && res.json("0.name") === "Bert", "First crocodile is Bert");
    let pepe_ok = assert(crocs_ok && res.json("1.name") === "Pepe", "Second crocodile is Pepe");
  });
}

function createUserTest() {
  group("User creation", ()=> {
    let res = http.post(`${BASE_URL}/user/register/`, {
      first_name: 'Crocodile',
      last_name: 'Owner',
      username: USERNAME,
      password: PASSWORD,
    });

    assert(res.status === 201, 'created user') &&
    assert(res.json('username') === USERNAME, 'created user has correct email');
  });
}

function authenticationTest() {
  group("Authentication", ()=> {
    let loginRes = http.post(`${BASE_URL}/auth/token/login/`, {
      username: USERNAME,
      password: PASSWORD
    });

    assert(loginRes.status === 200, 'logged in successfully') &&
    assert(loginRes.json('access'), 'got access token');
  });
}

export default function testSuite() {
  // execute all test cases
  simpleFunctionalTest();
  createUserTest();
  authenticationTest();
}

If you have any further questions, feel free to ask!

[rakeshsukla53]

The problem is exit code. Even with threshold you will notice the exit code is 0 when you run inside the docker containers. Let me check again.

[pawel]

The exit code for docker should be the same as for standalone k6.
I just checked with the script I pasted before. The result is the same.

Let me know if you have a different result.

[rakeshsukla53]

test_1  |
test_1  |     data_received..............: 5.8 kB 31 kB/s
test_1  |     data_sent..................: 730 B  3.9 kB/s
test_1  |   ✗ failedTestCases............: 1      5.365365/s
test_1  |     http_req_blocked...........: avg=50.92ms  min=25.1µs   med=50.92ms  max=101.82ms p(90)=91.64ms  p(95)=96.73ms
test_1  |     http_req_connecting........: avg=14.76ms  min=0s       med=14.76ms  max=29.53ms  p(90)=26.57ms  p(95)=28.05ms
test_1  |     http_req_duration..........: avg=42.68ms  min=27.92ms  med=42.68ms  max=57.44ms  p(90)=54.49ms  p(95)=55.96ms
test_1  |     http_req_receiving.........: avg=560.2µs  min=190.4µs  med=560.2µs  max=930µs    p(90)=856.04µs p(95)=893.02µs
test_1  |     http_req_sending...........: avg=177.4µs  min=67.2µs   med=177.4µs  max=287.6µs  p(90)=265.56µs p(95)=276.58µs
test_1  |     http_req_tls_handshaking...: avg=34.69ms  min=0s       med=34.69ms  max=69.39ms  p(90)=62.45ms  p(95)=65.92ms
test_1  |     http_req_waiting...........: avg=41.94ms  min=27.45ms  med=41.94ms  max=56.44ms  p(90)=53.54ms  p(95)=54.99ms
test_1  |     http_reqs..................: 2      10.730731/s
test_1  |     iteration_duration.........: avg=187.56ms min=187.56ms med=187.56ms max=187.56ms p(90)=187.56ms p(95)=187.56ms
test_1  |     iterations.................: 1      5.365365/s
test_1  |     vus........................: 1      min=1 max=1
test_1  |     vus_max....................: 1      min=1 max=1
test_1  |
test_1  | time="2020-09-24T18:08:11Z" level=info msg="Test finished" i=1 t=186.3806ms
test_1  | time="2020-09-24T18:08:11Z" level=error msg="some thresholds have failed"
platform-api-tests_test_1 exited with code 99

I am seeing the correct exit code now. Thank you