Run K6 browser tests "headless=false" in Azure pipeline

I am a QA at my company currently trying to implement K6 and K6Browser scripts to improve our performance checks. So far it has gone quite well but we have hit an issue.

The old browser tests (Selenium) the new scripts are based on would trigger app insights that we monitor in a dashboard. I found out some of these telemetry points are only sent on browser/page close. Running the tests in headless mode does not trigger these so are not captured, however running headless=false does.

A couple of questions for anyone who might know:
While using the K6 Azure plugin is it possible to pass K6_BROWSER_HEADLESS=“false” in the args: for the scripts? Documentation suggests it might be possible but I haven’t got it to work yet

Secondary question (which might need posted separately) - Is there a difference in how K6 closes browsers in UI vs headless mode that might not let telemetry to be captured in time?

Hi @davidstuart,

In one of these cases, the k6 browser closes the browser: when the script ends, a fatal error occurs during the script execution, or the script calls browser.close. If there is a discrepancy between headless and headful, that might indicate a bug. Can you help us reproduce the issue with an example script?

I lack knowledge about how the k6 Azure extension works. But, it should be possible to pass the K6_BROWSER_HEADLESS as it’s a k6 environment variable, not specific to k6 Azure. cc: @pepecano, can you chime in if you have more ideas for the Azure extension? Thanks!

Hi @inancgumus,

I unfortunately can share the script directly or have a system where you can run it as the app I work on is 100% behind a login.
This is an example of one of the scripts I am running with details removed but hopefully gives an idea of what is being run.
I am also trying to get a little more detail on how the current telemetry point is triggered and will share that once I have a little more

import {sleep} from "k6";
import {browser} from "k6/experimental/browser";
const BASE_URL = "";
const username = "";
const password = "";
const jobData = JSON.parse(open("..jobCreate.json"));
const jobToClone = JSON.parse(open("..jobEditSave.json"));

export const options = {
    scenarios: {
        ui: {
            executor: "shared-iterations",
            vus: 1,
            iterations: 50,
            options: {browser: {type: "chromium"}}
        }
    },
    thresholds: {checks: ["rate==1.0"]}
};
export default async function main() {
    const jobId = jobToClone[0];
    const newJob = jobData[0];

    const context = browser.newContext();
    const page = context.newPage();
    try {
	
	    // Fill in Job info
        const titleSelector = page.locator("[data-test=titleSelector]");
        const existingEngagementRemoteCodeNoRadioButton = page.locator("[data-test=existingEngagementRemoteCodeNoRadioButton]");
        const jobTemplateYesRadioButton = page.locator("[data-test=jobTemplateYesRadioButton]");
        const jobTemplateInput = page.locator("(//input[@id='jobTemplateInput'])[2]");
        const jobTemplateResult = page.locator("[data-test=jobTemplateResult]");
        const editJobNameInput = page.locator("[data-test=editJobNameInput]");
        const editJobClientNameInput = page.locator("[data-test=editJobClientNameInput");
        const createEngagementButton = page.locator("[data-test=createEngagementButton]");
        const jobHeaderName = page.locator("[data-test=jobHeaderName]");
        await page.goto(`${BASE_URL}create/job`);
		
        // Login
        await page.waitForSelector("[data-test='Username']");
        await page.locator("[data-test='Username']").type(`${username}`);
        await page.locator("[data-test='Password']").type(`${password}`);
        await Promise.all([page.waitForNavigation(),
            page.locator("[data-test='LoginButton']").click()]);

        await titleSelector.waitFor({state: "visible"});
        // Select no for remote engagement code

        await Promise.all([existingEngagementRemoteCodeNoRadioButton.waitFor({state: "visible"}),
            existingEngagementRemoteCodeNoRadioButton.click({force: true})]);
        await existingEngagementRemoteCodeNoRadioButton.isChecked();

        // Select yes for existing engagement template
        await Promise.all([jobTemplateYesRadioButton.waitFor({state: "visible"}),
            jobTemplateYesRadioButton.click({force: true})]);
        await jobTemplateYesRadioButton.isChecked();

        // Search for known job to clone
        await Promise.all([jobTemplateInput.type(`${jobId.jobId}`),
            page.keyboard.press("Enter")]);

        await Promise.all([jobTemplateResult.waitFor({state: "visible"}),
            jobTemplateResult.click({force: true})]);

        await editJobNameInput.fill(`${newJob.Name} ${new Date().toLocaleDateString()}`);
        await editJobClientNameInput.fill(`${newJob.ClientName} ${new Date().toLocaleDateString()}`);

        await Promise.all([createEngagementButton.waitFor({state: "visible"}),
            createEngagementButton.click()]);

        // Wait for the Job edit page to load of the new created engagement
        await jobHeaderName.waitFor({state: "visible"});
        sleep(10);
    } finally {
        //await Promise.all([context.clearCookies(),
        //    page.close()]);
		page.close();
    }
}

I also tried today to get the Azure plugin running in UI mode (headless=false) but I don’t think I got it working.
I tried the following but not sure if they are correct

steps:
  - task: k6-load-test@0
    displayName: Run PerformanceTest
    inputs:
      filename: 'Folder\PerTest.js'
	  args: set "K6_BROWSER_HEADLESS=false"
    timeoutInMinutes: 20
	
	
steps:
  - task: k6-load-test@0
    displayName: Run PerformanceTest
    inputs:
      filename: 'Folder\PerTest.js'
	  args: "&& set \"K6_BROWSER_HEADLESS=false\""
    timeoutInMinutes: 20

I am running this on a windows agent which is triggering the command via cmd so perhaps I have the format wrong. I did also trying with a “script:” block before the test ran.

Thanks! I’m not able to reproduce this issue. This might be an issue while using the Azure extension. Can someone with knowledge of the Azure extension chime in? :bowing_man: cc: @eyeveebee @pepecano.

Apologies for the delay in updating here.

I have changed the pipeline we have for this so that the run uses a self hosted agent I can connect to while the tests are running.
I found this if I set the environment variable on the agent or run with this argument I can seen Chrome processes start and a K6 process as well

steps:
  - task: k6-load-test@0
    displayName: Run PerformanceTest
    inputs:
      filename: 'Folder\PerTest.js'
	  args: "&& set \"K6_BROWSER_HEADLESS=false\""

In this case although I can see Chrome run it does not actually run in a visible window on the machine. I don’t know if this is expected as I know you can get some odd interactions with windows user and what is running visible. Should I see a visible window if I am on the agent at the same time?

Any thoughts on the above headed mode question? We have tried a few things and while “I think” it is running in headed mode on a static agent as we can see the processes kick in(although we cannot see when we are connect)

The other issue was around some app insights telemetry that didn’t seem to be getting triggered correctly. We have a mix of customEvents and customMetrics and it seems like the latter, customMetrics, aren’t getting triggered/sent correctly. Is there any ideas on how we can stabilise that?

Just posting an update here. We attempted the same tests we wrote in K6 with Playwright and actually encountered the same issue where some page close events were not being sent.

This was eventually tracked down to how Playwright have implemented page.close() where they don’t execute beforeunload handlers. This was resolved in the playwright scripts by using the following

page.close({runBeforeUnload: true})

This will allow our custom telemetry to be sent as the page is closed.

I tested this with our K6 scripts but it looks like it doesn’t use this option. Is there any scope to potentially implementing this as well?

Hey,

Could you post a new issue on our Github repository. We’ll try to prioritize it.

Thanks!

1 Like