Originally posted at https://k6.io/blog/announcing-xk6-browser-testing/
We’re excited to launch xk6-browser at Grafana ObservabilityCON today, an extension to k6 adding support for browser automation via the Chrome Devtools Protocol (CDP).
k6 was built because we weren’t satisfied with the developer experience offered by existing load testing solutions. We believe in shifting performance testing, and with it observability, to the left. We promote a proactive approach to building and operating reliable systems, uncovering issues before they’re released to the end user. We encourage the inclusion of load tests in your CI pipeline to prevent system updates from introducing performance regressions.
This public beta release of xk6-browser extends the possible use case beyond what is supported by the existing protocol-level capabilities. From a test authoring and scripting perspective, this means that you can script in terms of user actions (such as navigations, mouse and keyboard actions, taking screenshots) rather than by making HTTP requests, WebSocket messages, etc. This has advantages, improving the developer experience in use cases where scripting using protocol-level APIs would be cumbersome, fragile and difficult to maintain.
With xk6-browser, you can interact with the browser to test your web applications end-to-end while accessing all of the k6 core features, including protocol-level APIs and other k6 extensions. It’s a single tool for both protocol and browser-level testing.
Why browser testing?
Well, for starters it’s been the most requested feature for years. We had resisted adding browser-level testing support to k6 as we wanted to focus on testing backend systems first, making sure we provided a good developer experience for those use cases before tackling other areas of testing.
The golden rule of web performance states that 80-90% of the load time of a web page or application is spent in the frontend. It thus makes sense to start optimizing performance in the frontend. Our counter argument to this has always been that this ratio between load time spent in backend vs frontend very quickly can flip when a backend system approaches its concurrency limit (as illustrated by the graphic below). In our experience it’s very common that teams aren’t aware of what their system’s concurrency limit is and thus have no idea of when the flip will happen.
xk6-browser allows for mixing browser-level and protocol-level APIs. You can now simulate the bulk of traffic with protocol-level scenario(s) as usual, and, at the same time, have a Virtual User interacting with your web site or application using a real browser to collect frontend metrics (DOM content loaded, load, first contentful paint, etc.). You can finally understand both sides of the coin with k6!
How to get started
We have initially built this browser-level capability as a k6 extension. This allows us to experiment without changes to the k6 codebase. To use browser-level APIs in your k6 test, you need a k6 binary built with the xk6-browser extension. You can either download one of the release binaries from GitHub, or build a custom version of k6 from source.
To build from source, first ensure you have the prerequisites:
- Go toolchain
- Git
Then:
1- Build k6 version.
# Install xk6
go install go.k6.io/xk6/cmd/xk6@latest
# build xk6-browser binary
xk6 build --output xk6-browser --with github.com/grafana/xk6-browser
2- Create a test script and import the k6/x/browser module.
import { check } from 'k6';
import { chromium } from 'k6/x/browser';
export default function () {
const browser = chromium.launch({
headless: false,
slowMo: '500ms' // slow down by 500ms
});
const context = browser.newContext();
const page = context.newPage();
// Goto front page, find login link and click it
page.goto('https://test.k6.io/', { waitUntil: 'networkidle' });
Promise.all([
page.waitForNavigation(),
page.locator('a[href="/my_messages.php"]').click(),
]).then(() => {
// Enter login credentials and login
page.locator('input[name="login"]').type('admin');
page.locator('input[name="password"]').type('123');
// We expect the form submission to trigger a navigation, so to prevent a
// race condition, setup a waiter concurrently while waiting for the click
// to resolve.
return Promise.all([
page.waitForNavigation(),
page.locator('input[type="submit"]').click(),
]);
}).then(() => {
check(page, {
'header': page.locator('h2').textContent() == 'Welcome, admin!',
});
}).finally(() => {
page.close();
browser.close();
});
}
3- Run the test script.
./xk6-browser run browser-test.js
When k6 launches and interacts with the browser, it will automatically collect frontend metrics and report them as with other k6 metrics.
/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/ \ | ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __________ \ |__| \__\ \_____/ .io
execution: local
script: browser-test.js
output: -
scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
* default: 1 iterations for each of 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)
running (00m05.4s), 0/1 VUs, 1 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs 00m05.4s/10m0s 1/1 iters, 1 per VU
✓ header
browser_dom_content_loaded.......: avg=166.8ms min=116µs med=62.27ms max=542.53ms p(90)=404.56ms p(95)=473.54ms
browser_first_contentful_paint...: avg=201.28ms min=38.72ms med=83.82ms max=481.3ms p(90)=401.8ms p(95)=441.55ms
browser_first_meaningful_paint...: avg=227.1ms min=38.92ms med=83.87ms max=558.5ms p(90)=463.58ms p(95)=511.04ms
browser_first_paint..............: avg=201.21ms min=38.59ms med=83.79ms max=481.25ms p(90)=401.75ms p(95)=441.5ms
browser_loaded...................: avg=158.83ms min=1.21ms med=61.9ms max=510.33ms p(90)=381.93ms p(95)=446.13ms
checks...........................: 100.00% ✓ 1 ✗ 0
data_received....................: 34 kB 6.3 kB/s
data_sent........................: 3.6 kB 654 B/s
http_req_connecting..............: avg=35.11ms min=0s med=0s max=316ms p(90)=63.2ms p(95)=189.59ms
http_req_duration................: avg=211.2ms min=1.67ms med=171.67ms max=572.01ms p(90)=490.59ms p(95)=531.3ms
http_req_receiving...............: avg=125ms min=0s med=156ms max=199ms p(90)=199ms p(95)=199ms
http_req_sending.................: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s
http_req_tls_handshaking.........: avg=22.55ms min=0s med=0s max=203ms p(90)=40.6ms p(95)=121.79ms
http_reqs........................: 9 1.654275/s
iteration_duration...............: avg=5.43s min=5.43s med=5.43s max=5.43s p(90)=5.43s p(95)=5.43s
iterations.......................: 1 0.183808/s
vus..............................: 1 min=1 max=1
vus_max..........................: 1 min=1 max=1
With xk6-browser, k6 will now report the following list of new frontend metrics:
browser_dom_content_loaded: Time from start of document load until DOMContentLoaded event is fired.browser_loaded: Time from start of document load until load event is fired.browser_first_paint: Time from start of document load until first paint event happens.browser_first_contentful_paint: Time from start of document load until first contentful paint happens.browser_first_meaningful_paint: Time from start of document load until first meaningful paint happens.
xk6-browser API
The xk6-browser API aims for rough compatibility with the Playwright API for NodeJS, meaning k6 users don’t have to learn an entirely new API.
Note that because k6 does not currently run in NodeJS or support the Event Loop. xk6-browser APIs are synchronous and can slightly differ from their Playwright counterpart.
For browser support, the initial focus is on providing solid support for Chromium-based browsers. Over time we want to support Firefox and Webkit-based browsers as well.
What’s next?
The team is working on project stability and reliability. End-to-end tests are sometimes known to be flaky and unreliable, producing failures and inconsistent results. In that case, the testing process can become very inefficient. Reducing flakiness is a primary goal, so you can be confident in your testing.
We’re adding more APIs and looking forward to feedback from users to aid us in prioritizing what to build next! Join the #xk6-browser channel on the k6 Slack and let us know what you think.
Learn more
You can find out more by reading the documentation and GitHub repository. If you get stuck or have any questions for the team, please get in touch with us on the k6 Forum, GitHub, or Slack.
Special acknowledgement to the authors of Playwright and Puppeteer. This project is heavily influenced, and in some regards, based on their code.