Integration Testing with TestCafe & BrowserStack

Kaitlin Moreno
ITNEXT
Published in
5 min readFeb 25, 2018

--

via TestCafe

Click here to share this article on LinkedIn »

I was recently assigned the task of setting up integration tests for a project at work. We were deciding between two open-source solutions, Nightwatch and TestCafe. Ultimately we went with TestCafe because it is not based on Selenium, which allowed for a much easier setup and a lot less tooling. It also has out of the box support for ES6 syntax and a handy plugin for extending their built in selectors to easily test React components. Here is an article with some more pros/cons of using TestCafe vs Nightwatch.

BrowserStack is a cloud-based testing service that allows you to start up pretty much any relevant browser without the need for a virtual machine. They also offer free plans for open source projects. I’ve used BrowserStack for a while now to manually check our app for cross-browser compatibility issues, which is obviously not the greatest solution. Lucky for us TestCafe offers integration with the BrowserStack Automation API through a simple plugin, testcafe-browser-provider-browserstack, which allows us to run automated tests across our whole supported browser matrix.

Using TestCafe’s CLI with BrowserStack

Running tests in one or more of the BrowserStack cloud browsers is very simple. First install TestCafe and the BrowserStack plugin:

npm install --save-dev testcafe testcafe-browser-provider-browserstack

Next, set environment variables with yourBROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY in your shell configuration.

Finally, add a script to your package.json:

"scripts": {
"test:e2e": "testcafe 'browserstack:firefox@58.0:OS X HighSierra,browserstack:ie@11:Windows 10' e2e-tests/*.js --app 'commandToStartApp'"
}

Run npm run test:e2e and head over to your BrowserStack Automate dashboard to see your tests. The above command will run all tests in the ‘e2e-tests’ directory in Firefox 58 and IE 11. For a more in-depth explanation on setting up TestCafe using the CLI, check out Markus Oberleher’s article.

BrowserStack Parallel Worker Limit

The first lesson that I learned the hard way was that BrowserStack only offers a certain number of parallel workers depending on your plan. Our open-source plan happened to give us 5, however I naively ran the command shown above with 10+ browsers. To my horror I realized that if you let TestCafe queue a large number of browsers, it will crash all of your BrowserStack instances and no amount of frantically clicking “Stop Session” from the automate dashboard will save you. TestCafe expects that all specified browsers will be connected and automatically runs your tests in parallel. So how can we take advantage of our 5 allotted workers without blowing everything up?

TestCafe does not support running tests consecutively, so I needed to come up with my own solution. Instead of using the CLI interface, I created a script that uses the TestCafe Node API.

Using TestCafe’s Node API with BrowserStack

In the gist below, I’ve written an async function that creates a new server instance. This function will take a browser as a string,'browserstack:firefox@58.0:OS X HighSierra', or optionally as an array of strings, e.g.

[
"browserstack:safari@11.0:OS X High Sierra",
"browserstack:safari@10.1:OS X Sierra",
"browserstack:edge@16.0:Windows 10",
"browserstack:edge@15.0:Windows 10",
"browserstack:ie@11.0:Windows 10"
]

Same goes for the testFiles argument, which can either be a path or an array of paths to your desired test files.

Now let’s use this function to create a new TestCafe instance for each batch of browsers. In this case I have 10 browsers that I want to test. I have a max of 5 parallel workers available, so that means I’ll need to divide the browsers into 2 batches and each batch will be passed to a new TestCafe instance. Below you’ll see an array that defines our supported browser matrix, as well as a startTests function that loops through those batches. If your BrowserStack plan only supports 1 parallel worker, you can just define all your browsers in that array without breaking them out into sub arrays.

Now if you run this you’ll see that the first batch of browsers will boot up, and only when those are finished will the 2nd batch start.

You might have noticed that we are only defining one test above: e2e-tests/mytest.js. You probably want run a whole suite of tests, and it would be nice if we could just use a glob pattern to grab all our test files instead of hard coding an array of paths. Unfortunately the TestCafe Node API does not support glob patterns, so we need to create a simple helper function that does this for us. First:

npm install --save-dev glob glob-promise

Then we write the helper function and update our script like so:

Now we have a script that will initialize our BrowserStack workers in batches based on our allotted workers and grab the test files we want to run. Let’s update that ‘test:e2e’ script in our package.json:

"scripts": {
"test:e2e": "node scripts/startTests.js"
}

Using the BrowserStack Node API

We still have a problem here though. What if two devs are working in this repo, and happen to run integration tests at the same time? Or we might want to work this script into our CI/CD pipeline and would need to be careful not to run the command at the same time locally. In a way we’re back to where we started — there is no safeguard against overloading our BrowserStack workers.

To solve this issue I’m going to take advantage of the BrowserStack Node API. We can use it to find out how many running sessions are currently available before running our tests.

npm install --save-dev browserstack 

Next we’ll need to add one more environment variable in addition to the ones set earlier (BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY), the password to access your BrowserStack account:

export BROWSERSTACK_PASSWORD=abc123

Then we’ll add the helper function below. We create a new client with our credentials and then use the getApiStatus method to get the status of our running sessions.

getRunningBrowserstackSessions will return a response that looks something like this:

{ 
used_time: 214457,
total_available_time: ‘Unlimited Testing Time’,
running_sessions: 0,
sessions_limit: 5
}

I’m going to simply update our startTests function to warn us that there are not enough workers available and exit the script before executing anything. Check out the final script below:

A couple more ‘gotchas’ to watch out for

fixture`My first test`.page`http://localhost:3000/`

Hopefully this article helps you avoid some of the mistakes I made when using these tools. Happy testing!

--

--