Cypress Parallelization on CircleCI with JUnit
Cypress is a great testing suite. I find that writing E2E tests are very fast and easy thanks to its retry-ability and jQuery-like selectors.
What's even more impressive is the well-made CircleCI Orbs and parallelization features that come out of the box with the Cypress Dashboard
Having tried the Cypress Dashboard, it's very nice. I would have been interested in moving forward with it, but unfortunately my org's use-case involves running dozens of tests every 15 minutes. When you consider that each E2E test takes up one test recording, things can add up quickly:
Since CircleCI is perfectly capable of storing the test recordings as MP4s and failures as screenshots, I don't really need Dashboard other than for its parallelization features. This doesn't make it worth the $400+/month price-tag that my org would have to pay just to achieve parallel testing.
Sorry, Cypress
Others seem to feel the same way, since Sorry Cypress is available as a replacement that you can host, maintain, hack, and update yourself.
I'm happy without a dashboard, so after briefly exploring using it as a sidekick container on my CircleCI builds, I decided that this solution was still overkill for what I needed.
"Free" Parallelization
The concept of parallelizing tests are not proprietary, and can be easily done without the Cypress Dashboard.
CircleCI has very nice documentation on how to parallelize your tests, including a nifty CLI tool that will automatically glob and split your spec files based on timings.
The not-so-nice part is that Cypress doesn't include any examples that are friendly with CircleCI's documentation, pointing to their Dashboard as the only solution.
The Cypress community also doesn't seem to have any examples handy, so I dove in and made my own.
Hijack the command
The first thing needed is to split the files being sent to the cypress run
command. This can be done using the command
property in the cypress/run
step:
- cypress/run: # "run" job comes from "cypress" orb
requires:
- cypress/install
command: |
shopt -s globstar
circleci tests glob cypress/integration/**/*.spec.js | circleci tests split --split-by=timings > /tmp/tests-to-run
yarn run cypress run -s $(cat /tmp/tests-to-run)
parallelism: 4
store_artifacts: true
yarn: true
Notice the shopt -s globstar
command. Without that, the circleci tests glob
CLI command won't recursively get nested spec files from the test folders.
Store the timing
The --split-by=timings
command is pretty useless until CircleCI can actually know about how long the tests took on a per-file basis.
There's two distinct steps to addressing this:
- Configure Cypress to output test result files
- Send these result files to CircleCI
Both steps are quite simple, but there's a snag that I'll talk about at the end.
Configuring Cypress
To set up the Mocha junit reporter that's already built into Cypress, you can set up your cypress.json
config to include the following:
{
"reporter": "junit",
"reporterOptions": {
"mochaFile": "cypress/results/output-[hash].xml"
}
}
This will result in files being saved for each spec file in the cypress/results
folder, with [hash]
representing the MD5 hash sum of the files' contents.
Storing the results
Then in CircleCI, you can store these newly-created test results:
post-steps:
- store_test_results:
path: cypress/results
The result looks good!
As a bonus, you should also see the results of your tests in the "Tests" tab, along with any failure details that was being outputted by the command line for that particular test:
Filename snag
Everything seemed to look great, but the timings didn't seem to be splitting very well. Looking at the logs of my new custom command yielded a clue:
Diving further into the generated junit XML files showed that the mocha-junit-reporter library that Cypress utilizes would only attach the spec file name to a detached testsuite
node called "Root Suite". It would leave all of the other testsuite
nodes with no filenames at all.
I didn't want to dive into the internals of another library, so I wrote a quick script that populates the file
attribute into the remaining testsuite
nodes. You'll need to add xml2js
to your project for this to run:
const fs = require("fs")
const parseString = require("xml2js").parseString
const xml2js = require("xml2js")
fs.readdir("./cypress/results", (err, files) => {
if (err) {
return console.log(err)
}
files.forEach((file) => {
const filePath = `./cypress/results/${file}`
fs.readFile(filePath, "utf8", (err, data) => {
if (err) {
return console.log(err)
}
parseString(data, (err, xml) => {
if (err) {
return console.log(err)
}
// Handy for debugging
// console.log(filePath, data)
// console.dir(xml)
// An empty test run will yield an empty XML, so this safeguards from that
if (!xml.testsuites.testsuite) {
return
}
const file = xml.testsuites.testsuite[0].$.file
xml.testsuites.testsuite.forEach((testsuite, index) => {
if (index > 0) {
testsuite.$.file = file
}
})
const builder = new xml2js.Builder()
const xmlOut = builder.buildObject(xml)
fs.writeFile(filePath, xmlOut, (err) => {
if (err) throw err
})
})
})
})
})
Then finally add this command to the post-steps
for it to work:
- run:
name: Fix junit file attributes
command: |
yarn node scripts/fix-junit-xml.js
Final Result
By the end, your workflow should look something like this:
workflows:
test:
jobs:
- cypress/install:
yarn: true
- cypress/run: # "run" job comes from "cypress" orb
requires:
- cypress/install
command: |
shopt -s globstar
circleci tests glob cypress/integration/**/*.spec.js | circleci tests split --split-by=timings > /tmp/tests-to-run
yarn run cypress run -s $(cat /tmp/tests-to-run)
parallelism: 4 # or whatever parallelism you want!
store_artifacts: true
yarn: true
post-steps:
- run:
name: Fix junit file attributes
command: |
yarn node scripts/fix-junit-xml.js
- store_test_results:
path: cypress/results