Cypress Parallelization on CircleCI with JUnit

Cypress is a great testing suite. I find that writing E2E tests are very fast and easy thanks to its retry-ability and jQuery-like selectors.

What's even more impressive is the well-made CircleCI Orbs and parallelization features that come out of the box with the Cypress Dashboard

Having tried the Cypress Dashboard, it's very nice. I would have been interested in moving forward with it, but unfortunately my org's use-case involves running dozens of tests every 15 minutes. When you consider that each E2E test takes up one test recording, things can add up quickly:

282k Projected Recordings

Since CircleCI is perfectly capable of storing the test recordings as MP4s and failures as screenshots, I don't really need Dashboard other than for its parallelization features. This doesn't make it worth the $400+/month price-tag that my org would have to pay just to achieve parallel testing.

Sorry, Cypress

Others seem to feel the same way, since Sorry Cypress is available as a replacement that you can host, maintain, hack, and update yourself.

I'm happy without a dashboard, so after briefly exploring using it as a sidekick container on my CircleCI builds, I decided that this solution was still overkill for what I needed.

"Free" Parallelization

The concept of parallelizing tests are not proprietary, and can be easily done without the Cypress Dashboard.

CircleCI has very nice documentation on how to parallelize your tests, including a nifty CLI tool that will automatically glob and split your spec files based on timings.

The not-so-nice part is that Cypress doesn't include any examples that are friendly with CircleCI's documentation, pointing to their Dashboard as the only solution.

The Cypress community also doesn't seem to have any examples handy, so I dove in and made my own.

Hijack the command

The first thing needed is to split the files being sent to the cypress run command. This can be done using the command property in the cypress/run step:

      - cypress/run: # "run" job comes from "cypress" orb
          requires:
            - cypress/install
          command: |
            shopt -s globstar
            circleci tests glob cypress/integration/**/*.spec.js | circleci tests split --split-by=timings > /tmp/tests-to-run
            yarn run cypress run -s $(cat /tmp/tests-to-run)
          parallelism: 4
          store_artifacts: true
          yarn: true

Notice the shopt -s globstar command. Without that, the circleci tests glob CLI command won't recursively get nested spec files from the test folders.

Store the timing

The --split-by=timings command is pretty useless until CircleCI can actually know about how long the tests took on a per-file basis.

There's two distinct steps to addressing this:

  1. Configure Cypress to output test result files
  2. Send these result files to CircleCI

Both steps are quite simple, but there's a snag that I'll talk about at the end.

Configuring Cypress

To set up the Mocha junit reporter that's already built into Cypress, you can set up your cypress.json config to include the following:

{
  "reporter": "junit",
  "reporterOptions": {
    "mochaFile": "cypress/results/output-[hash].xml"
  }
}

This will result in files being saved for each spec file in the cypress/results folder, with [hash] representing the MD5 hash sum of the files' contents.

Storing the results

Then in CircleCI, you can store these newly-created test results:

          post-steps:
            - store_test_results:
                path: cypress/results

The result looks good!

Test Results Uploaded Successfully

As a bonus, you should also see the results of your tests in the "Tests" tab, along with any failure details that was being outputted by the command line for that particular test:

Good job, your tests pass now

Filename snag

Everything seemed to look great, but the timings didn't seem to be splitting very well. Looking at the logs of my new custom command yielded a clue:

Error autodetecting timing type

Diving further into the generated junit XML files showed that the mocha-junit-reporter library that Cypress utilizes would only attach the spec file name to a detached testsuite node called "Root Suite". It would leave all of the other testsuite nodes with no filenames at all.

No File, No Good

I didn't want to dive into the internals of another library, so I wrote a quick script that populates the file attribute into the remaining testsuite nodes. You'll need to add xml2js to your project for this to run:

const fs = require("fs")
const parseString = require("xml2js").parseString
const xml2js = require("xml2js")

fs.readdir("./cypress/results", (err, files) => {
  if (err) {
    return console.log(err)
  }
  files.forEach((file) => {
    const filePath = `./cypress/results/${file}`
    fs.readFile(filePath, "utf8", (err, data) => {
      if (err) {
        return console.log(err)
      }

      parseString(data, (err, xml) => {
        if (err) {
          return console.log(err)
        }

        // Handy for debugging
        // console.log(filePath, data)
        // console.dir(xml)

        // An empty test run will yield an empty XML, so this safeguards from that
        if (!xml.testsuites.testsuite) {
          return
        }

        const file = xml.testsuites.testsuite[0].$.file
        xml.testsuites.testsuite.forEach((testsuite, index) => {
          if (index > 0) {
            testsuite.$.file = file
          }
        })

        const builder = new xml2js.Builder()
        const xmlOut = builder.buildObject(xml)
        fs.writeFile(filePath, xmlOut, (err) => {
          if (err) throw err
        })
      })
    })
  })
})

Then finally add this command to the post-steps for it to work:

            - run:
                name: Fix junit file attributes
                command: |
                  yarn node scripts/fix-junit-xml.js

Final Result

By the end, your workflow should look something like this:

workflows:
  test:
    jobs:
      - cypress/install:
          yarn: true
      - cypress/run: # "run" job comes from "cypress" orb
          requires:
            - cypress/install
          command: |
            shopt -s globstar
            circleci tests glob cypress/integration/**/*.spec.js | circleci tests split --split-by=timings > /tmp/tests-to-run
            yarn run cypress run -s $(cat /tmp/tests-to-run)
          parallelism: 4 # or whatever parallelism you want!
          store_artifacts: true
          yarn: true
          post-steps:
            - run:
                name: Fix junit file attributes
                command: |
                  yarn node scripts/fix-junit-xml.js
            - store_test_results:
                path: cypress/results