Notes on Cypress and visual regression testing

I’ve been playing around with Cypress for end-to-end testing ever since I heard Paul Gilzow’s talk on the subject (“Introduction to Automated End-to-End Testing”) at the 2024 WPCampus conference. We did a pilot ten years ago to implement some limited E2E testing using Behat but it didn’t get very far. Building your own Behat suite without core support was a heavy lift (our Moodle plugins are a different matter).

We did build out some functional testing for our Microsite, which is a static copy of our WordPress theme, implemented in Twig. I took the opportunity of yet another chromium/selenium issue to reimplement our tests in Cypress, and to add visual regression testing for our components. An added complexity is that the tests need to work cross-platform: in developer local environments and in our continuous integration environment. It’s working, but there were a few challenges.

First, I got started by installing the Cypress Visual Regression package. Unsurprisingly, you’ll need the virtual framebuffer (xvfb) package installed. You’ll also want a consistent headless browser across all systems. Chrome is an easy answer, but I’m sure others would have worked. This is what my CI configuration looked like on a Debian-derived container:

- apt-get update -y
- apt-get install -y libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libnss3 libxss1 libasound2 libxtst6 xauth xvfb
- wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
- apt install -y ./google-chrome-stable_current_amd64.deb

I wound up making a number of changes to the default configuration in my cypress.config.js:

const { defineConfig } = require("cypress");
const { configureVisualRegression } = require('cypress-visual-regression')

module.exports = defineConfig({
  trashAssetsBeforeRuns: true,
  viewportHeight: 720,
  viewportWidth: 1280,
  video: false,
  e2e: {
    env: {
      visualRegressionType: 'regression',
      visualRegressionDiffDirectory: 'cypress/snapshots/diff',
    },
    screenshotsFolder: './cypress/snapshots/actual',
    setupNodeEvents(on, config) {
      // implement node event listeners here
      on('before:browser:launch', (browser, launchOptions) => {
        if (browser.name === 'chrome' && browser.family === 'chromium') {
          launchOptions.args.push('--force-color-profile=srgb');
          launchOptions.args.push('--font-render-hinting=none');
          launchOptions.args.push('--force-device-scale-factor=1')
          launchOptions.args.push('--headless')
        }
        return launchOptions
      })
      configureVisualRegression(on)
    },
  },
});

These changes all come down to the fact that a Mac laptop with a Retina display will produce a different screenshot than headless Chrome in a CI environment. Assuming you’re allowing a 0.05% variation, different fonts or a different number of pixels will easily cause a failure. Fortunately, we can eliminate that behavior by forcing the two systems to behave the same.

One other important note here. To activate the Chrome-specific flags, you need to specifically invoke Chrome when running tests, like so:

npx cypress run --browser chrome