Skip to content

Latest commit

 

History

History
214 lines (143 loc) · 10.5 KB

cypress-testing.md

File metadata and controls

214 lines (143 loc) · 10.5 KB
Raw

Cypress testing

Currently, EUI only uses Cypress component testing as opposed to its full E2E test runner. This is ideal for isolating/focusing on specific components while reducing traditional E2E start times.

Running tests

yarn test-cypress runs component tests headlessly (no window appears)

yarn test-cypress-a11y runs component accessibility tests headlessly (no window appears)

yarn test-cypress-dev launches a chrome window controlled by Cypress, which lists out the discovered tests and allows executing/interacting from that window.

Setting the theme

By default tests are run using the light theme. Dark mode can be enabled by passing the --theme=dark option.

Skipping CSS compilation

To ensure tests use up-to-date styles, the test runner compiles our SCSS to CSS before executing Cypress. This adds some processing time before the tests can run, and often the existing locally-built styles are still accurate. The CSS compilation step can be skipped by passing the --skip-css flag to yarn test-cypress, yarn test-cypress-dev and yarn test-cypress-a11y.

Testing specific version of React

By default, EUI Cypress tests are run using the latest supported version of React. You can change that behavior and run e2e tests using a different React version by passing the --react-version option set to 16, 17 or 18.

Cypress arguments

You can also pass Cypress CLI arguments. For example:

# Only run a single specific test file, e.g. onBoarding.js
yarn test-cypress --spec '**/{file}.spec.tsx'

# Opt to run Chrome headlessly
yarn test-cypress --headless

# Overriding config settings, e.g. enabling video recording
yarn test-cypress-dev --config video=true

# Overriding env variables
yarn test-cypress-dev --env password=foobar

Writing tests

When to write Cypress tests

Cypress tests should primarily be written for functionality that Jest/JSDom can't realistically replicate, for example:

  • Focus state
  • Portals
  • 3rd party dependencies
  • Native DOM behaviors (e.g., scrolling, label<>input clicks)

How to write Cypress tests

Cypress has its own specific cy. API/commands, of which you will likely be using cy.get(), cy.find(), and either cy.click() or cy.type() to interact with DOM elements. Here's a small example test:

import { mount } from '@cypress/react';
import TestComponent from './test_component';

describe('TestComponent', () => {
  it('takes user input, submits it, and displays the resulting output', () => {
    mount(<TestComponent />);

    cy.get('[data-test-subj="someInput"]').type('hello world');
    cy.get('[data-test-subj="submitButton"]').click();

    cy.get('[data-test-subj="someOutput"]').contains('HELLO WORLD');
  });
});

Naming your test files

Create Cypress test files with the following name patterns to run in specific situations:

  • {component name}.spec.tsx will run a full component test as part of every build
  • {component name}.a11y.tsx will run an accessibility test using Cypress Axe

These test files should be in the same directory as {component_name}.tsx.

Do's and don'ts

  • DO read through Cypress's best practices recommendations
  • DO use the data-test-subj attribute to mark parts of a component you want to find later.
  • DON'T depend upon class names or other implementation details for finding nodes, if possible.
  • DON'T extend the cy. global namespace - instead prefer to import helper functions directly

Recording failed Cypress tests on CI

EUI now has the ability to record failed Cypress tests as Buildkite CI artifacts. This feature is turned off by default, but can be turned on by changing line 50 of the cypress.config.ts file from video: false to video: true. You can verify videos are being recorded by changing a Cypress test to fail on purpose, then running yarn test-cypress locally. Video files will be stored in the cypress/videos/ directory.

The EUI team has configured Cypress to only keep videos for failing tests. Videos are automatically compressed to make the artifact upload smaller.

Cypress Axe

EUI components are tested for accessibility as part of a scheduled task. This allows us to test changes to the DOM such as accordions being opened, or modal dialogs being triggered, more comprehensively. We use cypress-axe to access the underlying axe-core API methods and rulesets.

How to write cypress-axe tests

// Names must follow the `{component name}.a11y.tsx` pattern to be run correctly
// accordion.a11y.tsx

describe('Automated accessibility check', () => {
  it('has zero violations when expanded', () => {
    cy.mount(
      <EuiAccordion {...noArrowProps}>
        <EuiPanel color="subdued">
          Any content inside of <strong>EuiAccordion</strong> will appear
          here. We will include <a href="#">a link</a> to confirm focus.
        </EuiPanel>
      </EuiAccordion>
    );
    cy.get('button.euiAccordion__button').click();
    cy.checkAxe();
  });
});

Configuring cy.checkAxe()

The cy.checkAxe() helper method has four optional parameters:

  • skipFailures - Set to true to enable report-only mode. Report-only will run the entire suite without causing tests to return early.
  • context - This could be the document or a selector such as a class name, id, or element. The context default is div#__cy_root.
  • axeConfig - The axe.run API can be modified to include or exclude elements, individual rules, or entire rulesets.
  • callback - Define a custom violations callbac if you need to add side effects or change the reporting structure
// Create a custom ruleset by extending EUI defaults

import { defaultAxeConfig } from '../../cypress/support/a11y/axeCheck';

const customAxeConfig = {
 ...defaultAxeConfig,
 runOnly: {
   type: 'tag',
   // Add best-practices to existing rulesets
   values: [...defaultAxeConfig.runOnly.values, 'best-practices'],
 },
};

// Violations will use the custome ruleset and cause tests to fail
cy.checkAxe(false, customAxeConfig);

Cypress Real Events

Cypress default events are simulated. That means that all events like cy.click or cy.type are fired from JavaScript. That's why these events will be untrusted (event.isTrusted will be false) and they can behave a little different from real native events. But for some cases, it can be impossible to use simulated events, for example, to fill a native alert or copy to the clipboard. This plugin solves this problem.

Cypress Real Events

Why Cypress Real Events?

Cypress Real Events uses the Chrome Devtools Protocol to handle behaviors like a real browser. This gives us a better way to test complex events like mouse hover and keyboard focus. By using real events and making assertions against them, we can test keyboard and screen reader accessibility as users change the local state.

How to write Cypress (real event) tests

The Cypress Real Events API works seamlessly with existing cy() methods. If you want to press a button using Cypress Real Events, you could use realPress('Tab') as a replacement for the cy.tab() synthetic method. If you want to press multiple keys (also known as a chord), you should pass an array like ['Shift', 'Tab'] to your helper method.

All Cypress Real Events methods are prefixed with the string "real". Here's a small example test:

import TestComponent from './test_component';

describe('TestComponent', () => {
  it('presses a button using the Enter key', () => {
    /* Use the `realMount()` command to set focus in the test window */
    cy.realMount(<TestComponent />);
    
    /* Activate a button with a real keypress event */
    cy.get('[data-test-subj="submitButton"]').realPress('Enter');
    
    /* Assert the button has focus and the aria-expanded attribute has updated */
    cy.focused().invoke('attr', 'aria-expanded').should('equal', 'true');
  });

  it('presses a button using the Space key', () => {
    /* Assert the button also accepts the Spacebar keypress */
    cy.realMount(<TestComponent />);
    cy.get('[data-test-subj="submitButton"]').realPress('Space');
    cy.focused().invoke('attr', 'aria-expanded').should('equal', 'true');
  });
});

Do's and don'ts for Cypress Real Events

  • DO follow all previous guidance for writing Cypress tests
  • DO use the correct mounting method:
    • Use cy.realMount() if your component doesn't receive focus automatically OR
    • Use cy.mount() for components that receive focus on render
  • DO be on the lookout for new features!

Debugging tests

For debugging failures locally, use yarn test-cypress-dev, which allows you to run a single specific test suite and runs tests in a browser window, making dev tools available to you so you can pause and inspect DOM as needed.

⚠️ As a general rule, we generally recommend taking a break and not clicking around while tests are running. This can eliminate or lower the possibility of hard-to-reproduce/intermittently flaky behavior and timeouts due to user interference.

Artifacts

All failed tests will output a screenshot to a screenshots/ folder (cypress/screenshots). We strongly recommend starting there for debugging failed tests to inspect error messages and UI state at point of failure.

To track what Cypress is doing while running tests, you can pass in --config video=true which will output screencaptures to the cypress/videos/ folder for all tests (both successful and failing). This can potentially provide more context leading up to the failure point, if a static screenshot isn't providing enough information.

ℹ️ We have videos turned off in our config to reduce test runtime, especially on CI, but suggest re-enabling it for any deep debugging.

CI debugging

Failing screenshot artifacts (as well as Cypress logs) are generated by our Jenkins CI.

TODO: Instructions on where to click to see the relevant artifact(s)/screenshots