We're excited to announce the release of @cerner/terra-functional-testing which will be replacing the automation testing responsibilities of terra-toolkit. We've made some changes and gave our testing infrastructure a much needed upgrade. This document will outline the major changes and provide useful tips for upgrading from terra-toolkit to @cerner/terra-functional-testing v1.
Since terra-toolkit was originally released in 2017 many new features have been added to create an all-in-one development toolkit. These new features introduced responsibilities for the development site, webpack, jest, reporting, and automation testing. Bundling all these responsibilities into single package has made it difficult to introduce new changes. With the upgrade to WebDriverIO v6 we decided to use this as an opportunity for restructuring our development tools.
The terra-toolkit repository has been converted into a lerna monorepo that contains independent packages for each of our development tools. The terra-toolkit development package has been moved into the terra-toolkit-boneyard repository and will receive minor updates and bug fixes as teams begin the transition to terra-functional-testing.
Moving forward, development responsibilities have been separated out into individual packages that encapsulate specific functionality. As we transition packages into the terra-toolkit monorepo each package will be scoped under the @cerner NPM organization. For example, automation testing has been moved into @cerner/terra-functional-testing and webpack concerns have been moved into @cerner/webpack-config-terra.
There are quite a few breaking changes with this upgrade. Most notably are the breaking changes from WebDriverIO v4 to v6. We're jumping two major versions as part of this upgrade to get our testing infrastructure a much needed upgrade. Many of the browser command APIs have changed to align more consistently. As part of this upgrade teams should expect to update many of their tests. We'll outline a few examples throughout this guide.
The most notable breaking changes are the breaking changes from WebDriverIO v4 to WebDriverIO v6. We'll outline a few of the most common API changes we encountered ourselves as we uplifted our own repositories. A complete list of the breaking changes can be found here:
Commands used during testing have been updated to create a clear distinction between commands acting on the browser/client and commands acting on an element.
For example, in WebDriverIO v4 a click interaction could be called on the browser by passing it a selector, but in v5 and above the action is called on the element instance. When upgrading each of these commands must be updated to the new syntax.
- browser.click('#element'); // Old v4 click. This has been removed.
+ $('#element').click(); // New v6 click. The action is called on the element instance.Here are a few of the most common scenarios we encountered:
// Clicks
- browser.click('#element');
+ $('#element').click();
// Set Value
- browser.setValue('input', 'value');
+ $('input').setValue('value');
// Add Value
- browser.addValue('input', 'value');
+ $('input').addValue('value');
// Wait For Visible
- browser.waitForVisible('#element');
+ $('#element').waitForDisplayed();
// Move To
- browser.moveToObject('#element', 0, 10);
+ $('#element').moveTo({ xOffset: 0, yOffset: 10 });We're dropping support for node 8 as part of the transition to terra-functional-testing. Node 8 reached end of life status December 31st, 2019. Teams still on node 8 will need to upgrade to node 10.
We plan to start upgrading our node version on a more routine cadence following this transition. Our range of node version support is restricted by our transitive dependency on fibers which is necessary for running our WebDriverIO testing environment synchronously.
We've removed the chai assertion framework and opted into using the built in assertion library that ships with WebDriverIO. This new assertion framework closely resembles the testing environment most of us are used to in Jest, because it is Jest's expect library. This should help align testing between Jest and WebDriverIO.
A few common changes look something like this:
- expect(true).to.be.true; // Chai assertion
+ expect(true).toBe(true); // Expect/Jest assertion.
- expect(true).to.be.false; // Chai assertion
+ expect(true).toBe(false); // Expect/Jest assertion.
- expect(1).to.equal(1); // Chai assertion
+ expect(1).toEqual(1); // Expect/Jest assertion.WebDriverIO provides a collection of matchers that can be used for assertions in your tests.
The Terra.it commands have been removed. We noticed they encouraged anti-patterns while writing tests and made testing more confusing.
These commands have been removed:
- Terra.it.validatesElement
- Terra.it.isAccessible
- Terra.it.matchesScreenshot
These commands can still be used but have had some changes:
- Terra.validates.accessibility
- Terra.validates.element
- Terra.validates.screenshot
The validates screenshot and validates element commands now require a screenshot name.
it('should click and validate the element', () => {
$('#element').click();
Terra.validates.element('screenshot name'); // A unique screenshot name must be provided or the command will throw an error.
});The viewports option has been removed from Terra.validates.screenshot. The viewport option array created additional viewport permutations within other viewport permutations. To more consistently align viewport testing the option was removed. To specify an array of viewports use the Terra.describeViewports helper.
+ Terra.describeViewports('Terra.validates', ['tiny', 'small'], () => {
it('should click and validate the element', () => {
$('#element').click();
- Terra.validates.screenshot('screenshot name', { viewports: ['tiny', 'small'] });
+ Terra.validates.screenshot('screenshot name');
});
+ });We've forked and taken ownership of the visual regression service to give us better control of making changes when necessary. Over time we'll be working to resolve common errors encountered with the visual regression service. Such as the notorious x,y range error.
For now though, there are a few breaking changes to the screenshot strategy that will require regenerating screenshots.
- Screenshot names are no longer auto-generated using the describe blocks.
- Screenshots now require a unique name be provided. No screenshots within the same spec file should be identical. Screenshots in other spec files are not affected.
- Screenshots will be scoped under a theme when written to file under _snapshots_.
We've removed the string concatenation of describe blocks to form a screenshot filename. Previously, the following example would generate a screenshot name of block_1_block_2_screenshot_name.
describe('block 1', () => {
describe('block 2', () => {
it('should generate a screenshot', () => {
Terra.validates.element('screenshot name'); // Previously screen generated as block_1_block_2_screenshot_name.
});
});
});This was causing/forcing teams to modify their test structure to produce a meaningful screenshot name. A screenshot could be generated with a tremendously long filename and required sanitizing each of the block descriptions to remove special characters. Describe blocks should be used to describe and group tests together. They should not be introducing side effects to the generated screenshot name.
Moving forward, a unique name must be provided for the screenshot. This should help shorten screenshot names, provide a meaningful name, and encourage developers to write tests semantically.
describe('block 1', () => {
describe('block 2', () => {
it('should generate a screenshot', () => {
Terra.validates.element('unique screenshot name'); // This will now generate a screenshot named unique_screenshot_name.
});
});
});As theming has become more prevalent in our ecosystem screenshots are now always written into a theme directory in _snapshots_. If your application does not explicitly use a theme the screenshots will output under the default terra-default-theme directory. This change will require regenerating screenshots to move them into the appropriate theme directory.
├── __snapshots__
│ ├── reference
│ │ ├── clinical-lowlight-theme
│ │ │ └── en
│ │ │ └── chrome_huge
│ │ │ └── example-spec
│ │ │ ├── screenshot-1.png
│ │ │ └── screenshot-2.png
│ │ ├── orion-fusion-theme
│ │ │ └── en
│ │ │ └── chrome_huge
│ │ │ └── example-spec
│ │ │ ├── screenshot-1.png
│ │ │ └── screenshot-2.png
│ │ └── terra-default-theme
│ │ └── en
│ │ └── chrome_huge
│ │ └── example-spec
│ │ ├── screenshot-1.png
│ │ └── screenshot-2.pngThe American/Chicago timezone has been removed from the docker images. The timezone will default to the timezone provided by the docker image. This will align the selenium grid and local docker images to the same timezone.
The dependency on axe-core has been upgraded from v3.5.3 to v4.0.2. There were no new rules introduced, but previous rules had fixes added that might start detecting accessibility violations after the upgrade that were not previously being detected.
Check out the release notes for more detail information about the changes.
The default selector has been changed from [data-terra-dev-site-content] *:first-child to [data-terra-test-content] *:first-child. This selector is used as the default content region for capturing screenshots. This change should not affect most teams. For teams that are affected a custom selector can be provided using service options.
The formFactor, locale, and theme options that used to be available in the browser.options global object no longer reside in this object. These options along with other data are now available in the Terra.serviceOptions global object.
Any usage of the browser global object should be moved inside the it block because browser will be undefined when accessed outside the scope of the it block. This reason for this behavior is due to how the WebDriverIO testing lifecycle hooks are implemented in v6.
A custom WebDriverIO reporter has been created to facilitate monitoring accessibility violations and reporting them upon the completion of the test run. The goal of this reporter is to allow us to more frequently upgrade our version of axe-core without as many breaking changes. When axe adds new rules those rules can be marked as warnings and the accessibility reporter will report them to the terminal without failing as a violation. Please monitor your terminal output to stay proactive on accessibility warnings.
Our command line interface has been transitioned to Terra CLI to create a uniform experience across our scripts. All new commands are integrated plugins for the terra-cli. Command options are constructed using yargs;
As part of this upgrade teams will be updating the scripts used to invoke wdio to use the terra-cli commands. The terra wdio script will invoke the test runner.
Note: The terra cli array parameters have a different syntax. Array items are now delimited by a space.
// package.json
{
"scripts": {
- "test:wdio": "wdio"
+ "test:wdio": "terra wdio"
- "test:wdio:local": "tt-wdio --gridUrl='grid.test.example.com' --locales=['de','en-AU'] --browsers=['chrome','firefox','ie']"
+ "test:wdio:local": "terra wdio --gridUrl grid.test.example.com --locales de en-AU --browsers chrome firefox ie"
}
}A CLI option has been added to auto-update reference screenshots during the test run. When enabled all screenshots generated during the test run will automatically overwrite the existing reference screenshot. Be sure to manually validate screenshot updates when using this feature.
Via the Terra CLI:
terra wdio --updateScreenshots
# -u is a shortcut for the same command
terra wdio -uWe've added support to automatically detect the VPN IP address. It is no longer necessary to manually specify the WDIO_EXTERNAL_HOST when running wdio. These changes were tested successfully on Mac OS Catalina. Results may vary when running on different platforms and operating systems. It's not perfect and we'll make tweaks as necessary. If this is not working for you let us know. Be sure to specify the operating system and platform you're developing on.
There are a few prerequisites to get started. We'll be removing terra-toolkit as a dependency later in this guide. Removing terra-toolkit is going to impact starting the local development site and webpack configurations. The first step of this transition will be to upgrade to @cerner/webpack-config-terra.
If you are already using @cerner/webpack-config-terra ^1.0.0 you can skip this section.
A complete upgrade guide can be found here. Webpack can be upgraded separately from terra-functional-testing. This change does not need to be bundled with test updates, but it does need to happen before the test updates.
Install @cerner/webpack-config-terra, @cerner/terra-aggregate-translations, and postcss ^8.2.1 (a required peer dependency of webpack-config-terra):
npm install --save-dev @cerner/webpack-config-terra @cerner/terra-aggregate-translations [email protected]If your project utilizes the tt-serve-static bin command go ahead and install terra-cli as you'll need it to upgrade the static asset utilities:
npm install --save-dev @cerner/terra-cliUpdate any references to the old webpack configuration:
// webpack.config.js
- const webpackConfig = require('terra-dev-site/config/webpack/webpack.config');
+ const webpackConfig = require('@cerner/webpack-config-terra');Update any references to the old terra-aggregate-translations:
// jestGlobalSetup.js
- const aggregateTranslations = require('terra-aggregate-translations');
+ const aggregateTranslations = require('@cerner/terra-aggregate-translations');Update any scripts in the package.json to use webpack-dev-server. Use terra-cli for any static sites. Remove any reference to tt-clean-screenshots since cleaning the diff, latest, and error screenshot directories is now executed automaticatically for each test run. Remove terra-aggregate-translations if installed:
// package.json
{
"scripts": {
- "clean:obsolete-wdio-snapshots": "tt-clean-screenshots",
- "start": "tt-serve",
- "start-prod": "tt-serve --env.disableHotReloading -p",
- "start-static": "npm run pack && tt-serve-static",
+ "start": "webpack-dev-server",
+ "start-prod": "webpack-dev-server --env.disableHotReloading -p",
+ "start-static": "npm run pack && terra express-server --site ./build",
},
"devDependencies": {
+ "@cerner/terra-aggregate-translations": "^2.0.1",
+ "@cerner/terra-cli": "^1.3.0",
+ "@cerner/webpack-config-terra": "^1.2.0",
+ "postcss": "^8.2.1",
- "terra-aggregate-translations": "^1.0.0"
}
}At this point it is recommended to do a clean install prior to testing each of the scripts. If you have the script go ahead and run npm run clean:install. Alternatively ensure that the old node_modules directory is deleted and reinstall all dependencies.
npm run clean:installVerify each of the scripts are working as intended:
npm run start
npm run start-prod
npm run start-staticNote: If you don't have some of these commands it is not necessary to add them. Just ensure the commands you do have are working as expected. All of the tt-serve commands should be updated to use webpack-dev-server or the terra express server.
If you are already using @cerner/eslint-config-terra ^5.0.0 you can skip this section.
Upgrading @cerner/eslint-config-terra is optional, but recommended. WebDriverIO v6 syntax requires using the $ global which has been added to the eslint config. If you choose not to upgrade eslint-config-terra at this time you will need to add the global declaration manually in each spec file.
Adding the global manually to each spec file is only necessary if you are not upgrading to @cerner/eslint-config-terra ^5.0.0:
/* global $ */A complete upgrade guide can be found here.
Install @cerner/eslint-config-terra:
npm install --save-dev @cerner/eslint-config-terraRemove the old eslint-config-terra and extend @cerner/terra from the eslintConfig key:
// package.json
{
"eslintConfig": {
- "extends": "terra",
+ "extends": "@cerner/terra",
},
"devDependencies": {
- "eslint-config-terra": "^3.3.0",
+ "@cerner/eslint-config-terra": "^5.0.0",
+ "eslint": "^7.0.0" // Upgrade ESlint to v7.
}
}At this point it is recommended to do a clean install prior to running lint. If you have the script go ahead and run npm run clean:install. Alternatively ensure that the old node_modules directory is deleted and reinstall all dependencies.
npm run clean:installRun eslint and resolve any of the new violations introduced by the new version of ESLint plugins transitively introduced.
npm run lintAfter upgrading webpack and eslint your project is ready to start the migration to @cerner/terra-functional-testing. Before getting started make sure you're prepared to work through the entire uplift. We'll be removing the dependency on terra-toolkit. The implications are that the entire test suite needs to be uplifted. Our recommendation will be to split up the work and go through each spec file one by one. Enabling only the tests you are currently uplifting and disabling the others.
Install @cerner/terra-functional-testing and @cerner/terra-cli:
npm install --save-dev @cerner/terra-functional-testing @cerner/terra-cliRemove terra-toolkit from the package.json:
// package.json
{
"devDependencies": {
- "terra-toolkit": "^6.0.0",
+ "@cerner/terra-cli": "^1.0.0",
+ "@cerner/terra-functional-testing": "^1.0.0"
}
}Update any references to the previous wdio.conf.js:
// wdio.conf.js
- const wdioConfig = require('terra-toolkit/config/wdio/wdio.conf');
+ const { config } = require('@cerner/terra-functional-testing');
- module.exports = wdioConfig;
+ exports.config = config;Update scripts to use the Terra CLI:
Note: The terra cli array parameters have a different syntax.
// package.json
{
"scripts": {
- "test:wdio": "wdio"
+ "test:wdio": "terra wdio"
- "test:wdio:local": "tt-wdio --gridUrl='grid.test.example.com' --locales=['de','en-AU'] --browsers=['chrome','firefox','ie']"
+ "test:wdio:local": "terra wdio --gridUrl grid.test.example.com --locales de en-AU --browsers chrome firefox ie"
}
}A list of the test runner CLI options can be found here.
At this point it is recommended to do a clean install to remove stale installations of terra-toolkit. If you have the script go ahead and run npm run clean:install. Alternatively ensure that the old node_modules directory is deleted and reinstall all dependencies.
npm run clean:installFrom here you are ready to start upgrading each of the spec files within your project. We recommend you uplifting each spec one by one by modifying the wdio.config.js. We used this strategy ourselves during our uplift and we think it helps.
Choose a spec file to uplift and limit the test run to only that file.
// wdio.conf.js
const { config } = require('@cerner/terra-functional-testing');
config.specs = [
'/tests/wdio/path/example-spec',
]
exports.config = config;Here are some resources you'll want to bookmark as you work through each spec file.
Work through the spec file and update the deprecated browser commands. Read through the WebDriverIO breaking changes section above for more information.
Here are a few of the most common scenarios we encountered:
// Clicks
- browser.click('#element');
+ $('#element').click();
// Set Value
- browser.setValue('input', 'value');
+ $('input').setValue('value');
// Add Value
- browser.addValue('input', 'value');
+ $('input').addValue('value');
// Wait For Visible
- browser.waitForVisible('#element');
+ $('#element').waitForDisplayed();
// Move To
- browser.moveToObject('#element', 0, 10);
+ $('#element').moveTo({ xOffset: 0, yOffset: 10 });Update any chai assertions to expect assertions. Read through the assertion breaking changes section above for more information.
A few common changes look something like this:
- expect(true).to.be.true; // Chai assertion
+ expect(true).toBe(true); // Expect/Jest assertion.
- expect(true).to.be.false; // Chai assertion
+ expect(true).toBe(false); // Expect/Jest assertion.
- expect(1).to.equal(1); // Chai assertion
+ expect(1).toEqual(1); // Expect/Jest assertion.Ensure each Terra.validates.element and Terra.validates.screenshot is provided a unique name. Read through the screenshots section above for more information.
it('should click and validate the element', () => {
$('#element').click();
Terra.validates.element('screenshot name'); // A unique screenshot name must be provided or the command will throw an error.
});Delete all screenshots associated to that spec file and initiate the test runner to regenerate screenshots. Resolve any errors that are thrown. Ensure every test generating a screenshot provides a unique screenshot name. Once the test run is successful move on to the next spec file and repeat the process.
The following example will throw a mismatch error because two screenshots are being generated with the same name.
// example-spec.js
it('should click and validate the element', () => {
$('#element-1').click();
Terra.validates.element('screenshot name');
});
it('should click and validate the element', () => {
$('#element-2').click();
Terra.validates.element('screenshot name'); // Will throw an error because a previous screenshot was generated with the same name.
});WebDriverIO v7 was released on February 9th, 2021 and is on our radar. We'll be focusing on getting teams upgraded onto WebDriverIO v6 and node >= 10 as a stepping stone before releasing an upgrade to WebDriverIO v7. The changes from WebDriverIO v6 to v7 drop support for node 10 and appear to largely only impact typescript declarations. We'll investigate the breaking changes and work necessary for upgrading. We plan to release an upgrade in the near future pending the investigation and transition strategy for node >= 12.