Jest axe a11y testing by yuliacech · Pull Request #127185 · elastic/kibana

yuliacech · 2022-03-08T18:22:19Z

Summary

This PR adds jest-axe library to run axe automated a11y testing in jest. An axe_helpers file in kbn/test-jest-helpers exports a expectToBeAccessible function that can be used directly in tests. As an example, I added a11y tests for Indices tab in Index Management. To reuse a Kibana wide axe configuration, I moved a file with constants into kbn-test package.

Motivation

We can already run axe tests in functional and Cypress tests. Those tests are great for e2e testing as they work with real ES resources in a test cluster. The downside is that those tests can be slow to re-run, because often we need to spin up a new ES test cluster for a "clean slate" (for example, if a test creates an index and fails before clean up has finished). Also it's more difficult to setup an error state in these tests, for example, if a connection to ES is lost. And some conditions can't be changed from inside the test, for example when testing Cloud vs on-prem UI differences.

Because of these reasons, I personally prefer to use CITs - component integration tests. In those tests, we mount a high-order component, for example, Index Management component with tabs for indices, data streams and index templates. We mock Kibana dependencies and API requests and can test all UI interactions including loading and error states. This PR adds a possibility to also test for a11y violations on those UI states.

The downside of a11y testing in CITs that I noticed is that mocked versions of child components can cause false positives or false negatives. For example, EUI components are replaced by their mocked version defined in testenv files. This can be updated in the EUI repo (see elastic/eui#5702 and elastic/eui#5709)

…aration of concerns. Also added tests for indices tab in Index Management.

elastichelix · 2022-03-08T21:21:24Z

packages/kbn-test-jest-helpers/src/axe_helpers.ts

+  impactLevel?: ImpactValue
+): Promise<void> => {
+  const violations = await getA11yViolations(component, impactLevel);
+  expect(violations).toHaveLength(0);


Will the output for this include all of the violations?

That is a great question! I think I need to update the logic here, because it will only get 'critical' violations by default. I don't even think I need a parameter for impactLevel here. I'll ask in the a11y channel which level we want to test for.

Great call removing the impactLevel. Our goal is to find and remove all accessibility issues, regardless of their impact. It has been my experience that serious and moderate issues can cause significant UX problems in large enough quantities. They're not breaking UX issues per se, but each one causes friction and adds to the accumulated cognitive load.

elastichelix · 2022-03-08T21:23:42Z

x-pack/plugins/index_management/__jest__/client_integration/helpers/http_requests.ts

+
+    server.respondWith('GET', `${API_BASE_PATH}/mapping/:name`, [
+      status,
+      { 'Content-Type': 'application/json' },


would it make sense to represent this content type block as a constant HEADERS or something to avoid repeating or is there an expectation these might be different across tests?

yes, this file is very repetitive. I'll see how all the same lines can be refactored into a function or similar.

…es that are now in kbn-test package

…actored http requests helper file.

elasticmachine · 2022-03-10T18:00:23Z

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

elasticmachine · 2022-03-10T18:00:23Z

Pinging @elastic/kibana-accessibility (Project:Accessibility)

scottybollinger

Enterprise Search changes LGTM

elasticmachine · 2022-03-10T18:00:46Z

Pinging @elastic/apm-ui (Team:apm)

sabarasaba

nice work @yuliacech! Changes lgtm! Would love to see an issue created to patch the tests up once that eui#5709 lands in kibana 🚀

sabarasaba · 2022-03-14T09:12:57Z

x-pack/plugins/index_management/__jest__/a11y/indices_tab.a11y.test.ts

+    });
+    const { component } = testBed;
+    component.update();
+    // this is expected to fail and needs to be updated when eui#5674 is available in Kibana


👍🏼 Wondering if we should perhaps create a new issue to update this bit of code when this eui issue is fixed

I changed the comment to mention that the problem will be resolved with EUI 52.0.0 update (probably in this PR). I think we don't need an extra issue for that because when EUI version is updated, the tests will start to fail and will need to be updated as well.

sabarasaba · 2022-03-14T09:31:49Z

x-pack/plugins/index_management/__jest__/client_integration/helpers/http_requests.ts

-    ]);
-  };
+export const init = () => {
+  const server = sinon.fakeServer.create();


Slightly unrelated, but there's another pr that is trying to fix the incorrect usage of axios+sinon that we have throughout our apps in order to use HttpSetup instead. There's a few very nice abstractions that are created there that I can see you also thought about here! I'll create an issue to patch this up later on when the other PR is merged

spalger

kbn-test changes LGTM

1Copenut

Thank you for adding these tests @yuliacech. I added one comment affirming your decision to remove the impactLevel argument from the expectToBeAccessible() calls.

1Copenut · 2022-03-28T14:11:29Z

packages/kbn-test-jest-helpers/src/axe_helpers.ts

+  impactLevel?: ImpactValue
+): Promise<void> => {
+  const violations = await getA11yViolations(component, impactLevel);
+  expect(violations).toHaveLength(0);


Great call removing the impactLevel. Our goal is to find and remove all accessibility issues, regardless of their impact. It has been my experience that serious and moderate issues can cause significant UX problems in large enough quantities. They're not breaking UX issues per se, but each one causes friction and adds to the accumulated cognitive load.

kibana-ci · 2022-03-28T18:51:28Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/test`	214	224	+10
`@kbn/test-jest-helpers`	101	103	+2
total			+12

Unknown metric groups

API count

id	before	after	diff
`@kbn/test`	252	262	+10
`@kbn/test-jest-helpers`	128	132	+4
total			+14

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	7	9	+2

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	7	9	+2

History

💔 Build #34132 failed f7f9905
💔 Build #34024 failed 09fa429
💔 Build #34001 failed df62d6d
💔 Build #33963 failed f1748ff
💔 Build #29545 failed f8c0bc8
💔 Build #29542 failed 8c263df

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

yuliacech added 2 commits March 7, 2022 18:33

[A11y] Added axe testing to testBed to be used in jest tests (CITs)

72753f5

[A11y] Refactored axe helpers to be outside of testBed for better sep…

490a221

…aration of concerns. Also added tests for indices tab in Index Management.

elastichelix reviewed Mar 8, 2022

View reviewed changes

Merge branch 'main' into jest_axe_a11y_testing

00f248a

yuliacech mentioned this pull request Mar 9, 2022

[A11y] Added aria-label to the flyout testenv file elastic/eui#5702

Merged

1 task

yuliacech added 2 commits March 9, 2022 19:37

[A11y] Switched axe_helpers to use a Kibana wide axe config and rul…

3637886

…es that are now in kbn-test package

[A11y] Switched axe_helpers to use a Kibana wide axe config and rul…

a08d801

…es that are now in kbn-test package

yuliacech mentioned this pull request Mar 10, 2022

[A11y] Render aria-label as text in the icon testenv file elastic/eui#5709

Merged

1 task

yuliacech and others added 3 commits March 10, 2022 18:42

[A11y] Completed a11y tests in Index Management Indices tab. Also ref…

04be538

…actored http requests helper file.

Removed import comment

d89efe6

Merge branch 'main' into jest_axe_a11y_testing

c690915

yuliacech marked this pull request as ready for review March 10, 2022 17:57

yuliacech requested a review from a team as a code owner March 10, 2022 17:57

yuliacech requested a review from a team March 10, 2022 17:57

yuliacech requested a review from a team as a code owner March 10, 2022 17:57

yuliacech requested a review from a team March 10, 2022 17:57

yuliacech added Feature:Index Management Index and index templates UI Project:Accessibility release_note:skip Skip the PR/issue when compiling release notes Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more t// v8.2.0 labels Mar 10, 2022

scottybollinger approved these changes Mar 10, 2022

View reviewed changes

botelastic bot added the Team:APM - DEPRECATED Use Team:obs-ux-infra_services. label Mar 10, 2022

yuliacech added 2 commits March 10, 2022 19:03

Reverted yarn.lock change

8c263df

Correct yarn.lock changes

f8c0bc8

sabarasaba approved these changes Mar 14, 2022

View reviewed changes

spalger approved these changes Mar 14, 2022

View reviewed changes

dgieselaar approved these changes Mar 25, 2022

View reviewed changes

yuliacech and others added 3 commits March 28, 2022 14:08

Merge branch 'main' into jest_axe_a11y_testing

f1748ff

[IM] Updated the branch after merging main

df62d6d

[IM] Reverted unrelated ResponseError interface changes

09fa429

1Copenut reviewed Mar 28, 2022

View reviewed changes

[IM] Fixed eslint issue with kbn/test import

f7f9905

yuliacech enabled auto-merge (squash) March 28, 2022 16:42

Merge branch 'main' into jest_axe_a11y_testing

e47dd65

yuliacech disabled auto-merge March 28, 2022 18:46

yuliacech added the backport:skip This PR does not require backporting label Mar 29, 2022

yuliacech merged commit bef90a5 into elastic:main Mar 29, 2022

alisonelizabeth mentioned this pull request Mar 31, 2022

Failing test: Jest Tests.x-pack/plugins/index_management/__jest__/a11y - A11y Indices tab index details flyout settings tab #128833

Closed

yuliacech deleted the jest_axe_a11y_testing branch November 22, 2022 09:28

Conversation

yuliacech commented Mar 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 10, 2022

Uh oh!

elasticmachine commented Mar 10, 2022

Uh oh!

scottybollinger left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 10, 2022

Uh oh!

sabarasaba left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

spalger left a comment

Choose a reason for hiding this comment

Uh oh!

1Copenut left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kibana-ci commented Mar 28, 2022

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

API count

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

yuliacech commented Mar 8, 2022 •

edited

Loading