Allow matcher to accept Axe results object #23

gabalafou · 2022-08-11T08:02:21Z

Current behavior before this pull request

You cannot pass anything other than a Playwright handle (page, locator, etc.) to the Expect.js matcher:

expect(page).toBeAccessible()

Behavior after this pull request

After this pull request, you should be able to pass either a Playwright handle or an Axe results object:

const { results } = await waitForAxeResults(page);
expect(results).toBeAccessible();

Background / motivation

Refer to discussion in issue #21

changeset-bot · 2022-08-11T08:02:24Z

🦋 Changeset detected

Latest commit: 1a0f2ef

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
expect-axe-playwright	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

gabalafou

done self-reviewing

README.md

gabalafou · 2022-08-11T09:44:45Z

src/utils/axe.ts

@@ -30,7 +30,7 @@ export async function injectAxe(locator: Locator) {
 * Runs axe on an element handle. The script must already be injected
 * using `injectAxe`.
 */
-export function runAxe(locator: Locator, options: RunOptions = {}) {
+export function runAxeAfterInject(locator: Locator, options: RunOptions = {}) {


Couldn't think of a better name, sorry.

mskelton · 2022-08-12T15:01:59Z

src/matchers/toBeAccessible/index.ts


    // If there are violations, attach an HTML report to the test for additional
    // visibility into the issue.
    if (!ok) {
      const html = await createHTMLReport(results)
-      const filename = opts.filename || 'axe-report.html'


Removing this is incorrect as it breaks the ability to define the filename at the project level.

mskelton · 2022-08-12T15:02:38Z

src/axe/index.ts

+  const opts = merge(info.project.use.axeOptions, options)
+  const locator = resolveLocator(handle)
+  await injectAxe(locator)
+  return await poll(locator, timeout, async () => {


The polling behavior feels like something that should be part of the matcher, not this low level utility.

mskelton · 2022-08-12T15:03:11Z

src/axe/index.ts

+import { Handle, resolveLocator } from '../utils/matcher'
+import { poll } from '../utils/poll'
+
+export async function runAxe(


This should be defined inside of utils/axe rather than the index file.

mskelton · 2022-08-12T15:03:52Z

src/utils/axe.ts

@@ -30,7 +30,7 @@ export async function injectAxe(locator: Locator) {
 * Runs axe on an element handle. The script must already be injected
 * using `injectAxe`.
 */
-export function runAxe(locator: Locator, options: RunOptions = {}) {
+export function runAxeAfterInject(locator: Locator, options: RunOptions = {}) {


checkLocator perhaps?

mskelton · 2022-08-12T15:04:21Z

src/matchers/toBeAccessible/index.ts

+    let ok: boolean
+    let results: AxeResults
+    if ((handleOrResults as AxeResults).violations) {
+      results = handleOrResults as AxeResults
+      ok = !!results.violations.length
+    } else {
+      ({ ok, results } = await runAxe(handleOrResults as Handle, { timeout, ...options }))
+    }


This code feels a bit messy. A helper function that does the conditional and returns separate objects would be cleaner.

gabalafou

Take 2! :)
I tried to implement all review suggestions.

src/matchers/toBeAccessible/index.ts

mskelton · 2022-09-08T13:45:11Z

src/waitForAxeResults.spec.ts

+    const { ok } = await waitForAxeResults(page)
+    await expect(ok).toBeTruthy()


Always prefer toBe(true) as it tests that an actual boolean was returned.

Suggested change

const { ok } = await waitForAxeResults(page)

await expect(ok).toBeTruthy()

const { ok } = await waitForAxeResults(page)

await expect(ok).toBe(true)

Additionally, could we test the results object that is returned?

mskelton · 2022-09-08T13:45:37Z

src/waitForAxeResults.spec.ts

+    const content = await readFile("inaccessible.html")
+    await page.setContent(content)
+    const { ok } = await waitForAxeResults(page)
+    await expect(ok).toBeFalsy()


Same about testing the results object and using toBe(false)

src/waitForAxeResults.ts

Co-authored-by: Mark Skelton <[email protected]>

gabalafou · 2022-09-08T18:59:40Z

Thanks for the code review! How does it look now?

mskelton · 2022-09-08T19:00:52Z

Looks like there are some ESLint and Prettier issues. Once those are fixed, looks good.

mskelton · 2022-09-09T13:40:37Z

Forgot to mention, this needs a changeset with a version bump (minor) and change notes.

gabalafou · 2022-09-27T00:48:52Z

CHANGELOG.md

+
+### Minor Changes
+
+- 4276db6: Allow matcher to accept Axe results object


Hmm. Should I squash the commits and force push on this PR before referencing the commit SHA here?

I added this commit SHA manually, but it refers to only one of several commits on this PR branch.

The changelog should not be manually edited. Please add a changeset with the change notes.

https://github.com/changesets/changesets/blob/main/docs/adding-a-changeset.md#i-am-in-a-single-package-repository

Okay, second attempt :)

I read those instructions yesterday, but I've never used this tool before, so when I ran yarn changeset yesterday and it added a file to the .changesets/ directory, and I saw that there were no other files in that directory, I thought that meant that I did something wrong or that there were more steps I needed to complete. So then I ran yarn changeset version, which added the note to the change log and the version bump.

To satisfy my own curiosity, I'm trying to wrap my mind around how this works. If you have the time, could you read what I've written below and see if I'm on the right track? No worries if not :)

Looking at the git log of the .changesets directory, it looks to me that something like the following happens:

Create a PR, push commits

Add a changeset file to the .changesets directory. (Via yarn changeset and committing and pushing the result).

Squash-merge the commits on the PR. (I assume this is how Changeset knows which SHA to use in the change log. Although I suppose it's probably smart enough to use the SHA of a merge commit.)

The GitHub action you've configured (on push to main) uses the changesets/action to create a pull request that is essentially the output of yarn changeset version.

When you merge that PR into main, the changesets/action gets triggered a second time, but this time it runs the yarn release command?

Out of curiosity, how do the GitHub releases get added and/or how are the commits tagged with the version number on GitHub? I'm guessing the Changesets Release Action takes care of all that? Since it has an input called createGithubReleases, defined as:

createGithubReleases - A boolean value to indicate whether to create Github releases after publish or not. Default to true

Yep, you're right on track with how changesets works. And yes, the action creates the GitHub release as well. It's honestly such an awesome tool for versioning, way superior to the old Lerna flow we used to use for our monorepos before. The other awesome part is it's all PR based, so you can have strict controls on push access, everything including versioning goes through a pull request, so no one needs direct push access when updating versions and publishing.

It looks really neat!

Okay so if I may bother you with another question... 😁

One thing I wonder, since the changeset file only records the delta of the semantic versioning, meaning it only records whether the set of changes represent major, minor, or patch version upgrades, it seems that the order in which PRs get merged matters.

Scenario 1:

PR Allow matcher to accept Axe results object #23 is merged, package goes from version 2.2.1 to 2.3.0

PR Rename toBeAccessible to toPassAxe #24 is merged, version 2.3.0 to 3.0.0

Scenario 2:

PR Rename toBeAccessible to toPassAxe #24 is merged, package goes from version 2.2.1 to 3.0.0

PR Allow matcher to accept Axe results object #23 is merged, version 3.0.0 to 3.0.1

In my particular case with these two PRs, I think either order is fine, but it seems like a pretty heavy assumption that this is generally the case, especially where major changes are concerned.

Order doesn't matter, all changesets will be merged to create the final version using the highest version bump. This allows notes in the changelog to still be separated by major/minor/patch, but the version bump will be correct. In this case, regardless of the merge order the new version will be 3.0.0.

Of course, that's assuming I wait to release until this PR is merged, which I'm planning on doing 🙂

That makes sense. Neat!

Thank you for answering my questions. 😃

This reverts commit 59b10cd.

mskelton · 2022-09-27T20:47:23Z

src/matchers/toBeAccessible/index.ts

    const info = test.info()
    const opts = merge(info.project.use.axeOptions, options)


This merge needs to happen before calling getResults and the opts object needs to be passed to getResults. This allows for defining config at the project level.

Oh, looks like getResults is doing that already. Interesting

Yeah, there's some repetition, but I'm not sure how to avoid it.

Do you want me to try to refactor this in a way that avoids the repetition? Or do you want some time to think about it?

Nah, it's fine. You could make a function that does it so the test.info and merge calls are consistent if you want.

src/waitForAxeResults.ts

Co-authored-by: Mark Skelton <[email protected]>

src/waitForAxeResults.ts

mskelton · 2022-09-28T13:40:39Z

Looks like there are merge conflicts

mskelton

Great work!

src/utils/options.ts

Allow matcher to accept results object

4276db6

gabalafou requested a review from a team as a code owner August 11, 2022 08:02

gabalafou requested review from avo and removed request for a team August 11, 2022 08:02

gabalafou changed the title ~~Allow matcher to accept results object~~ Allow matcher to accept Axe results object Aug 11, 2022

gabalafou commented Aug 11, 2022

View reviewed changes

gabalafou mentioned this pull request Aug 11, 2022

Discuss - expose results/errors to callers? #21

Closed

mskelton reviewed Aug 12, 2022

View reviewed changes

gabalafou added 3 commits September 7, 2022 21:47

requested changes from review

5a819e2

oops semicolons

becb144

fix readme

88908aa

gabalafou commented Sep 8, 2022

View reviewed changes

mskelton reviewed Sep 8, 2022

View reviewed changes

gabalafou mentioned this pull request Sep 8, 2022

Team update - Sep 07, 2022 Quansight-Labs/jupyter-a11y-mgmt#157

Closed

gabalafou and others added 2 commits September 8, 2022 13:37

Apply suggestions from code review

d3c05f7

Co-authored-by: Mark Skelton <[email protected]>

update tests

ca83d20

prettier

0d2e7a1

add note to changelog, bump version (minor)

59b10cd

gabalafou commented Sep 27, 2022

View reviewed changes

gabalafou added 2 commits September 27, 2022 09:06

Revert "add note to changelog, bump version (minor)"

caf2b1b

This reverts commit 59b10cd.

create changeset file

e6fd3ed

mskelton approved these changes Sep 27, 2022

View reviewed changes

Update src/waitForAxeResults.ts

1362f15

Co-authored-by: Mark Skelton <[email protected]>

gabalafou commented Sep 28, 2022

View reviewed changes

src/waitForAxeResults.ts Outdated Show resolved Hide resolved

Update src/waitForAxeResults.ts

14d5e00

gabalafou added 2 commits September 28, 2022 15:23

getOptions

58f4408

Merge remote-tracking branch 'upstream/main' into run-axe

3cb0cb4

mskelton approved these changes Sep 28, 2022

View reviewed changes

src/utils/options.ts Outdated Show resolved Hide resolved

Update src/utils/options.ts

1a0f2ef

mskelton enabled auto-merge (squash) September 28, 2022 22:27

mskelton merged commit af9d4c8 into Widen:main Sep 28, 2022

This was referenced Sep 29, 2022

Add reports key to options #22

Closed

Team update - Sep 28, 2022 Quansight-Labs/jupyter-a11y-mgmt#160

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow matcher to accept Axe results object #23

Allow matcher to accept Axe results object #23

gabalafou commented Aug 11, 2022 •

edited

Loading

changeset-bot bot commented Aug 11, 2022 •

edited

Loading

gabalafou left a comment

gabalafou Aug 11, 2022

mskelton Aug 12, 2022

mskelton Aug 12, 2022

mskelton Aug 12, 2022

mskelton Aug 12, 2022

mskelton Aug 12, 2022

gabalafou left a comment

mskelton Sep 8, 2022

mskelton Sep 8, 2022

mskelton Sep 8, 2022

gabalafou commented Sep 8, 2022

mskelton commented Sep 8, 2022

mskelton commented Sep 9, 2022

gabalafou Sep 27, 2022

mskelton Sep 27, 2022

gabalafou Sep 27, 2022

mskelton Sep 27, 2022

gabalafou Sep 27, 2022

mskelton Sep 27, 2022

gabalafou Sep 27, 2022

mskelton Sep 27, 2022

mskelton Sep 27, 2022

gabalafou Sep 28, 2022

mskelton Sep 28, 2022

mskelton commented Sep 28, 2022

mskelton left a comment

		const { ok } = await waitForAxeResults(page)
		await expect(ok).toBeTruthy()


		### Minor Changes

		- 4276db6: Allow matcher to accept Axe results object

		const info = test.info()
		const opts = merge(info.project.use.axeOptions, options)

Allow matcher to accept Axe results object #23

Allow matcher to accept Axe results object #23

Conversation

gabalafou commented Aug 11, 2022 • edited Loading

Current behavior before this pull request

Behavior after this pull request

Background / motivation

changeset-bot bot commented Aug 11, 2022 • edited Loading

🦋 Changeset detected

gabalafou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabalafou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabalafou commented Sep 8, 2022

mskelton commented Sep 8, 2022

mskelton commented Sep 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mskelton commented Sep 28, 2022

mskelton left a comment

Choose a reason for hiding this comment

gabalafou commented Aug 11, 2022 •

edited

Loading

changeset-bot bot commented Aug 11, 2022 •

edited

Loading