Accessibility module in WebDriver BiDi?

[OrKoN]

Puppeteer exposes the information about the browser's a11y tree in a couple of ways:

1. via Accessibility.getFullAXTree to generate a snapshot of the a11y tree
2. via Accessibility.queryAXTree to query elements by the accessible name and role. This allows Puppeteer to implement element selectors based on the a11y tree data. solved via https://w3c.github.io/webdriver-bidi/#collect-nodes-using-accessibility-attributes

Additionally, there were requests recently to include A11y tree information into WebDriver https://goo.gle/chromedriver-a11ytree. I think we should discuss if we want to add these capabilities to WebDriver BiDi.

[christian-bromann]

One could argue that a very popular use case is to fetch an element via a11y name. WebdriverIO currently computes some complex xPath to make a good guess about it according to the spec but it doesn't work all time due to xPath limitations. It the would help a lot if this could be a primitive, e.g. a selector strategy.

[clshortfuse]

I would love for this to be supported. Had a request posted in the interop issue with reasonings for it:

Currently, I would like to ensure a Web Component presents a role at a certain point in the layout tree. For example:

<x-button role=none>
  #shadowroot
  <input type=button aria-labelledby=slot>
  <slot id=slot></slot>
</x-button>

And

<x-button role=button>
  #shadowroot
  <slot id=slot></slot>
</x-button>

<x-button> <!-- role set via ElementInternals -->
  #shadowroot
  <slot id=slot></slot>
</x-button>

It would make sense that the next "meaningful" ARIA role starting from <x-button> is button. But there are multiple ways to achieve this, including hiding the attribute via ElementInternals. I think I might be able to iterate the DOM and test each element manually, but I'm worried about closed shadowRoots. I have concerns with just iterating the DOM because I'm not really sure what's being exposed by the browsers to screen-readers. There might be groups or hidden nodes. The DOM and AXTree isn't guaranteed to be 1:1.

There are many web testing suites that just read the DOM, which isn't accurate. Manually testing in the browser is somewhat tedious, but at this point has helped me recognize some browser bugs with computed labels:

• https://bugzilla.mozilla.org/show_bug.cgi?id=1826194
• https://bugs.webkit.org/show_bug.cgi?id=254934

I am also aware of other AXLabel bugs, but haven't had time to file them. But most of them deal in one way or another with complex trees.

2. Use of a tree allows expanding testing beyond just label and role.

I recently found a mistake on my part where I had a custom element like above, but using <input type=checkbox>. I used aria-checked=mixed manually, assuming browsers did not apply .indeterminate as aria-checked=mixed. I probably had to do this with IE and it's just something I stuck with. When I used a tester (Playwright) that exposes their own version of an AXTree, I learned of this Chrome bug:

• https://bugs.chromium.org/p/chromium/issues/detail?id=1260371

So, after having checked the aria tree with its properties, I was able to rewrite some code around make sure all the 3 majors browsers properly show aria-checked=mixed

Related note (Option 4). I did write something a while ago to emulate a screen reader over a virtual COM port to test JAWS/NVDA, but getting it to work on anything but Windows was too hard of a task for me at the time so I abandoned it. The Webdriver solution seems good enough for me now.

Originally posted by @clshortfuse in web-platform-tests/interop-accessibility#2 (comment)

[css-meeting-bot]

The Browser Testing and Tools Working Group just discussed Accessibility module for WebDriver BiDi.

The full IRC log of that discussion

<davehunt> topic: Accessibility module for WebDriver BiDi
<AutomatedTester> github: https://github.com//issues/443
<jgraham> q?
<davehunt> orkon: Since last year we have the ability to locate DOM elements by accessibility attributes
<davehunt> orkon: Still not supported is getting snapshots of the accessibility tree
<davehunt> whimboo: I added this as it was a request from Playwright
<orkon> q+
<simonstewart> q?
<jcraig> q+
<jgraham> q+
<Jamie> q+
<davehunt> jimevans: We've already done the location of nodes. Being able to get the snapshot would be useful to me
<simonstewart> q+
<AutomatedTester> ack orkon
<jcraig> q+ to ask the need for the tree, since the implmentations will be different across WPT tests.
<jgraham> q-
<davehunt> orkon: We can't implement because there's no agreement yet on what the accessibility tree looks like
<jcraig> ack jcraig
<Zakim> jcraig, you wanted to ask the need for the tree, since the implmentations will be different across WPT tests.
<simonstewart> q-
<whimboo> orkon: you were right. the interop 2024 initiative was the reason
<davehunt> jcraig: Lot of accessibility automation stacks being worked on. WPT tests currently exists entirely in the browser.
<jgraham> q+
<AutomatedTester> q?
<davehunt> jcraig: We have an investigation to implement a tree dump. There's going to be different trees. I'd like to understand more about the needs are for getting the tree.
<davehunt> jimevans: Cross browser tree comparisons is something that we would be interested in.
<davehunt> jcraig: It's achievable in the near future to do comparisons by walking the tree if we have limited known test cases
<AutomatedTester> ack jamie
<jcraig> s/had an investigation/have an investigation/
<gsnedders> RRSAgent, make minutes
<RRSAgent> I have made the request to generate https://www.w3.org/2024/09/27-webdriver-minutes.html gsnedders
<davehunt> Jamie: The thinking is that if we get accessible node working well then it's possible to write a tree dump as a layer on top of that. Important to get accessible node right first. I think we're heading in a direction that allows this.
<AutomatedTester> ack jgraham
<Jamie> https://github.com/web-platform-tests/interop-accessibility/issues/51#issuecomment-2099462510
<simonstewart> q?
<AutomatedTester> q?
<jcraig> TreeDump Investiigation (abandoned) https://github.com/web-platform-tests/interop-accessibility/issues/51
<davehunt> jgraham: Puppeteer exposes an API for accessibility trees, Playwright API redirects to alternate tools. Agree that don't expect the accessibility tree to be comparable between browsers in the near future.
<jcraig> Some explanation of the reasoning for moving https://github.com/web-platform-tests/interop-accessibility/issues/90
<jcraig> s/moving/moving to a treewalker plan/