Start NVDA in a HTTP server mode to extract text during website End-to-End testing

[silbinarywolf]

Is your feature request related to a problem? Please describe.

When writing end-to-end tests for website, It would be amazing if I could write test code that communicates with a running NVDA process and that allows me to extract the text its about to read so we can compare it against what we expect. This would aid in helping catch UX regressions for the blind.

I've found with experience that the accessible experience of a website will degrade over time as maintenance work continues on a website. However, we might be able to lessen that trend by making it easier for developers to test blind UX as close to reality as possible. (ie. Not using react-axe or something, but knowing precisely what NVDA will speak out to a user)

Programmers love having code that confirms their code is OK and I'm hopeful that opening up the door of what they can test could improve the internet for everybody.

Describe the solution you'd like

NVDA could have a flag to start as a silent TCP/HTTP server wherein we send certain messages and get certain results back.
Something that looks like the NVDA E2E tests, as seen here but publically exposed so other applications can call them.

• spy.wait_for_speech_to_finish()
• speech = spy.get_speech_at_index_until_now(nextSpeechIndex)

Expectations of imagined implementation:

• Should be a way to easily install NVDA onto a CI box. A one-liner to get it going would be ideal.
• Should be proven to work with an E2E tool like Cypress or Selenium. Both would need a plugin written for them so that test code can have functions like nvda.WaitForSpeechToFinish. (These could just be HTTP requests that give back the expected information)

Describe alternatives you've considered

I haven't really thought about other ways you could pull this off.

Additional context

• With Windows being supported on more CI vendors like GithubCI, CircleCI and Travis. This is more feasible than it was 2-4 years ago.

[feerrenrut]

@silbinarywolf Thanks for the well researched issue!

This is something we have had many requests for, and would likely result in improvements in our own end to end tests. That said it's hard to prioritize this over the direct needs of NVDA users. Since this would serve quite a different group (developers) with requirements not shared by end users, I think this should be built as a separate project on top of NVDA. For this reason I will close this issue despite really liking the motivation here.

You may be interested to know that the system tests currently use the remote module from the RobotFramework (XML-RPC, which is a simple remote procedure call protocol using XML over HTTP) to communicate with the NVDA process. Theoretically the code running the tests could be on a different machine than the NVDA process, though extensions would be required to start the process and send presses.

For anyone interested in this, I would advise to start by helping to improve the stability of our current end to end tests. Stability of the Chrome tests (which are currently excluded from automatic test runs) are currently holding back wider use and adoption within the project. Wider use of these "system tests" will help to mature the NVDA system testing API, meaning an external project can depend on it with reduced risk.

Please get in touch if you (or anyone else) wish to pursue this idea so that we can collaborate

[feerrenrut]

You may be interested in the work going on to use NVDA to automatically test the ARIA-AT samples: w3c/aria-at#349