Update test authoring format to support automated tests

[alflennik]

In order to support automated tests in addition to manual tests, we will need to make some changes to how the tests are authored. This is also a great opportunity to make additional changes to fix known issues such as issue 349, 358, 358 and 363.

We have prepared a document with our high level motivations, as well as a demonstration spreadsheet in Google Sheets that shows the current tests.csv and commands.csv formats along with the new proposed formats.

I am looking forward to discussing this more and am eager to hear any feedback you have! I understand we may get a chance to discuss this in the upcoming CG meeting.

[jscholes]

@alflennik Thanks for putting this together! We definitely have some thoughts; we'll be sure to work through the proposal in detail and report back here.

One thing that stands out as needing immediate attention/clarification:

Where possible, make commands more generic, for example “push key until you reach the next form element” instead of “navigate to unchecked checkbox”.

The specificity of commands in the current system is a feature, not a bug. Navigating to the next form element is a command which screen readers support (F in NVDA, for example), and distinct from "navigate to next checkbox" (X). I think we want to continue testing both.

Finally for now, I ask people to provide comments in this GitHub thread, and not directly on the linked Google doc and spreadsheet. If there are already comments on those external assets, or new ones are added going forward, could I ask somebody to transfer them over here? This comments interface is far more accessible than what Google provides and I want to avoid conversation fragmentation.

[alflennik]

@jscholes about the comments, I just added a note to the spreadsheet and document directing any comments to this GitHub issue. There aren't any comments yet!

[alflennik]

This is an interesting point @jscholes. I see what you mean when you talk about the X and F keys. We need to test the X key, but the X key is only relevant to checkboxes. If we use a generic command like "go to the next form element" in the checkbox test suite, that would mean we are no longer testing the X key.

The current design allows us as test authors to define our own commands, but your point is already making me think about ways to update the commands.csv to make it easier to define a larger number of more specific commands. So that's probably a good place to start the conversation about the next iteration of the design.

[jscholes]

@alflennik One more thing to highlight upfront: in the proposed commands.csv tab of the spreadsheet, there is a lot of, "do X until Y", sort of stuff. The most recent resolution from Thursday CG meetings is that we don't want to do that. Instead, we want to always be explicit about command sequences.

So instead of, "press Down Arrow until you find an element with role checkbox", we want to assert that a user should have to press Down Arrow X number of times. If they press Down Arrow X number of times and don't find the checkbox, the test fails. If they press Down Arrow X number of times and find the checkbox, but also experience other unexpected behaviour, that's on a tester to report.

Ref: #363 (comment)

I'm cognizant of the fact that there doesn't seem to be much crossover between the attendance of meetings related to automation, and those related to test writing. As such, the proposed format seems to incorporate ideas that don't align with current CG consensus.

There are also a lot of aspects in this proposal that I don't really understand at first glance, and will need to dive deeper into. It may serve as an interesting, if anecdotal, datapoint that I work with these test files day in, day out, but many of these changes feel like such a radical departure from how tests are currently authored that I don't recognise them.

Looking forward to exploring and discussing further.

[alflennik]

This feedback has been super useful. I think it would be great to discuss this more before we present anything to the community group. So let me remove the agenda label, and @s3ththompson, would you mind reaching out to @jscholes and @sinabahram to schedule a call to chat about it?

I’m working on some revisions to the sheet based on the feedback, and I’ll post here when I add them into the sheet.

[s3ththompson]

hi folks, I want to apologize for not properly tracking the decisions from the last few Test Writing CG meetings and letting these details slip through the cracks. I know you've put a lot of time into discussing and building consensus. we've been in a bit of limbo on the Bocoup side, as the new team got up to speed and started onboarded (👋 @alflennik!). i'm confident that we'll be more in lockstep going forward!

[s3ththompson]

also, i agree with what Alex said that this would benefit from a more intimate discussion before we raise with the entire CG. certainly this proposal should be seen more as the start of a conversation rather than a wholesale design to be accepted or rejected.