Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to track status of a test #300

Open
mcking65 opened this issue Sep 24, 2020 · 2 comments
Open

Add ability to track status of a test #300

mcking65 opened this issue Sep 24, 2020 · 2 comments
Labels
app enhancement New feature or request process Related to processes for governing and managing the ARIA-AT project

Comments

@mcking65
Copy link
Contributor

Tests can be draft or final/ready to run.
Discussed adding a column to the CSV file where we write assertions.

@jscholes jscholes added Agenda+Community Group To discuss in the next workstream summary meeting (usually the last teleconference of the month) test-runner tests About assistive technology tests labels Feb 2, 2021
@jscholes
Copy link
Contributor

Over the past weeks, this issue has been quite heavily referenced in the context of allowing PRs to be merged as soon as possible, reducing the likelihood of merge conflicts down the line. The idea being that tests can be merged after more minimal review, making sure that they don't break anything or have any egregious problems. More nuanced review can then be carried out by members of the community group by executing or reviewing the tests in place, with new PRs being opened to resolve any concerns.

A flag will be needed, to allow people to distinguish between test plans in a draft state and those that are ready to run/finalised. During the March 4, 2021 community group meeting though, @mcking65 raised concerns that this issue may not be as simple as a single binary state flag. The concept of whether tests are ready-to-run may be more nuanced, incorporating aspects of test versioning, AT vendor consensus/review, etc. However, we'd still like to find a way forward; maybe a short-term fix to ensure that PR merging is not unnecessarily blocked.

@jscholes
Copy link
Contributor

jscholes commented Apr 7, 2021

Notes from March 11, 2021 CG meeting:

  • Thinking about the life cycle of a test plan: having only draft and finalised statuses would be too simplistic.
  • After draft status, a test plan is ready for more public consumption, particularly by AT vendors/devs. But we want to provide interested parties with an opportunity to comment on the plan at that stage, e.g. to disagree with the included commands, assertion priorities, or any other aspect.
  • Some parties may feel uncomfortable with a test plan being finalised if its assertions and/or results paint their interests in a negative light, e.g. because it highlights a bug in a screen reader which has been/is in the process of being fixed for the next release. However, we need to timebox the feedback collection period to avoid test plan finalisation being blocked forever by such concerns.
  • The ARIA-AT project wants to feel comfortable publishing results which identify bugs in ATs, browsers or other components, as long as the relevant parties/vendors agree with the tests as they are written. The responsibility is on such parties to identify sources of concern/disagreement, though, rather than on the ARIA-AT project to await agreement for an indefinite period.

With all of the above in mind, the group seemed to agree that a "wide review" status was appropriate after draft. This would be a window of opportunity for third parties to file concerns with the project about the content of a test plan; the proposed time period was 60 days. The ARIA-AT project would have a longer window to resolve any raised issues, the proposed period was 120 days which would always give us a minimum of 60 days to carry out discussion and remediation (if a vendor only engaged on the very last day of the wide review window).

There was also discussion about how wide review windows should be scheduled. Should there be several throughout the year?

@mcking65 mcking65 added process Related to processes for governing and managing the ARIA-AT project enhancement New feature or request app and removed tests About assistive technology tests Agenda+Community Group To discuss in the next workstream summary meeting (usually the last teleconference of the month) test-runner labels May 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
app enhancement New feature or request process Related to processes for governing and managing the ARIA-AT project
Projects
None yet
Development

No branches or pull requests

2 participants