Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Test Plans to the Test Queue: Plan version dropdown shows commits that do not include a change in the chosen test plan #579

Closed
mcking65 opened this issue Mar 27, 2023 · 3 comments · Fixed by #621 or #739
Assignees
Labels
bug Something isn't working

Comments

@mcking65
Copy link

Using the sandbox on March 25, 2023.

When adding a plan to the test queue, the system is surfacing commits that do not include changes to the selected plan.
This enables generation of significant downstream data integrity problems.

Test plans are versioned by their last date of change as derived from the aria-at commit history.
The test plan version is critical information when running a plan through the working mode.
For example, it is esential to know what version of a plan was run to generate a specific result set and when the plan was last changed.

To reproduce the problem:

  1. Open the add test plan experience
  2. Choose alert
  3. Note the list of versions includes every commit to the main branch in w3c/aria-at.
  4. Look at the file tree for aria-at.
    As of March 25, 2023, The last commit that change the
    Alert Test Plan Files
    was 0928bcf on December 8, 2022.

Actual behavior: Commits that do not include changes to the alert plan are listed in the version dropdown.

Expected behavior: Only commits to main with changes to the alert plan are listed.

Because of this problem, I was able to add multiple versions of the alert test plan to the test queue for VoiceOver with Chrome. The VoiceOver/Chrome table shows two rows for alert, one with a March 20 version and another with a March 8 version. This is a serious problem, because it indicates the plan has changed since December 8, but it has not changed. The ramifications of signaling change when in fact there was no change poses major risk to our ability to execute the working mode.

@mcking65 mcking65 added the bug Something isn't working label Mar 27, 2023
@ccanash ccanash assigned ccanash and unassigned ccanash Mar 29, 2023
@ccanash
Copy link
Contributor

ccanash commented Mar 29, 2023

Ready for Bocoup internal triage. It would be good to fix the dates that are shown in test queue. All should show December 8th

@ccanash ccanash assigned alflennik and unassigned ccanash Apr 12, 2023
@ccanash
Copy link
Contributor

ccanash commented Apr 12, 2023

This has been triaged and assigned to @alflennik

@mcking65
Copy link
Author

mcking65 commented Aug 2, 2023

Looking at the commit history for the alert test directory in the main branch of aria-at, I see the following versions for the alert plan:

  1. V22.12.08
  2. V22.04.14
  3. V22.03.17
  4. V21.10.15

image of alert directory commit history

In staging, When I open the add test plans to the test queue disclosure, choose alert, and look at the list of versions, the list is different. It is missing the Mar 17, 2022 version, and it has 3 others that are not in the above list: May 5, 2022, Mar 9, 2022, and Nov 18, 2021.

image of alert versions dropdown in staging

I didn't yet make any other comparisons; I checked only alert.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Development

Successfully merging a pull request may close this issue.

4 participants