You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When adding a plan to the test queue, the system is surfacing commits that do not include changes to the selected plan.
This enables generation of significant downstream data integrity problems.
Test plans are versioned by their last date of change as derived from the aria-at commit history.
The test plan version is critical information when running a plan through the working mode.
For example, it is esential to know what version of a plan was run to generate a specific result set and when the plan was last changed.
To reproduce the problem:
Open the add test plan experience
Choose alert
Note the list of versions includes every commit to the main branch in w3c/aria-at.
Actual behavior: Commits that do not include changes to the alert plan are listed in the version dropdown.
Expected behavior: Only commits to main with changes to the alert plan are listed.
Because of this problem, I was able to add multiple versions of the alert test plan to the test queue for VoiceOver with Chrome. The VoiceOver/Chrome table shows two rows for alert, one with a March 20 version and another with a March 8 version. This is a serious problem, because it indicates the plan has changed since December 8, but it has not changed. The ramifications of signaling change when in fact there was no change poses major risk to our ability to execute the working mode.
The text was updated successfully, but these errors were encountered:
In staging, When I open the add test plans to the test queue disclosure, choose alert, and look at the list of versions, the list is different. It is missing the Mar 17, 2022 version, and it has 3 others that are not in the above list: May 5, 2022, Mar 9, 2022, and Nov 18, 2021.
I didn't yet make any other comparisons; I checked only alert.
Using the sandbox on March 25, 2023.
When adding a plan to the test queue, the system is surfacing commits that do not include changes to the selected plan.
This enables generation of significant downstream data integrity problems.
Test plans are versioned by their last date of change as derived from the aria-at commit history.
The test plan version is critical information when running a plan through the working mode.
For example, it is esential to know what version of a plan was run to generate a specific result set and when the plan was last changed.
To reproduce the problem:
As of March 25, 2023, The last commit that change the
Alert Test Plan Files
was 0928bcf on December 8, 2022.
Actual behavior: Commits that do not include changes to the alert plan are listed in the version dropdown.
Expected behavior: Only commits to main with changes to the alert plan are listed.
Because of this problem, I was able to add multiple versions of the alert test plan to the test queue for VoiceOver with Chrome. The VoiceOver/Chrome table shows two rows for alert, one with a March 20 version and another with a March 8 version. This is a serious problem, because it indicates the plan has changed since December 8, but it has not changed. The ramifications of signaling change when in fact there was no change poses major risk to our ability to execute the working mode.
The text was updated successfully, but these errors were encountered: