Aria AT Wireframes

Test Management

This page is accessible from the main menu. There are two sections in this page: 1) Initiate a Test Cycle and 2) Test Cycle Status

1. Initiate a Test Cycle

Here, with the click of a button, a new Test Cycle initiation can be executed. This will take the user to another page where the Test Cycle can be configured.

2. Test Cycle Status

This is a table that displays the Test Cycles that have been initiated. For every Test Cycle there's a row in the table. This table has the following columns:

1. Test Cycle: It contains a link with the name given to the Test Cycle when it was configured. Clicking this will take the admin to the Test Cycle Summary Page.
2. Date: The date in which the Test Cycle was initiated.
3. Status: In progress, raw results available, draft results available and reviewed results available.
4. More Details: The cells of this table contain buttons/links to raw results, draft results and reviewed results.

Initiating a Test Cycle

This page is generated from the Test Management Page. There are 3 main sections in this page: 1) Test Cycle Configuration, 2) Test plans that are missing assignees and 3) Test Plans Available

1. Test Cycle Configuration

This is a table where the admin can choose a custom name for the cycle, initiate a cycle from a specific commit and choose specific Assistive Technology and Browser versions. This table has the following characteristics:

1. Test Cycle name: This is a text field where a custom name can be added
2. Specific commit: From a dropdown, the admin can choose a specific commit
3. AT and Browser versions: This table has as many rows as AT and browser combinations are available. There's a row per AT and Browser combination. It has the following columns:
   1. Assistive Technology: Each cell has a dropdown that contains the available AT released versions such as: beta, stable, dev, etc
   2. Assistive Technology Version Number: This cell contains a text field where the admin can input a specific version number for the AT
   3. Browser: Each cell has a dropdown that contains the available browser stage released versions such as: beta, stable, etc
   4. Browser Version number: This cell contains a text field where the admin can input a specific version number for the Browser.

2. Test plans that are missing assignees

This table displays test plans that are missing assignees. When a Test Cycle configuration is initiated, testers are assigned automatically by the system depending on the assistive technologies and browsers they can use and have recorded in their profile page. It is important to mention that even if the admin leaves test plans that are missing testers, the Test Cycle can still be initiated. Tests without a specific assignee will be available to all testers in the system. This table has the following characteristics:

1. A status message at the top with a summary of the total number of assignees that are missing.
2. The following columns:
   1. Test Plans: Where the admin can see test plans that are missing assignees
   2. Assistive Technology and Browser: Where the admin can see the AT/Browser combination under a Test plan that require assignees.
   3. Testers: Where the admin can assign testers from a dropdown.

3. Test Plans Available

This section include all available Test Plans in the system. The admin can decide to submit them all as part of a Test Cycle or choose a subset of these. Each Test Plan has its own table with all the available AT and Browser combinations available in the system as well as the testers assigned to execute certain Test plans with certain AT and Browser combinations. This table has the following characteristics:

1. Test plan name: This is the title for the test plan. There's a checkbox next to it to select the test plan as part of the Test Cycle. All test plans are selected by default.
2. The Test Plan table has the following columns:
   1. AT and Browser: This column lists the AT and Browser combinations available in the system. One combination is displayed by row. Each includes a checkbox to make a custom selection of AT/Browsers per Test Plan.
   2. Tester 1: This includes a dropdown where tester 1 is automatically selected by the system. The admin has the ability to override this selection choosing a different tester from the dropdown.
   3. Tester 2: Same characteristics of Tester 1.

At the bottom of this page, there's a UI component that contains a checkbox to check/uncheck all test plans and a button to initiate the Test Cycle.

Test Cycle Summary Page

This page shows a high level summary of the status of a Test Cycle and is accessible from the Test Management page by clicking the name of the Test Cycle from the Test Cycle Status table. There are 3 sections in this page: 1) Summary of configurations, 2) Test plans that still need assignees and 3) Test Plans included in the Test Cycle

1. Summary of configurations

This section displays the summary of configurations chosen while initiating the Test Cycle: Test Cycle name, git commit, specific versions of AT and Browsers if any were selected.

2. Test plans that still need assignees

This table is like the one we have in the "Initiating a Test Cycle" page. The admin can still select testers if needed and the system will update this automatically.

3. Test Plans included in the Test Cycle

This table is like the one we have in the "Initiating a Test Cycle" page, where the admin configures and initiates a Test Cycle, with two addition:

1. A message with a summary of the progress for the Test Plan
2. A progress column that shows the percentage of progress and how many tests have been executed and how many are left, e.g. 6/16.

At the bottom of this page, there's a UI component that contains a checkbox to check/uncheck all test plans and a button to review the raw results for the Test Cycle.

Executing Tests

This page is also known as the "Testers Queue". It shows the Test Plans that were defined by the admin when created the Test Cycle. These Test Plans are grouped by AT and Browser combination and for every combination there is a table that includes the test plans, testers, progress and the option to initiate the execution of all test plans belonging to said AT/Browser combination. This table has the following columns:

1. Test Plan: where all the Test plans available are listed.
2. Tester 1: This include the name of the tester, unless no one has been assigned. In that case, a button to assign yourself as tester will be displayed
3. Tester 2: This includes the name of the other tester that is executing the same Test Plan using the same AT and Browser, unless there's no assignee. In that case the text: "No assignee" will be displayed.
4. Progress: It shows the percentage of progress and how many tests have been executed and how many are left, e.g. 6/16.

Once the tester has decided what AT, Browser and Test to execute, the process looks pretty similar to what we currently have in the prototype. A few changes have been made:

1. Skip a test: when the tester decides to skip a test, a dialog will open. This dialog contains details about the test being skipped as well as a button to open a Github issue to describe elaborate on the reason for skipping the test. Once the Github issue has been created, the tester can now go back to the application and from the dialog, click continue to move to the next test in the queue.
2. Save progress: if the tester needs to pause the execution of a test, they can simply click this button and leave the application.

Note: the process for executing Tests is the same we have in the prototype.

Raw Test Results

This page shows a summary of the raw results of a Test Cycle that was executed by a tester. There are 2 main sections in this page: 1) Summary of configurations 2) All the Test Plans executed as part of this cycle.

1. Summary of configurations

This section displays the summary of configurations chosen while initiating the Test Cycle: Test Cycle name, git commit, specific versions of AT and Browsers if any were selected.

2. Test Plans

Each Test plan section has two main sections: 1) Tests with conflicting results and 2) Successful Test results.

1. Tests with conflicting results

A table with a summary of the tests that have conflicting test results and need to be evaluated and approved by the admin. This table has a message with a summary at the top that shows the total number of tests with conflicting results in the test plan and the following characteristics and columns:

1. Test name: The name of the test, e.g. "Navigate into a checkbox group in interaction mode.". Next to it in the next column, sub-rows for every AT and Browser are displayed.
2. AT and Browser: Each AT and Browser row in this table has two sub-rows for every tester so results can be compared. In addition to the AT and Browser name, there's a "details" link to see the test failures table, which looks the same as the one we have in the prototype, with the exception of this being now displayed in a dialog.
3. Testers: The name of the tester that executed the test
4. Required: summary of results. Same format we have in the prototype
5. Optional: summary of results. Same format we have in the prototype
6. Unexpected Behavior
7. Action: the cells in this column have a radio button for every tester so the Admin can decide which test is correct.

Once the admin has decided what results are correct, they can click the "apply changes" button, which will subsequently update the results in the tables that have successful test results.

2. Successful Tests results

There are two ways to visualize test results in this page: 1) grouping the test plans by AT and Browser combination and 2) grouping by tests. Either option can be chosen by using a dropdown menu

Grouping by AT and Browser combination

When grouping by AT and Browser combination, a table for every combination will be displayed. Each of these tables has the following characteristics and columns:

1. Test: The name of the test e.g. "Navigate into a checkbox group in interaction mode." and a "details" link to see the test failures table, which looks the same as the one we have in the prototype, with the exception of this being now displayed in a dialog.
2. Testers: the name of the two testers that executed the test
3. Required: summary of results. Same format we have in the prototype
4. Optional: summary of results. Same format we have in the prototype
5. Unexpected Behavior

Grouping by tests

When grouping by tests, one single table is displayed per Test Plan. This is a compressed way to review results. This table has the following characteristics and columns:

1. Test: The name of the test e.g. "Navigate into a checkbox group in interaction mode." next to it, in the next column, sub-rows for every AT and Browser are displayed.
2. AT and Browser: In addition to the AT and Browser name, there's a "details" link to see the test failures table, which looks the same as the one we have in the prototype, with the exception of this being now displayed in a dialog.
3. Testers: Name of both testers that executed the tests.
4. Required: summary of results. Same format we have in the prototype
5. Optional: summary of results. Same format we have in the prototype
6. Unexpected Behavior

At the bottom of the page, there's a button to generate Draft Results

Draft Results

This page shows a summary of the results of a Test Cycle that was executed by a tester. It is generated from the Raw results and the only difference from the Raw Results page is that the "Tests with conflicting results" table is no longer displayed. At the bottom of this page, the admin is able to publish the results.