Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(#7623): Resize ConductorAxis properly #7624

Merged
merged 37 commits into from
Mar 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
30e4fef
fix: resize conductor properly
ozyx Mar 20, 2024
56b631c
refactor: more computed properties, unregister listener
ozyx Mar 20, 2024
fb85188
fix: beforeUnmounted hook
ozyx Mar 20, 2024
3702348
test(visual): add time conductor visual test for fixed mode
ozyx Mar 20, 2024
22935a0
fix: initialize to `null`
ozyx Mar 20, 2024
f3915ff
feat: extend the base `screenshot` function to mask elements which wi…
ozyx Mar 21, 2024
ca5a9da
docs: add types for fixtures
ozyx Mar 21, 2024
8989c6d
fix: remove unneeded await
ozyx Mar 22, 2024
30322b5
chore: add sinon timers types package back
ozyx Mar 22, 2024
a6b01c7
docs: remove unused docs
ozyx Mar 22, 2024
af04b1d
doc: remove unused docs
ozyx Mar 22, 2024
49b404e
test: add visual realtime url, update imports
ozyx Mar 22, 2024
0226761
feat: provide wrapped page.screenshot fixture that applies defaults
ozyx Mar 22, 2024
72fe5b1
test: add basic timeConductor snapshot tests
ozyx Mar 22, 2024
80fbeaf
chore: update eslint config
ozyx Mar 22, 2024
69d2934
lint: remove unused disable directives
ozyx Mar 22, 2024
212859e
test: remove redundant navigation
ozyx Mar 22, 2024
f38cda8
fix: remove listeners
ozyx Mar 22, 2024
7efd21e
fix: maybe stabilize unit tests
ozyx Mar 22, 2024
9f601b6
docs: remove
ozyx Mar 22, 2024
981168b
fix: provide sourcemaps in unit tests
ozyx Mar 23, 2024
a2d7bbc
test: add regression snapshot test for time conductor axis
ozyx Mar 25, 2024
328aefa
lint: remove unused imports
ozyx Mar 25, 2024
e67b69d
feat(e2e): add fixture to manually tick the clock and use it
ozyx Mar 25, 2024
7aeb59e
test: reactivate test now that we don't use deploysentinel :(
ozyx Mar 25, 2024
a67a336
test: update snapshots
ozyx Mar 25, 2024
b632c27
test: add test for clockOptions and tick fixtures
ozyx Mar 26, 2024
557c223
test: add afterEach stub and fixme
ozyx Mar 26, 2024
2e9dab1
test: try and stabilize fault management flake
ozyx Mar 26, 2024
d9e4c56
lint: defy the word gods
ozyx Mar 26, 2024
ebc4433
chore: ignore `*-darwin.png` screenshots
ozyx Mar 26, 2024
6e0284d
chore: remove darwin screenshot binaries
ozyx Mar 26, 2024
681d38c
docs: markdownlint
ozyx Mar 26, 2024
fce11e8
docs: remove MacOS specific instructions from snapshot testing
ozyx Mar 26, 2024
b89cb73
fix: remove a11y
ozyx Mar 26, 2024
2d0bca0
test: try to stabilize visual test
ozyx Mar 26, 2024
ce0fdea
fix: woops those are already ignored heehee
ozyx Mar 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions .eslintrc.cjs
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
const LEGACY_FILES = ['example/**'];
module.exports = {
/** @type {import('eslint').Linter.Config} */
const config = {
env: {
browser: true,
es6: true,
es2024: true,
jasmine: true,
amd: true
node: true,
worker: true,
serviceworker: true
},
globals: {
_: 'readonly'
Expand All @@ -23,10 +26,11 @@ module.exports = {
parser: '@babel/eslint-parser',
requireConfigFile: false,
allowImportExportEverywhere: true,
ecmaVersion: 2015,
ecmaVersion: 'latest',
ecmaFeatures: {
impliedStrict: true
}
},
sourceType: 'module'
},
rules: {
'simple-import-sort/imports': 'warn',
Expand Down Expand Up @@ -152,7 +156,7 @@ module.exports = {
cases: {
pascalCase: true
},
ignore: ['^.*\\.js$']
ignore: ['^.*\\.(js|cjs|mjs)$']
}
],
'vue/first-attribute-linebreak': 'error',
Expand All @@ -179,3 +183,5 @@ module.exports = {
}
]
};

module.exports = config;
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -47,3 +47,6 @@ index.html.bak
.nyc_output
coverage
codecov

# Don't commit MacOS screenshots
*-darwin.png
45 changes: 24 additions & 21 deletions e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,28 +76,30 @@ To read about how to write a good visual test, please see [How to write a great

`npm run test:e2e:visual` commands will run all of the visual tests against a local instance of Open MCT. If no `PERCY_TOKEN` API key is found in the terminal or command line environment variables, no visual comparisons will be made.

- `npm run test:e2e:visual:ci` will run against every commit and PR.
- `npm run test:e2e:visual:full` will run every night with additional comparisons made for Larger Displays and with the `snow` theme.
- `npm run test:e2e:visual:ci` will run against every commit and PR.
- `npm run test:e2e:visual:full` will run every night with additional comparisons made for Larger Displays and with the `snow` theme.

#### Percy.io

To make this possible, we're leveraging a 3rd party service, [Percy](https://percy.io/). This service maintains a copy of all changes, users, scm-metadata, and baselines to verify that the application looks and feels the same _unless approved by a Open MCT developer_. To request a Percy API token, please reach out to the Open MCT Dev team on GitHub. For more information, please see the official [Percy documentation](https://docs.percy.io/docs/visual-testing-basics).

At present, we are using percy with two configuration files: `./e2e/.percy.nightly.yml` and `./e2e/.percy.ci.yml`. This is mainly to reduce the number of snapshots.
At present, we are using percy with two configuration files: `./e2e/.percy.nightly.yml` and `./e2e/.percy.ci.yml`. This is mainly to reduce the number of snapshots.

### Advanced: Snapshot Testing (Not Recommended)

While snapshot testing offers a precise way to detect changes in your application without relying on third-party services like Percy.io, we've found that it doesn't offer any advantages over visual testing in our use-cases. Therefore, snapshot testing is **not recommended** for further implementation.

#### CI vs Manual Checks

Snapshot tests can be reliably executed in Continuous Integration (CI) environments but lack the manual oversight provided by visual testing platforms like Percy.io. This means they may miss issues that a human reviewer could catch during manual checks.

#### Example
A single visual test assertion in Percy.io can be executed across 10 different browser and resolution combinations without additional setup, providing comprehensive testing with minimal configuration. In contrast, a snapshot test is restricted to a single OS and browser resolution, requiring more effort to achieve the same level of coverage.

A single visual test assertion in Percy.io can be executed across 10 different browser and resolution combinations without additional setup, providing comprehensive testing with minimal configuration. In contrast, a snapshot test is restricted to a single OS and browser resolution, requiring more effort to achieve the same level of coverage.

#### Further Reading
For those interested in the mechanics of snapshot testing with Playwright, you can refer to the [Playwright Snapshots Documentation](https://playwright.dev/docs/test-snapshots). However, keep in mind that we do not recommend using this approach.

For those interested in the mechanics of snapshot testing with Playwright, you can refer to the [Playwright Snapshots Documentation](https://playwright.dev/docs/test-snapshots). However, keep in mind that we do not recommend using this approach.

#### Open MCT's implementation

Expand All @@ -118,14 +120,6 @@ When the `@snapshot` tests fail, they will need to be evaluated to determine if

To compare a snapshot, run a test and open the html report with the 'Expected' vs 'Actual' screenshot. If the actual screenshot is preferred, then the source-controlled 'Expected' snapshots will need to be updated with the following scripts.

MacOS

```
npm run test:e2e:updatesnapshots
```

Linux/CI

```sh
// Replace {X.X.X} with the current Playwright version
// from our package.json or circleCI configuration file
Expand Down Expand Up @@ -335,9 +329,11 @@ We have a Mission-need to support iPad and mobile devices. To run our test suite
In general, our test suite is not designed to run against mobile devices as the mobile experience is a focused version of the application. Core functionality is missing (chiefly the 'Create' button). To bypass the object creation, we leverage the `storageState` properties for starting the mobile tests with localstorage.

For now, the mobile tests will exist in the /tests/mobile/ suites and be executed with the

```sh
npm run test:e2e:mobile
```

command.

#### **Skipping or executing tests based on browser, os, and/os browser version:**
Expand Down Expand Up @@ -377,13 +373,15 @@ In general, strive to test only through the UI as a user would. As stated in the
By adhering to this principle, we can create tests that are both robust and reflective of actual user experiences.

#### How to make tests robust to function in other contexts (VISTA, COUCHDB, YAMCS, VIPER, etc.)

1. Leverage the use of `appActions.js` methods such as `createDomainObjectWithDefaults()`. This ensures that your tests will create unique instances of objects for your test to interact with.
1. Do not assert on the order or structure of objects available unless you created them yourself. These tests may be used against a persistent datastore like couchdb with many objects in the tree.
1. Do not search for your created objects. Open MCT does not performance uniqueness checks so it's possible that your tests will break when run twice.
1. Avoid creating locator aliases. This likely means that you're compensating for a bad locator. Improve the application instead.
1. Leverage `await page.goto('./', { waitUntil: 'domcontentloaded' });` instead of `{ waitUntil: 'networkidle' }`. Tests run against deployments with websockets often have issues with the networkidle detection.

#### How to make tests faster and more resilient

1. Avoid app interaction when possible. The best way of doing this is to navigate directly by URL:

```js
Expand All @@ -396,10 +394,11 @@ By adhering to this principle, we can create tests that are both robust and refl

1. Leverage `await page.goto('./', { waitUntil: 'domcontentloaded' });`
- Initial navigation should _almost_ always use the `{ waitUntil: 'domcontentloaded' }` option.
1. Avoid repeated setup to test a single assertion. Write longer tests with multiple soft assertions.
1. Avoid repeated setup to test a single assertion. Write longer tests with multiple soft assertions.
This ensures that your changes will be picked up with large refactors.

##### Utilizing LocalStorage

1. In order to save test runtime in the case of tests that require a decent amount of initial setup (such as in the case of testing complex displays), you may use [Playwright's `storageState` feature](https://playwright.dev/docs/api/class-browsercontext#browser-context-storage-state) to generate and load localStorage states.
1. To generate a localStorage state to be used in a test:
- Add an e2e test to our generateLocalStorageData suite which sets the initial state (creating/configuring objects, etc.), saving it in the `test-data` folder:
Expand All @@ -420,7 +419,6 @@ By adhering to this principle, we can create tests that are both robust and refl
});
```


### How to write a great test

- Avoid using css locators to find elements to the page. Use modern web accessible locators like `getByRole`
Expand All @@ -436,7 +434,7 @@ By adhering to this principle, we can create tests that are both robust and refl
await notesInput.fill(testNotes);
```

#### How to Write a Great Visual Test
#### How to Write a Great Visual Test

1. **Look for the Unknown Unknowns**: Avoid asserting on specific differences in the visual diff. Visual tests are most effective for identifying unknown unknowns.

Expand All @@ -445,23 +443,27 @@ By adhering to this principle, we can create tests that are both robust and refl
3. **Expect the Unexpected**: Use functional expect statements only to verify assumptions about the state between steps. A great visual test doesn't fail during the test itself, but rather when changes are reviewed in Percy.io.

4. **Control Variability**: Account for variations inherent in working with time-based telemetry and clocks.
- Utilize `percyCSS` to ignore time-based elements. For more details, consult our [percyCSS file](./.percy.ci.yml).
- Use Open MCT's fixed-time mode unless explicitly testing realtime clock
- Employ the `createExampleTelemetryObject` appAction to source telemetry and specify a `name` to avoid autogenerated names.
- Avoid creating objects with a time component like timers and clocks.

- Utilize `percyCSS` to ignore time-based elements. For more details, consult our [percyCSS file](./.percy.ci.yml).
- Use Open MCT's fixed-time mode unless explicitly testing realtime clock
- Employ the `createExampleTelemetryObject` appAction to source telemetry and specify a `name` to avoid autogenerated names.
- Avoid creating objects with a time component like timers and clocks.

5. **Hide the Tree and Inspector**: Generally, your test will not require comparisons involving the tree and inspector. These aspects are covered in component-specific tests (explained below). To exclude them from the comparison by default, navigate to the root of the main view with the tree and inspector hidden:
- `await page.goto('./#/browse/mine?hideTree=true&hideInspector=true')`

6. **Component-Specific Tests**: If you wish to focus on a particular component, use the `/visual-a11y/component/` folder and limit the scope of the comparison to that component. For instance:

```js
await percySnapshot(page, `Tree Pane w/ single level expanded (theme: ${theme})`, {
scope: treePane
});
```

- Note: The `scope` variable can be any valid CSS selector.

7. **Write many `percySnapshot` commands in a single test**: In line with our approach to longer functional tests, we recommend that many test percySnapshots are taken in a single test. For instance:

```js
//<Some interesting state>
await percySnapshot(page, `Before object expanded (theme: ${theme})`);
Expand Down Expand Up @@ -511,6 +513,7 @@ test.describe('foo test suite', () => {
});
});
```

More info and options for `overrideClock` can be found in [baseFixtures.js](baseFixtures.js)

- Working with multiple pages
Expand Down Expand Up @@ -539,7 +542,6 @@ const key = getFirstKeyFromOpenMctJson(jsonData);
expect(jsonData.openmct[key]).toHaveProperty('name', 'e2e folder');
```


### Reporting

Test Reporting is done through official Playwright reporters and the CI Systems which execute them.
Expand Down Expand Up @@ -615,6 +617,7 @@ A single e2e test in Open MCT is extended to run:
### Writing Tests

Playwright provides 3 supported methods of debugging and authoring tests:

- A 'watch mode' for running tests locally and debugging on the fly
- A 'debug mode' for debugging tests and writing assertions against tests
- A 'VSCode plugin' for debugging tests within the VSCode IDE.
Expand Down
62 changes: 51 additions & 11 deletions e2e/avpFixtures.js
Original file line number Diff line number Diff line change
Expand Up @@ -36,27 +36,67 @@
import AxeBuilder from '@axe-core/playwright';
import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';

import { expect, test } from './pluginFixtures.js';

// Constants for repeated values
const TEST_RESULTS_DIR = './test-results';
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const TEST_RESULTS_DIR = path.join(__dirname, './test-results');

const extendedTest = test.extend({
/**
* Overrides the default screenshot function to apply default options that should apply to all
* screenshots taken in the AVP tests.
*
* @param {import('@playwright/test').PlaywrightTestArgs} args - The Playwright test arguments.
* @param {Function} use - The function to use the page object.
* Defaults:
* - Disables animations
* - Masks the clock indicator
* - Masks the time conductor last update time in realtime mode
* - Masks the time conductor start bounds in fixed mode
* - Masks the time conductor end bounds in fixed mode
*/
page: async ({ page }, use) => {
const playwrightScreenshot = page.screenshot;

/**
* Override the screenshot function to always mask a given set of locators which will always
* show variance across screenshots. Defaults may be overridden by passing in options to the
* screenshot function.
* @param {import('@playwright/test').PageScreenshotOptions} options - The options for the screenshot.
* @returns {Promise<Buffer>} Returns the screenshot as a buffer.
*/
page.screenshot = async function (options = {}) {
const mask = [
this.getByLabel('Clock Indicator'), // Mask the clock indicator
ozyx marked this conversation as resolved.
Show resolved Hide resolved
this.getByLabel('Last update'), // Mask the time conductor last update time in realtime mode
this.getByLabel('Start bounds'), // Mask the time conductor start bounds in fixed mode
this.getByLabel('End bounds') // Mask the time conductor end bounds in fixed mode
];

const result = await playwrightScreenshot.call(this, {
animations: 'disabled',
mask,
...options // Pass through or override any options
});
return result;
};

await use(page);
}
});

/**
* Scans for accessibility violations on a page and writes a report to disk if violations are found.
* Automatically asserts that no violations should be present.
*
* @typedef {Object} GenerateReportOptions
* @property {string} [reportName] - The name for the report file.
*
* @param {import('playwright').Page} page - The page object from Playwright.
* @param {string} testCaseName - The name of the test case.
* @param {GenerateReportOptions} [options={}] - The options for the report generation.
*
* @returns {Promise<object|null>} Returns the accessibility scan results if violations are found,
* otherwise returns null.
* @param {{ reportName?: string }} [options={}] - The options for the report generation.
* @returns {Promise<Object|null>} Returns the accessibility scan results if violations are found, otherwise returns null.
*/
/* eslint-disable no-undef */

export async function scanForA11yViolations(page, testCaseName, options = {}) {
const builder = new AxeBuilder({ page });
builder.withTags(['wcag2aa']);
Expand Down Expand Up @@ -93,4 +133,4 @@ export async function scanForA11yViolations(page, testCaseName, options = {}) {
}
}

export { expect, test };
export { expect, extendedTest as test };
53 changes: 41 additions & 12 deletions e2e/baseFixtures.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
/* eslint-disable no-undef */
/*****************************************************************************
* Open MCT, Copyright (c) 2014-2024, United States Government
* as represented by the Administrator of the National Aeronautics and Space
Expand Down Expand Up @@ -111,6 +110,40 @@ const extendedTest = test.extend({
scope: 'test'
}
],
/**
* Exposes a function to manually tick the clock. This is useful when overriding the clock to not
* tick (`shouldAdvanceTime: false`) for visual tests, as events such as re-renders and router params
* updates are clock-driven and must be manually ticked.
*
* Usage:
* ```js
* test.describe('Manual Clock Tick', () => {
* test.use({
* clockOptions: {
* now: MISSION_TIME, // Set to the desired time
* shouldAdvanceTime: false // Clock overridden to no longer tick
* }
* });
* test('Visual - Manual Clock Tick', async ({ page, tick }) => {
* // Tick the clock 2 seconds in the future
* await tick(2000);
* });
* });
* ```
*
* @param {Object} param0
* @param {import('@playwright/test').Page} param0.page
* @param {import('@playwright/test').Use} param0.use
*/
tick: async ({ page }, use) => {
// eslint-disable-next-line func-style
const tick = async (milliseconds) => {
await page.evaluate((_milliseconds) => {
window.__clock.tick(_milliseconds);
}, milliseconds);
ozyx marked this conversation as resolved.
Show resolved Hide resolved
};
await use(tick);
},
/**
* Extends the base context class to add codecoverage shim.
* @see {@link https://github.com/mxschmitt/playwright-test-coverage Github Project}
Expand Down Expand Up @@ -154,17 +187,13 @@ const extendedTest = test.extend({
// function in the generatorWorker context. This is necessary
// to ensure that example telemetry data is generated for the new clock time.
if (clockOptions?.now !== undefined) {
page.on(
'worker',
(worker) => {
if (worker.url().includes('generatorWorker')) {
worker.evaluate((time) => {
self.Date.now = () => time;
});
}
},
clockOptions.now
);
page.on('worker', (worker) => {
if (worker.url().includes('generatorWorker')) {
worker.evaluate((time) => {
self.Date.now = () => time;
}, clockOptions.now);
}
});
}

// Capture any console errors during test execution
Expand Down
Loading
Loading