Skip to content

Tests: Shuffle test cases in CI - #83111

Merged
ciampo merged 6 commits into
trunkfrom
codex/vitest-test-shuffle-ci
Sep 25, 2026
Merged

ciampo merged 6 commits into
trunkfrom
codex/vitest-test-shuffle-ci

Conversation

@ciampo

@ciampo ciampo commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Follow up to #80855.

What?

Shuffle tests inside each file as well as files in the existing Node/jsdom and Chromium CI jobs. Use a different seed for each new workflow run.

Why?

File shuffling alone cannot expose shared state between tests in the same file. Varying the seed exercises more orders, while retaining it makes failures reproducible.

How?

Enable both shuffle flags and pass GITHUB_RUN_NUMBER from CI. All shards and Chromium share the seed; reruns retain it. Each job logs its seed and reproduction command. Local runs use Vitest's current-time default unless an explicit --sequence.seed is supplied. Update the command policy and contributor guide to match.

Repair three setup dependencies exposed by seven fresh random seeds: initialize the clipboard before installing spies in Compose unmount tests, mark the seeded core-data record as resolved before editing it, and initialize the VIPS worker module within the cancellation test group. The cancellation tests also verify worker delegation when it returns false.

Testing Instructions

  1. Run npm run test:unit:vitest:shuffled -- --sequence.seed=12345 --project=node --project=jsdom --shard=1/4. Repeat for shards 2/4, 3/4, and 4/4 on Node 24 and Node 26.
  2. On Node 24, run npm run test:unit:vitest:shuffled -- --sequence.seed=12345 --project=browser.
  3. Change the seed to exercise another order. To reproduce a CI failure, copy its logged command and use the same checkout and Node.js version.
  4. Check that a new CI run logs a new seed and a rerun retains the original seed.
Verification

All 63 matrix cases passed on a5f05d10e2d60111aa7e9d3fc339ad912514c523, based on trunk 72e0cc4f894dde390838f64e588cc25b294c0fa3, with no failures or retries in the final sweep. Each seed ran all four Node/jsdom shards on Node 24.20.0 and Node 26.9.0, plus Chromium on Node 24.20.0. The seven seeds were drawn without replacement: 1244467973, 518844352, 317867813, 189184704, 372323832, 1484297706, 2139874380. Local runs used macOS arm64, two workers per Node shard, and separate Chromium runs.

Every Node runtime/seed passed 35,185 tests; every Chromium run passed 121 files and 1,680 tests. Existing expected-failure cases and skips were unchanged.

Each setup defect was reproduced before its repair. All four repaired files passed all seven seeds on both Node versions. The clipboard and VIPS failures also passed individually after failing alone before the fixes. Build, type checking, lint, test conventions, routing, and independent adversarial review passed.

Unit CI also passed on this commit. All nine shuffled jobs logged and used seed 179071; the other GitHub checks passed or were intentionally skipped.

All 58 infrastructure tests passed with explicit and automatic local seeds. A shell probe checked 27 actual workflow-command invocations across runtime/shard combinations, new run numbers, and a rerun, verifying seed forwarding and reproduction commands.

Prerequisite repairs and verification limits

Merged prerequisites cover mutable fixtures and registries in #83098, #83099, #83104, and #83107; UI setup in #83101 and #83103; RTC timer teardown in #83366; LinkControl readiness in #83489; and Browser iframe lookup in #83495. The timer, readiness, and iframe repairs address races, rather than proven dependence between test cases.

An earlier sweep on c66b6a20 hit a native Node 26/V8 memory abort at seed 189184704, shard 4/4. The exact command passed on isolated replay without source changes. Its cause remains unexplained.

Two earlier diagnostic Browser runs at seed 80856 reported eight other timeouts in Composite, Popover, Autocomplete, SearchableSelect, PaletteEdit, and FontSizePicker tests. Their cause remains unknown. The saved logs and current cleanup did not justify a repair; subsequent passing runs do not establish that those timeouts are fixed.

No assertions are weakened, and isolation, timeouts, retries, and skips are unchanged. No finite set of passing seeds proves order independence.

Testing Instructions for Keyboard

Not applicable. This PR changes test infrastructure and test setup only.

Use of AI Tools

Codex rebased the branch, implemented the seed changes and test repairs, performed independent adversarial reviews, and ran verification.

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

🤖 PR meta 🤖

📦 Bundle size

Size Change: -16 B (0%)

Total Size: 8.22 MB

📦 View Changed
Filename Size Change
build/scripts/editor/index.min.js 613 kB -16 B (0%)

4912a1d Run

@ciampo ciampo added the [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. label Sep 22, 2026
@ciampo
ciampo changed the base branch from codex/vitest-order-data-deprecation to codex/vitest-core-data-timer-cleanup September 22, 2026 15:06
@ciampo
ciampo force-pushed the codex/vitest-core-data-timer-cleanup branch 2 times, most recently from 384fc97 to 7175ce6 Compare September 24, 2026 10:43
@ciampo
ciampo force-pushed the codex/vitest-test-shuffle-ci branch from 53ba41e to 2581a56 Compare September 24, 2026 11:10
@ciampo
ciampo changed the base branch from codex/vitest-core-data-timer-cleanup to codex/vitest-link-control-readiness September 24, 2026 11:10
@ciampo
ciampo force-pushed the codex/vitest-test-shuffle-ci branch from 2581a56 to 27556ae Compare September 24, 2026 12:59
@ciampo
ciampo changed the base branch from codex/vitest-link-control-readiness to codex/vitest-live-browser-frame September 24, 2026 12:59
@ciampo
ciampo force-pushed the codex/vitest-live-browser-frame branch from 9952050 to 25fc936 Compare September 24, 2026 15:32
Base automatically changed from codex/vitest-live-browser-frame to trunk September 24, 2026 16:00
@ciampo
ciampo force-pushed the codex/vitest-test-shuffle-ci branch from 27556ae to 6dc9efc Compare September 24, 2026 16:45
@ciampo
ciampo marked this pull request as ready for review September 24, 2026 17:10
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: ciampo <mciampini@git.wordpress.org>
Co-authored-by: manzoorwanijk <manzoorwanijk@git.wordpress.org>
Co-authored-by: Mamaduka <mamaduka@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

@ciampo
ciampo requested a review from desrosj as a code owner September 24, 2026 17:27
@github-actions github-actions Bot added the [Package] Compose /packages/compose label Sep 24, 2026
@ciampo
ciampo requested a review from Mamaduka as a code owner September 24, 2026 19:52
@github-actions github-actions Bot added the [Package] Core data /packages/core-data label Sep 24, 2026
@ciampo ciampo self-assigned this Sep 24, 2026

@manzoorwanijk manzoorwanijk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. Just a couple of inline comments

Comment thread .github/workflows/unit-test.yml
Comment thread .github/workflows/unit-test.yml Outdated

@Mamaduka Mamaduka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Thanks, @ciampo!

@ciampo
ciampo force-pushed the codex/vitest-test-shuffle-ci branch from a5f05d1 to 4912a1d Compare September 25, 2026 09:08
@ciampo
ciampo merged commit 8a6dfa7 into trunk Sep 25, 2026
70 of 72 checks passed
@ciampo
ciampo deleted the codex/vitest-test-shuffle-ci branch September 25, 2026 09:20
@github-actions github-actions Bot added this to the Gutenberg 24.2 milestone Sep 25, 2026
manzoorwanijk added a commit that referenced this pull request Sep 28, 2026
The dependency is evaluated once per file, so the assertion that it leaves
foreign markup alone only holds on the first import. CI shuffles test order
within a file since #83111, which makes ordering an unreliable guarantee.
manzoorwanijk added a commit that referenced this pull request Sep 30, 2026
The dependency is evaluated once per file, so the assertion that it leaves
foreign markup alone only holds on the first import. CI shuffles test order
within a file since #83111, which makes ordering an unreliable guarantee.
manzoorwanijk added a commit that referenced this pull request Sep 30, 2026
#83132)

* Playlist: Replace the waveform player patch with the dependency's opt-out

The `@arraypress/waveform-player` patch disabled the dependency's
document-wide scan for `[data-waveform-player]` and removed its
`data-play-icon` / `data-pause-icon` parsing.

1.26.0 ships a supported opt-out for the scan, so the patch is no longer
needed. Apply `data-waveform-autoinit="false"` before the dependency is
imported, then restore the document's own value: the opt-out is
page-global, and leaving it set would suppress declarative players
belonging to other plugins.

Imports are hoisted, so the opt-out lives in its own module and the
dependency gets a single wrapped import site, enforced with
`no-restricted-imports`. With the scan off, only containers the Playlist
block builds itself are initialized, and those never carry icon
attributes.

* Playlist: Harden the waveform player opt-out against review findings

Assert the opt-out at the moment the dependency is evaluated, with a mock
factory that records the attribute it sees. The existing test only
observes the real scan, which `vi.resetModules()` cannot re-run for an
externalized dependency, so it is meaningful on the first import alone.

Extend the restricted import to the dependency's `./dist/*` subpaths,
which `no-restricted-imports` paths matching does not cover.

* Playlist: Add the PR reference to the changelog entry

* Playlist: Import the waveform player's `no-autoinit` entry

1.27.0 ships `@arraypress/waveform-player/no-autoinit`, the same library
without the import-time scan for `[data-waveform-player]`. Importing it
directly replaces the `data-waveform-autoinit` set-and-restore wrapper,
along with its import-order constraint.

The restricted import now bans the default entry and every other subpath,
allowing only `no-autoinit`.

* Playlist: Isolate the waveform player auto-init test

The dependency is evaluated once per file, so the assertion that it leaves
foreign markup alone only holds on the first import. CI shuffles test order
within a file since #83111, which makes ordering an unreliable guarantee.

Co-authored-by: manzoorwanijk <manzoorwanijk@git.wordpress.org>
Co-authored-by: jeryj <jeryj@git.wordpress.org>
Co-authored-by: ciampo <mciampini@git.wordpress.org>
Co-authored-by: scruffian <scruffian@git.wordpress.org>
widoz pushed a commit to widoz/gutenberg that referenced this pull request Oct 2, 2026
* Tests: Shuffle test cases in CI

* Tests: Vary shuffle seeds between CI runs

* Tests: Initialize clipboard mocks before unmount test spies

* Tests: Mark the seeded core-data record as resolved

* Tests: Initialize vips worker before cancellation tests

* Tests: Clarify shuffled failure handling and simplify logs

---

Co-authored-by: ciampo <mciampini@git.wordpress.org>
Co-authored-by: manzoorwanijk <manzoorwanijk@git.wordpress.org>
Co-authored-by: Mamaduka <mamaduka@git.wordpress.org>
widoz pushed a commit to widoz/gutenberg that referenced this pull request Oct 2, 2026
WordPress#83132)

* Playlist: Replace the waveform player patch with the dependency's opt-out

The `@arraypress/waveform-player` patch disabled the dependency's
document-wide scan for `[data-waveform-player]` and removed its
`data-play-icon` / `data-pause-icon` parsing.

1.26.0 ships a supported opt-out for the scan, so the patch is no longer
needed. Apply `data-waveform-autoinit="false"` before the dependency is
imported, then restore the document's own value: the opt-out is
page-global, and leaving it set would suppress declarative players
belonging to other plugins.

Imports are hoisted, so the opt-out lives in its own module and the
dependency gets a single wrapped import site, enforced with
`no-restricted-imports`. With the scan off, only containers the Playlist
block builds itself are initialized, and those never carry icon
attributes.

* Playlist: Harden the waveform player opt-out against review findings

Assert the opt-out at the moment the dependency is evaluated, with a mock
factory that records the attribute it sees. The existing test only
observes the real scan, which `vi.resetModules()` cannot re-run for an
externalized dependency, so it is meaningful on the first import alone.

Extend the restricted import to the dependency's `./dist/*` subpaths,
which `no-restricted-imports` paths matching does not cover.

* Playlist: Add the PR reference to the changelog entry

* Playlist: Import the waveform player's `no-autoinit` entry

1.27.0 ships `@arraypress/waveform-player/no-autoinit`, the same library
without the import-time scan for `[data-waveform-player]`. Importing it
directly replaces the `data-waveform-autoinit` set-and-restore wrapper,
along with its import-order constraint.

The restricted import now bans the default entry and every other subpath,
allowing only `no-autoinit`.

* Playlist: Isolate the waveform player auto-init test

The dependency is evaluated once per file, so the assertion that it leaves
foreign markup alone only holds on the first import. CI shuffles test order
within a file since WordPress#83111, which makes ordering an unreliable guarantee.

Co-authored-by: manzoorwanijk <manzoorwanijk@git.wordpress.org>
Co-authored-by: jeryj <jeryj@git.wordpress.org>
Co-authored-by: ciampo <mciampini@git.wordpress.org>
Co-authored-by: scruffian <scruffian@git.wordpress.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

[Package] Compose /packages/compose [Package] Core data /packages/core-data [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants