Repository navigation
Tests: Shuffle test cases in CI - #83111
Merged
Merged
Conversation
🤖 PR meta 🤖📦 Bundle sizeSize Change: -16 B (0%) Total Size: 8.22 MB 📦 View Changed
|
ciampo
changed the base branch from
codex/vitest-order-data-deprecation
to
codex/vitest-core-data-timer-cleanup
September 22, 2026 15:06
ciampo
force-pushed
the
codex/vitest-core-data-timer-cleanup
branch
2 times, most recently
from
September 24, 2026 10:43
384fc97 to
7175ce6
Compare
ciampo
force-pushed
the
codex/vitest-test-shuffle-ci
branch
from
September 24, 2026 11:10
53ba41e to
2581a56
Compare
ciampo
changed the base branch from
codex/vitest-core-data-timer-cleanup
to
codex/vitest-link-control-readiness
September 24, 2026 11:10
ciampo
force-pushed
the
codex/vitest-test-shuffle-ci
branch
from
September 24, 2026 12:59
2581a56 to
27556ae
Compare
ciampo
changed the base branch from
codex/vitest-link-control-readiness
to
codex/vitest-live-browser-frame
September 24, 2026 12:59
ciampo
force-pushed
the
codex/vitest-live-browser-frame
branch
from
September 24, 2026 15:32
9952050 to
25fc936
Compare
ciampo
force-pushed
the
codex/vitest-test-shuffle-ci
branch
from
September 24, 2026 16:45
27556ae to
6dc9efc
Compare
ciampo
marked this pull request as ready for review
September 24, 2026 17:10
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
manzoorwanijk
approved these changes
Sep 25, 2026
manzoorwanijk
left a comment
Member
There was a problem hiding this comment.
This looks good. Just a couple of inline comments
Mamaduka
approved these changes
Sep 25, 2026
ciampo
force-pushed
the
codex/vitest-test-shuffle-ci
branch
from
September 25, 2026 09:08
a5f05d1 to
4912a1d
Compare
manzoorwanijk
added a commit
that referenced
this pull request
Sep 28, 2026
The dependency is evaluated once per file, so the assertion that it leaves foreign markup alone only holds on the first import. CI shuffles test order within a file since #83111, which makes ordering an unreliable guarantee.
manzoorwanijk
added a commit
that referenced
this pull request
Sep 30, 2026
The dependency is evaluated once per file, so the assertion that it leaves foreign markup alone only holds on the first import. CI shuffles test order within a file since #83111, which makes ordering an unreliable guarantee.
manzoorwanijk
added a commit
that referenced
this pull request
Sep 30, 2026
#83132) * Playlist: Replace the waveform player patch with the dependency's opt-out The `@arraypress/waveform-player` patch disabled the dependency's document-wide scan for `[data-waveform-player]` and removed its `data-play-icon` / `data-pause-icon` parsing. 1.26.0 ships a supported opt-out for the scan, so the patch is no longer needed. Apply `data-waveform-autoinit="false"` before the dependency is imported, then restore the document's own value: the opt-out is page-global, and leaving it set would suppress declarative players belonging to other plugins. Imports are hoisted, so the opt-out lives in its own module and the dependency gets a single wrapped import site, enforced with `no-restricted-imports`. With the scan off, only containers the Playlist block builds itself are initialized, and those never carry icon attributes. * Playlist: Harden the waveform player opt-out against review findings Assert the opt-out at the moment the dependency is evaluated, with a mock factory that records the attribute it sees. The existing test only observes the real scan, which `vi.resetModules()` cannot re-run for an externalized dependency, so it is meaningful on the first import alone. Extend the restricted import to the dependency's `./dist/*` subpaths, which `no-restricted-imports` paths matching does not cover. * Playlist: Add the PR reference to the changelog entry * Playlist: Import the waveform player's `no-autoinit` entry 1.27.0 ships `@arraypress/waveform-player/no-autoinit`, the same library without the import-time scan for `[data-waveform-player]`. Importing it directly replaces the `data-waveform-autoinit` set-and-restore wrapper, along with its import-order constraint. The restricted import now bans the default entry and every other subpath, allowing only `no-autoinit`. * Playlist: Isolate the waveform player auto-init test The dependency is evaluated once per file, so the assertion that it leaves foreign markup alone only holds on the first import. CI shuffles test order within a file since #83111, which makes ordering an unreliable guarantee. Co-authored-by: manzoorwanijk <manzoorwanijk@git.wordpress.org> Co-authored-by: jeryj <jeryj@git.wordpress.org> Co-authored-by: ciampo <mciampini@git.wordpress.org> Co-authored-by: scruffian <scruffian@git.wordpress.org>
widoz
pushed a commit
to widoz/gutenberg
that referenced
this pull request
Oct 2, 2026
* Tests: Shuffle test cases in CI * Tests: Vary shuffle seeds between CI runs * Tests: Initialize clipboard mocks before unmount test spies * Tests: Mark the seeded core-data record as resolved * Tests: Initialize vips worker before cancellation tests * Tests: Clarify shuffled failure handling and simplify logs --- Co-authored-by: ciampo <mciampini@git.wordpress.org> Co-authored-by: manzoorwanijk <manzoorwanijk@git.wordpress.org> Co-authored-by: Mamaduka <mamaduka@git.wordpress.org>
widoz
pushed a commit
to widoz/gutenberg
that referenced
this pull request
Oct 2, 2026
WordPress#83132) * Playlist: Replace the waveform player patch with the dependency's opt-out The `@arraypress/waveform-player` patch disabled the dependency's document-wide scan for `[data-waveform-player]` and removed its `data-play-icon` / `data-pause-icon` parsing. 1.26.0 ships a supported opt-out for the scan, so the patch is no longer needed. Apply `data-waveform-autoinit="false"` before the dependency is imported, then restore the document's own value: the opt-out is page-global, and leaving it set would suppress declarative players belonging to other plugins. Imports are hoisted, so the opt-out lives in its own module and the dependency gets a single wrapped import site, enforced with `no-restricted-imports`. With the scan off, only containers the Playlist block builds itself are initialized, and those never carry icon attributes. * Playlist: Harden the waveform player opt-out against review findings Assert the opt-out at the moment the dependency is evaluated, with a mock factory that records the attribute it sees. The existing test only observes the real scan, which `vi.resetModules()` cannot re-run for an externalized dependency, so it is meaningful on the first import alone. Extend the restricted import to the dependency's `./dist/*` subpaths, which `no-restricted-imports` paths matching does not cover. * Playlist: Add the PR reference to the changelog entry * Playlist: Import the waveform player's `no-autoinit` entry 1.27.0 ships `@arraypress/waveform-player/no-autoinit`, the same library without the import-time scan for `[data-waveform-player]`. Importing it directly replaces the `data-waveform-autoinit` set-and-restore wrapper, along with its import-order constraint. The restricted import now bans the default entry and every other subpath, allowing only `no-autoinit`. * Playlist: Isolate the waveform player auto-init test The dependency is evaluated once per file, so the assertion that it leaves foreign markup alone only holds on the first import. CI shuffles test order within a file since WordPress#83111, which makes ordering an unreliable guarantee. Co-authored-by: manzoorwanijk <manzoorwanijk@git.wordpress.org> Co-authored-by: jeryj <jeryj@git.wordpress.org> Co-authored-by: ciampo <mciampini@git.wordpress.org> Co-authored-by: scruffian <scruffian@git.wordpress.org>
This was referenced Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow up to #80855.
What?
Shuffle tests inside each file as well as files in the existing Node/jsdom and Chromium CI jobs. Use a different seed for each new workflow run.
Why?
File shuffling alone cannot expose shared state between tests in the same file. Varying the seed exercises more orders, while retaining it makes failures reproducible.
How?
Enable both shuffle flags and pass
GITHUB_RUN_NUMBERfrom CI. All shards and Chromium share the seed; reruns retain it. Each job logs its seed and reproduction command. Local runs use Vitest's current-time default unless an explicit--sequence.seedis supplied. Update the command policy and contributor guide to match.Repair three setup dependencies exposed by seven fresh random seeds: initialize the clipboard before installing spies in Compose unmount tests, mark the seeded core-data record as resolved before editing it, and initialize the VIPS worker module within the cancellation test group. The cancellation tests also verify worker delegation when it returns false.
Testing Instructions
npm run test:unit:vitest:shuffled -- --sequence.seed=12345 --project=node --project=jsdom --shard=1/4. Repeat for shards2/4,3/4, and4/4on Node 24 and Node 26.npm run test:unit:vitest:shuffled -- --sequence.seed=12345 --project=browser.Verification
All 63 matrix cases passed on
a5f05d10e2d60111aa7e9d3fc339ad912514c523, based on trunk72e0cc4f894dde390838f64e588cc25b294c0fa3, with no failures or retries in the final sweep. Each seed ran all four Node/jsdom shards on Node 24.20.0 and Node 26.9.0, plus Chromium on Node 24.20.0. The seven seeds were drawn without replacement:1244467973,518844352,317867813,189184704,372323832,1484297706,2139874380. Local runs used macOS arm64, two workers per Node shard, and separate Chromium runs.Every Node runtime/seed passed 35,185 tests; every Chromium run passed 121 files and 1,680 tests. Existing expected-failure cases and skips were unchanged.
Each setup defect was reproduced before its repair. All four repaired files passed all seven seeds on both Node versions. The clipboard and VIPS failures also passed individually after failing alone before the fixes. Build, type checking, lint, test conventions, routing, and independent adversarial review passed.
Unit CI also passed on this commit. All nine shuffled jobs logged and used seed
179071; the other GitHub checks passed or were intentionally skipped.All 58 infrastructure tests passed with explicit and automatic local seeds. A shell probe checked 27 actual workflow-command invocations across runtime/shard combinations, new run numbers, and a rerun, verifying seed forwarding and reproduction commands.
Prerequisite repairs and verification limits
Merged prerequisites cover mutable fixtures and registries in #83098, #83099, #83104, and #83107; UI setup in #83101 and #83103; RTC timer teardown in #83366; LinkControl readiness in #83489; and Browser iframe lookup in #83495. The timer, readiness, and iframe repairs address races, rather than proven dependence between test cases.
An earlier sweep on
c66b6a20hit a native Node 26/V8 memory abort at seed189184704, shard4/4. The exact command passed on isolated replay without source changes. Its cause remains unexplained.Two earlier diagnostic Browser runs at seed
80856reported eight other timeouts in Composite, Popover, Autocomplete, SearchableSelect, PaletteEdit, and FontSizePicker tests. Their cause remains unknown. The saved logs and current cleanup did not justify a repair; subsequent passing runs do not establish that those timeouts are fixed.No assertions are weakened, and isolation, timeouts, retries, and skips are unchanged. No finite set of passing seeds proves order independence.
Testing Instructions for Keyboard
Not applicable. This PR changes test infrastructure and test setup only.
Use of AI Tools
Codex rebased the branch, implemented the seed changes and test repairs, performed independent adversarial reviews, and ran verification.