Skip to content

test: remove spec directories with the retrying async helper, since rmSync never retries on Windows - #622

Merged
mrgoonie merged 4 commits into
mainfrom
test/563-async-spec-cleanup
Oct 7, 2026
Merged

mrgoonie merged 4 commits into
mainfrom
test/563-async-spec-cleanup

Conversation

@mrgoonie

@mrgoonie mrgoonie commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #563. Refs #535, #543, #561.

Why

On Windows, rmSync(path, { recursive: true, force: true, maxRetries }) reports a held directory as EPERM
(Node 24) or EBUSY (Node 22) at once and never runs the retries. fs.promises.rm with the same options does retry. Measured on this
machine (Windows 11, Node v24.11.0). The directory was held as the working directory of a child process for 1.5 s:

Form Result
rmSync(..., { maxRetries: 10, retryDelay: 100 }) threw EPERM after 0 ms
await rm(..., { maxRetries: 10, retryDelay: 100 }) removed the directory after 1536 ms, once the child exited

A held directory is also why many spec cleanups asked for retries in the first place.

What changes

  • Every retrying rmSync in test code now awaits removeTestDirectory() from tools/test-cleanup.ts (added by
    test: wait for a timed-out body before cleanup, and retry removal where Windows retries #561). This covers 58 spec files and the apps/runtime/test/live-nodes.ts helper: 64 call sites in total. Each
    afterEach/afterAll hook that cleans up became async, and rmSync was dropped from imports that no longer use it.
    Calls that were already in an async context were changed in place:
    • the finally blocks in coding-journey and service-container-engine;
    • the cleanup list in task-dispatch-scoped, whose steps are already awaited;
    • liveNodes().stopAll();
    • the image-generator reference app's spec, which already imports repo sources.
  • Two widget-dev tests keep a synchronous rmSync, now without maxRetries. They delete a watched folder and make
    it again within one turn of the event loop, and that synchronous step is what they test. A comment says why. The
    other widget-dev deletions use the helper.
  • New invariant test-cleanup-retries-asynchronously (tools/invariants/, wired into pnpm invariants). It fails
    when test code calls rmSync( with maxRetries. Test code means specs, plus the files under test/ and e2e/
    folders in apps, packages, packs, tools and examples. The check finds each call's closing parenthesis by
    counting brackets, so an options object that spans several lines is still read whole. The failure message names the
    file and line and points to the helper. On main, it reports all 64 call sites. On this branch it passes.
  • New spec tools/test/test-cleanup-retries.spec.ts:
    • unit tests for the invariant's matcher: a call that spans several lines, the promise form, calls without retries,
      and which paths count as test code;
    • a Windows-only test of the real behaviour, using a child process that keeps the directory as its working directory
      until it is released. In that test:
      • the synchronous form with maxRetries: 10 throws EPERM/EBUSY in under 500 ms, and the directory remains;
      • removeTestDirectory() succeeds only after the holder is released 700 ms later, which shows that the retries ran.

Changes after review (round 1)

  • Merged main. fix(browser): close the browser when its page cannot be opened #587's packs/browser-playwright/test/new-page-failure.spec.ts is converted: its afterAll is async and awaits removeTestDirectory(dir). Without this change, the merged tree failed the new invariant.
  • tools/smoke-widget-tooling.mjs is converted, and the invariant now covers it, because CI runs it on windows-latest. Its cleanUp() was already async, and it now awaits rm from node:fs/promises with the same retries.
  • The invariant reads code, not comments or strings.
    • It blanks comments and the text of string literals before it matches. A call named in a comment or string is no longer flagged, and a ( inside a string no longer makes it read to the end of the file. Code inside a template literal's ${...} is still read.
    • It follows rmSync under a name the file gives it: rmSync as x, { rmSync: x }, or const x = rmSync.
    • A call that shows the failing form on purpose carries // invariant-allow: sync-rm-retries. Only the Windows test in the new spec does.
    • Known limits, documented in the module: options passed in a variable, and a regular expression literal that holds a quote.
  • New matcher unit tests, one per case: a call in a comment, a parenthesis in a string, a call inside a string, template-literal expressions, each alias form, the allow marker, and the smoke script's scope.
  • The doc comments no longer say "Under Node 24": Node 22 throws at once too.

Production code is out of scope and tracked in #626: apps/runtime/src/worker-process.ts:344 and apps/runtime/src/application/widget-dev-sessions.ts:645 still call rmSync with maxRetries.

Verification (Windows 11, Node v24.11.0, before review)

Check Result
pnpm invariants 15/15 pass. On main, the new check fails with 64 call sites.
pnpm typecheck exit 0
pnpm lint exit 0
The 63 specs this touches, plus the 4 live-nodes.ts users, 3 runs in a row on the final head 63/63 files and 1054 tests passed on each run (an earlier set of runs had failures, explained below)
tools/test/test-cleanup-retries.spec.ts, 5 runs in a row 5/5 each run
pnpm test (full suite) 569 files passed, 7665 tests passed, exit 0

An earlier set of three batch runs had failures, and none came from the converted cleanup:

  • In two of those runs, the first version of the Windows test failed. It used a child process with a fixed lifetime,
    and under load the synchronous removal succeeded, so the child was not holding the directory at that moment. The
    child now reports that it is running and holds the directory until it is released. The test then passed 5 runs in a
    row on its own and in all 3 later batch runs.
  • In one of those runs, two tests in peer-delegation.spec.ts failed. It then passed 5 runs in a row on its own, and
    in all 3 later batch runs.

After review, at the new head (Windows 11, Node v24.11.0):

Check Result
tools/test/test-cleanup-retries.spec.ts, 3 runs in a row 10/10 each run
packs/browser-playwright/test/new-page-failure.spec.ts 1/1
pnpm invariants 15/15 pass
pnpm typecheck exit 0
eslint on the changed files exit 0
node --check tools/smoke-widget-tooling.mjs exit 0

Overlap with open PRs

Follow-ups (not in this PR)

…mSync never retries on Windows

Under Node 24 on Windows, rmSync with maxRetries reports a held directory as EPERM at once, so the retries in about sixty spec cleanups never ran. Each now awaits removeTestDirectory() from tools/test-cleanup.ts, whose promise form does retry. The two widget dev tests that delete and remake a folder within one turn of the event loop keep a synchronous rmSync without retries.

A new invariant, test-cleanup-retries-asynchronously, fails when test code calls rmSync with maxRetries, and a Windows-only spec shows a directory held as a child process's working directory: the synchronous form fails at once, and the helper removes it once the holder lets go.

Refs #535, #543
…, and read comments and strings out of the retry check

The check now blanks comments and string text before it reads a file, so a call named in a comment is not a call and a parenthesis in a string no longer unbalances the call around it. It follows rmSync under a name the file gives it, honours an invariant-allow marker for the failing form shown on purpose, and covers the widget tooling smoke CI runs on Windows.
…leanup

# Conflicts:
#	apps/runtime/test/widget-perform.spec.ts
@mrgoonie

mrgoonie commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Review attestation: ready to merge at 5c89d6d6ac23b99ad23e0c73b77cc6bcf430fa24, reviewed by agent:code-reviewer.

A push to this PR makes this attestation stale; the new head needs its own review.

@mrgoonie
mrgoonie enabled auto-merge (squash) October 7, 2026 22:07
@mrgoonie
mrgoonie merged commit 31949eb into main Oct 7, 2026
22 of 23 checks passed
@mrgoonie
mrgoonie deleted the test/563-async-spec-cleanup branch October 7, 2026 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: rmSync maxRetries never retries on Windows; move spec cleanup to the async retrying helper

1 participant