Skip to content

fix(widget-dev): install a granted build first, and wait on the engine rather than the clock in its tests - #669

Merged
mrgoonie merged 3 commits into
mainfrom
test/656-widget-dev-flakes
Oct 8, 2026
Merged

mrgoonie merged 3 commits into
mainfrom
test/656-widget-dev-flakes

Conversation

@mrgoonie

@mrgoonie mrgoonie commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Fixes #656
Fixes #658

Summary

Three widget dev tests failed under load or on slow runners. Two of them had a test-side cause, and one exposed a real product race.

Engine: a remade folder was read half-written (#656)

The tests "builds a folder another process makes again exactly once…" and "builds a folder another process made again once…" had a child process copy the folder straight back into place with cpSync. Under load, that copy can take longer than the 30 ms debounce. The engine then sees the folder back and builds what it finds, which is half the files. That build is correct engine behaviour, so the fault was in the test.

I reproduced both CI signatures deterministically. A child that creates the folder and only some of its files, and finishes 300 ms later, gives four failed rebuilds, ["change:failed","change:generation"], and clarkcant.json does not exist.

The child now copies into a sibling folder and renames it into place, retrying briefly on Windows EPERM/EBUSY. The folder is therefore whole the moment it reappears. Both tests also assert that the first build was a generation.

Engine: the grace test counted a build from before the removal (#658)

The test "does not call a watched folder gone while it may still come back…" slept for catch-up + 100 ms before removing the folder, but never cleared built. On macOS, FSEvents can report the writes that made the folder after the watcher has started. The change build that runs because of that event is reported as unchanged, and a slow runner pushes it past the sleep.

I reproduced the exact CI failure (expected [ 'unchanged' ] to deeply equal []) by saving the same content once just before the removal.

The test no longer sleeps. Builds run one at a time, so await engine.rebuild() first waits for every build already started to end. The test then clears built and removes the folder in the same synchronous step, and a change reported later finds the folder missing. The assertion expect(built).toEqual([]) is unchanged.

Runtime: an inbox grant could fail while a newer build was waiting (#656)

"keeps the newest build's snapshot…" sometimes got 503 PACKAGE_CACHE_UNAVAILABLE from the decision. The cause is a race in the product:

  1. The session's answer poll sees the grant before the inbox has finished installing it.
  2. It installs the newer build on that consent.
  3. It prunes the granted build's snapshot while the inbox is still copying it.

In the reverse order, the older generation could instead be activated over the newer one.

follow now waits, still asking, while the install that the grant started has not recorded its outcome on the package.install-approval stream. The wait is capped at 60 s, so a missing outcome record cannot hold the session forever.

A new test holds the inbox install's first read of the granted snapshot and posts a rebuild meanwhile. It then checks that the grant succeeds, that generation 2 becomes active afterwards, and that the granted snapshot survives. On the base code it fails with expected 400 to be 200. The EN and VI docs in docs/widget-development*.md now say the granted build installs first.

Verification

Stress used 8 parallel vitest processes × 20 rounds on Windows (node vitest.mjs run <spec>):

Spec Before After
packages/core/test/widget-dev-engine.spec.ts 2 / 160 failed 0 / 160 (twice, before and after the #658 change)
apps/runtime/test/widget-dev-sessions.spec.ts 0 / 160; 2 / 280 at 28 × 10 on the flaky test (twice) 0 / 160; 0 / 280 at 28 × 10

…d removed folder tests

A folder remade by copying straight into place can be read half-written once a loaded runner stretches the copy past the debounce, and the engine rightly builds what it found. The child now copies into a sibling and renames it into place. The grace test drained by sleeping past the catch-up, so a change build a late watcher event ran (FSEvents on macOS) was counted as a report about the removal; it now drains the build queue with a rebuild before removing the folder.

Refs #656, #658
The session's answer poll saw an inbox grant before the inbox finished installing it, installed the newer build on that consent and pruned the granted build's snapshot while the inbox was still copying it, so the grant failed with PACKAGE_CACHE_UNAVAILABLE. The session now waits, still asking, until the install the grant runs has recorded its outcome, bounded so an unrecorded ending never holds it for good.

Refs #656
# Conflicts:
#	docs/manifest.json
@mrgoonie

mrgoonie commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Review attestation: ready to merge at 9bde5b6b5d1eb892f870c60cba947af2548c3216, reviewed by agent:code-reviewer.

A push to this PR makes this attestation stale; the new head needs its own review.

@mrgoonie
mrgoonie enabled auto-merge (squash) October 8, 2026 06:49
@mrgoonie

mrgoonie commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

The review found nothing wrong with the code; the only blocker was a conflict in docs/manifest.json. I merged main (which now includes #665 and #668) and regenerated the manifest. On that tree, invariants pass 15/15, typecheck passes, and both focused specs pass 108/108. I attested head 9bde5b6 and enabled auto-merge. The non-blocking follow-ups (logging when the 60 s wait runs out, and the overlong doc lines) are folded into #670.

@mrgoonie
mrgoonie merged commit 90fe227 into main Oct 8, 2026
22 checks passed
@mrgoonie
mrgoonie deleted the test/656-widget-dev-flakes branch October 8, 2026 07:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: widget-dev-engine watched-folder grace spec flakes on macOS test: widget dev engine/session specs flake under parallel load

1 participant