test(system-one): verify resident PID teardown and preserve Windows evaluator subprocess

[dyphn1]

Summary

Fixes #284's test-harness teardown race while retaining the RS13 child-process boundary.

• Capture the Python resident PID before shutdown; on Windows, wait until both state-file unlink and PID termination before deleting the fixture. Keep bounded recursive-remove retries for handle-release latency.
• Add RS15 repeated start/stop teardown regression with a test-only Python stub that deliberately remains alive for 500 ms after state-file removal, proving why unlink alone is insufficient.
• Restore RS13's actual evaluator CLI subprocess, rather than substituting in-process evaluator.run().
• Add an explicit, tested Windows Node 23.x and 24.0–24.19 compatibility policy for RS13. Node 23's Windows shutdown assertion is reported in libuv assertion on Windows with Node.js 23.x nodejs/node#56645, and the Node 24 upstream fix (src: fix libuv assertion on windows nodejs/node#61999) shipped in v24.20.0. The original v24.15.0 crash is a high-confidence upstream match, not a confirmed native stack match.
• Add exact-version Windows CI coverage: Node v24.20.0 runs the full resident suite; Node v23.6.0 and v24.15.0 independently confirm the guard and visible RS13 skip. Node 22 runtime-floor and macOS/Linux/Windows remain enabled. This fixes the residual Node 23 gap without narrowing engines.node >=22.
• Document the Windows subprocess verification runtime requirement in README.
• Rebased onto current main (release 0.30.8).

Validation

Latest CI: https://github.com/dyphn1/Harness-everything/actions/runs/37722476831 — run #804 completed SUCCESS, 15/15 jobs green on head 75ef983.

• Windows Node 23.6.0: version-policy test passes; RS13 visibly skipped for libuv assertion on Windows with Node.js 23.x nodejs/node#56645.
• Windows Node 24.15.0: version-policy test passes; RS13 visibly skipped for src: fix libuv assertion on windows nodejs/node#61999.
• Windows Node 24.20.0: complete resident suite passes, including actual evaluator subprocess (RS13) and repeated PID teardown (RS15), without skip.
• Node 22 runtime-floor; Windows/macOS/Ubuntu self-regression and mutation checks; consistency; installer E2E; release semantics; skill quality: all pass.

Node 23.x is covered by explicit Windows-only guard without changing package-wide >=22 engines. The original crash's precise native stack frame is not confirmed.

Resolves #284 when the complete matrix is green.

[dyphn1]

Verification

CI run #801 completed 12/12 successful on commit a55a449.

Relevant evidence for #284:

• self-regression (windows-latest): PASS
• consistency (windows-latest): PASS
• installer-e2e (windows-latest): PASS
• Node 22 runtime-floor: PASS
• Ubuntu/macOS self-regression: PASS

The fix does not skip Windows tests and keeps RS12 as the evaluator CLI/process-boundary coverage while RS13 directly exercises the acceptance/evaluator contract.

[dyphn1]

Strict re-review: I would not merge this yet.

P1 — RS13 currently avoids the crashing boundary instead of bounding/fixing the supported-runtime crash

#284 reproduces specifically on Windows + Node 24.15.0 with exit code 3221226505 / 0xC0000409, while CI already passed before this PR. There is now strong upstream evidence that Windows Node 24.15.0 has native/libuv failures with the same 0xC0000409 signature around network/process teardown:

• nodejs/node#63620 — Windows Node 24.15.0, 0xC0000409 stack-buffer-overrun in the TCP connect path; 24.16.0 appears clean.
• nodejs/node#56645 — Windows libuv shutdown assertion with exit code 3221226505.
• nodejs/node#64322 — Windows Node 24.x process-exit libuv assertion.

This repository declares Node >=22, so 24.15.0 is currently inside the supported engine range. Changing RS13 from a real evaluator child process to an in-process call can make the test green without proving that the original supported-runtime failure is gone. RS12 exercises one evaluator CLI invocation, but it does not reproduce RS13's repeated acceptance/evaluator scenario or prove the 24.15.0 crash boundary is safe.

Before closing #284, do one of:

1. reproduce the original RS13 path on Windows Node 24.15.0 and demonstrate the real fix, or
2. explicitly classify Windows Node 24.15.0 (or the confirmed affected range) as a known-bad runtime with a visible guard/version requirement, while keeping the subprocess coverage intact.

Do not use the current floating node-version: 24 CI result as proof: the issue already stated that CI passed, and that lane now resolves to a newer 24.x runtime.

P2 — cleanup waits for state-file removal, not resident-process termination

The new cleanup waits until the state file disappears and then retries rmSync. In the Python server, the state file is unlinked during teardown before the process has necessarily fully exited. Therefore the test can pass once filesystem handles are released even if the detached resident process is still alive briefly (or leaked).

For a Windows-stability fix, capture the resident PID before shutdown and verify the process is actually gone (Windows-specific PID-existence check is sufficient; keep the existing POSIX zombie caveat in mind), then remove the directory. Add a focused teardown regression/stress case that proves repeated start/stop leaves neither live resident PIDs nor undeletable fixture directories.

P2 — current CI does not distinguish old behavior from the fix

CI run #801 is green, but the original #284 explicitly says Windows CI was already green on the failing main revision. The PR needs evidence that would have failed before the change and passes after it: either an exact-version Windows 24.15.0 reproduction/guard test for RS13 and a deterministic/stress teardown test for the EPERM race.

Merge hygiene

The branch is also currently 1 commit behind main (0.30.8 release commit). Rebase/update after the blockers above are fixed and rerun the final matrix against current main.

#243 cleanup was separately re-audited and looks correctly closed: Phase 1–3 plus the parser regression are on main, and the remaining live behavioral evidence is still tracked by #253.

[dyphn1]

Refined review finding — previous upstream comparison corrected

After checking Node's actual fixes, the RS13 evidence is clearer.

Correction: nodejs/node#63620 is not the primary match; it is a separate TCP-connect stack-corruption issue.

The close match is nodejs/node#56645 / #64322, fixed by nodejs/node#61999. That patch prevents PostDelayedTask() from calling uv_async_send() after DelayedTaskScheduler::Stop() has begun shutdown. The failure signature reported upstream is the same 3221226505 / 0xC0000409 Windows crash. The v24 backport first ships in Node 24.20.0 (release PR nodejs/node#65461, commit 05ec9c4771).

The #284 machine was on Node 24.15.0, so it definitely predates that upstream fix. This makes the current RS13 change (subprocess -> in-process evaluator.run()) the wrong fix direction: it removes the process-shutdown boundary where the known Node defect occurs.

Requested changes

1. Restore RS13's real evaluator subprocess path.
2. Encode an explicit Windows runtime policy for Node 24 versions that predate 24.20.0 (guard/visible skip or documented minimum patch level); do not silently avoid the subprocess.
3. Keep the cleanup improvement, but strengthen it to verify the detached resident PID is actually gone before deleting the fixture.
4. Add repeated start/stop teardown coverage that can distinguish the cleanup fix from the old implementation.
5. Rebase onto current main, then rerun the final matrix with Windows Node 24.20+.

We cannot prove from the original #284 evidence alone that its native frame is exactly src/win/async.c:76, because stderr/dump was not retained. So the correct wording is high-confidence upstream match, not confirmed native root cause. A 24.15.0 reproduction with stderr/dump would close that last evidence gap.

[dyphn1]

Verification follow-up (head a55a449)

I rechecked the open blockers against the current PR head.

P1 — RS13 no longer exercises the reported child-process exit path. The test now calls evaluator.run() in-process. The repository supports Node >=22, which includes the reported Node 24.15.0 runtime, while CI selects floating node-version: 24. The Node upstream fix in nodejs/node#61999 is merged for the related Windows assertion issue, but #284 did not retain a dump/stderr proving the exact native frame. Please keep the evaluator subprocess coverage and either reproduce/fix the failure on the affected runtime or add an explicit, visible runtime policy for the confirmed affected range.

P2 — Cleanup waits for state-file removal, not process exit. The Python server unlinks its state file during teardown before returning; the test can proceed while the detached PID is still alive. Please capture the resident PID, verify it has exited before deleting the fixture, and add repeated start/stop teardown coverage.

Evidence and hygiene: PR CI is 12/12 green. On Windows with Node 24.18.0, the resident suite passes 16/16 on both this PR and fetched current main, so that local run does not distinguish the fix from baseline. The branch is also one commit behind main; please update it after addressing the blockers.

[dyphn1]

Resolution / final verification for #284

Both review blockers are addressed on PR head c0b63cc (rebased onto main 0.30.8).

• RS13 subprocess preserved: acceptance/evaluator test again launches the real Node CLI subprocess. The reported Windows v24.15.0 crash is a high-confidence upstream libuv shutdown match, not stack-proven; the v24.20.0 upstream fix is tracked at src: fix libuv assertion on windows nodejs/node#61999.
• Explicit Windows runtime policy: v24.0–v24.19 visibly skip RS13 with the upstream reason (no silent switch to evaluator.run()); Node 22 stays supported. README documents the requirement.
• PID-aware teardown: capture the Python resident PID before stop(), wait for actual PID disappearance on Windows, then delete with bounded rmSync retries. RS15 starts/stops six real resident stubs whose Python process deliberately lingers 500 ms after state unlink.
• Version-specific CI evidence: pinned Windows Node 24.20.0: 18/18 passed, 0 skipped including RS13 and RS15. Pinned Windows Node 24.15.0: policy passed; RS13 explicitly skipped.
• Final CI: run #803 completed SUCCESS (14/14 jobs), including Node 22 floor, Windows/macOS/Linux self-regression, mutation checks, consistency, installer E2E, release semantics and skill quality.

The exact native frame of the original v24.15.0 crash was not retained, so this closes the practical reproducibility/compatibility gap without claiming the original native stack was proven. PR is left open for normal merge/review.

[dyphn1]

Remaining runtime-policy scope

One residual compatibility question: upstream nodejs/node#56645 reports the same Windows libuv assertion on Node 23.x and lists reproductions on 23.0.0, 23.4.0, and 23.6.0. The new affectedWindowsNode24() policy only marks Node 24.0–24.19 as affected, so RS13 would still run on Windows Node 23.

The repo currently declares engines.node >=22 and describes support as Node 22+, which admits Node 23. If the intended support contract is maintained LTS lines only, please make that explicit and reconcile the engine/documentation policy; otherwise, include the affected Node 23 range in the visible compatibility policy and test it.

[dyphn1]

Node 23 runtime-policy gap — resolved in 75ef983

Follow-up to the remaining runtime-policy scope comment.

Finding confirmed: Node's #56645 describes the Windows shutdown assertion on 23.0.0/23.4.0/23.6.0, and this repo's engines.node >=22 includes Node 23. The previous policy covering only Node 24.0–24.19 was incomplete.

Minimal correction (no global engine constraint change):

• Extend the RS13 visible Windows-only guard to all Node 23.x, in addition to Node 24.0–24.19, with separate upstream issue text for each range.
• Retain real RS13 evaluator subprocess on unaffected versions (Node 22, Node 24.20+, and other platforms).
• Expand RS00 boundary cases (positive/negative cases for Node 22/23/24/25/26 and Linux/macOS).
• Run an exact-version Windows Node 23.6.0 policy CI lane alongside Node 24.15.0 and patched 24.20.0; document in README.

Final evidence: CI #804 finished SUCCESS: 15/15 jobs green on 75ef983. Windows Node 23.6.0 1 policy pass + 1 explicit RS13 skip (nodejs/node#56645); Windows Node 24.15.0 likewise passes with its explicit RS13 skip; Windows Node 24.20.0 runs the actual RS13 evaluator subprocess and RS15 PID cleanup with 18/18 tests passing, 0 skipped. Existing Node 22, Linux/macOS/Windows self-regression, mutations, consistency, installer, release and skill-quality gates are all green.

Scope: The original 24.15.0 crash did not retain a native dump. The upstream association remains high-confidence, not proven from an exact crash frame. PR remains open until merge/review.