fix(compression): terminate idle workers instead of dropping them

[pacocartones]

Summary

• CompressionWorkerPool.finish() (open-sse/services/compression/compressionWorkerPool.ts:188) evicted an idle worker with this.remove(slot, false), and remove() (:198-203) only called worker.terminate() if (terminate). The idle path therefore deleted the slot from the pool set but never terminated the thread; compressionWorker.ts:13 keeps a parentPort.on("message") listener, so the worker's event loop and V8 isolate stayed alive forever, and dispatch() (:156) spawned a replacement because workers.size < size. The OMNI_COMPRESSION_WORKERS cap held in bookkeeping only: every idle cycle leaked one live thread. This is the mechanism confirmed on fix(backend): memory spike causing OOM (~1.5GB → ~16GB) when routing requests via combo #11804 (request-driven thread growth, ~90 threads parked in ep_poll, RSS far outside the main heap) and the one-line patch the reporter measured on a live instance: 57 leaked workers / 72 threads / 5.77 GB climbing before, 0 leaked / 15 threads / 846 MB flat after.
• remove() now always terminates the thread. The terminate parameter is removed rather than flipped at the idle call site: close() and fail() already passed true, so the parameter only existed to express the buggy case. A short comment on remove() records why dropping the slot alone is not enough. close(), fail(), timeouts and the queue/dispatch logic are otherwise unchanged; terminate() failures are still swallowed as before.
• Out of scope, on purpose: the callLogArtifactWorker hypothesis raised mid-thread (later marked unconfirmed by its author), the six bundled copies of the pool in the Next.js server chunks (they pick this change up at build time), and any change to the idle timeout or pool size defaults.

Related Issues

• Closes fix(backend): memory spike causing OOM (~1.5GB → ~16GB) when routing requests via combo #11804
• Related to fix(combo): always clear the loop-safety timer, not just on the happy path (#11804) #12245 (the combo loop-safety timer leak reported on the same issue, already merged)

Validation

• Change type: other (compression worker pool, open-sse/)
• Focused tests and category gates from the golden path: tests/unit/compression/compression-worker.test.ts + tests/unit/compression/compression-worker-file-resolution.test.ts 17/17 (4 suites), node scripts/check/check-complexity-ratchets.mjs --base-ref origin/release/v3.8.51 OK (0 violations, base 0), npm run check:changelog-integrity OK, npm run typecheck:core exit 0
• npm run lint — eslint run on the two touched source files only (--suppressions-location config/quality/eslint-suppressions.json), exit 0; the repository-wide command was not run
• Reconciled with the current active release base release/v3.8.51 (1a0375fba); focused checks rerun afterward
• Production-code changes include a new or updated automated test in this PR
• SonarQube is temporarily opt-in while the private project has no quota; it is not a PR gate.

Red/green: the new case fails on the base with idle eviction must terminate the worker thread: 0 !== 1 and passes with the change (three consecutive runs, 8/8 in the file).

Tests Added Or Updated

• tests/unit/compression/compression-worker.test.ts (1 new case, "terminates an idle worker instead of only dropping it from the pool"): spies on Worker.prototype.terminate and Worker.prototype.postMessage around a local { size: 1, idleMs: 50 } pool, runs one job, waits past the idle window, and asserts that exactly one worker was spawned, that terminate() was called once, and that the process does not retain the worker's MessagePort (process.getActiveResourcesInfo()). The finally block restores both prototypes, closes the pool, and terminates any worker the pool forgot, so a future regression fails the assertion instead of keeping the test runner alive. The seven existing cases are untouched.
• No stryker.conf.json change: open-sse/services/compression/ is not in the mutate list.

Coverage Notes

• open-sse/services/compression/compressionWorkerPool.ts: the idle path of finish() and the unconditional terminate() in remove() are exercised by the new case; close() by the new case's finally and the existing timeout case; fail() by the existing "fails open" case. No touched file lost coverage.

Reviewer Notes

• Behavioural change is limited to idle eviction: the thread is now terminated at the moment it is removed from the set, which is what close() and fail() already did. Nothing about job routing changes, so a request arriving after eviction still gets a freshly spawned worker exactly as before.
• Anyone patching an installed 3.8.50/3.8.51 bundle by hand should note that the minified idle call site appears in five files (four Next.js runtime chunks plus open-sse/mcp-server/server.js), as documented on the issue; a normal build from this branch covers all of them.
• No migrations, feature flags, or env changes.

[JasonBroderick]

Field confirmation from a self-hosted Docker deployment, and a recommendation: this is the PR to merge of the three open for #11804 / #12812.

Why this one

• The runtime change is identical to fix(compression): terminate idle workers on pool eviction #12813 (the idle-path line), so the fix is the same one that has now been measured on two independent production instances.
• It also removes the terminate parameter from remove(). All three callers already wanted termination, so nothing is lost, and no future call site can reintroduce the leak by passing false. That makes refactor(compression): drop the terminate flag from pool removal #12815's cleanup unnecessary once this merges.
• Its regression test counts real Worker.prototype.terminate calls and checks the active MessagePort count does not grow, then reaps leftovers so a regression fails instead of hanging the runner. That is the right assertion for this bug.

Evidence, v3.8.51 source (c3945a72) + the idle-path change, agentic workload through the default RTK/Caveman pipeline, no combos

• Server-process OS threads: 12 to 13, flat, for 3.5 days (daily maxima 13 / 12 / 13 / 13). Before the change they climbed one per idle eviction.
• Built bundle carries the terminating path (remove(e,!0),this.idleMs).
• What it does not fix, so it is not mis-attributed to this PR: on this instance the V8 heap (not native) still grows to the admission guard's ceiling in about 15 hours with threads flat. That is a separate retainer; heap snapshots are being collected and it will get its own issue.

I have posted the same measurements on #12812. Merging this one closes the worker leak for everyone and lets #12813 and #12815 be closed as superseded.

[diegosouzapw]

Closing as subsumed, but your test case was carried forward — details below.

#13091 landed earlier today with the same diagnosis and the same fix, reached independently: remove(slot, false) dropped the slot without terminate(), so the OS thread and its heap outlived the pool. The parameter is gone rather than defaulted, for the reason you gave — a pooled worker has no owner besides the pool, so there was never a caller that legitimately wanted the slot dropped and the thread kept. On the current tip:

$ git show origin/release/v3.8.51:open-sse/services/compression/compressionWorkerPool.ts | sed -n '201,208p'
  /** Drop a slot and release its OS thread. Removal always terminates: a pooled worker
   *  has no other owner, so skipping terminate() strands the thread permanently. */
  private async remove(slot: PoolWorker): Promise<void> {
    ...
    await slot.worker.terminate().catch(() => undefined);

Your test was not redundant, though. #13091 guards the behaviour by asserting the worker's exit event fires; yours counts actual terminate() calls and checks process.getActiveResourcesInfo() does not retain the MessagePort. That is the resource-level angle the other one does not cover, so it is now on the release branch via #13371, with you as Co-authored-by. Verified both directions there: 8/8 with the fix, and your case alone fails when remove() is made to skip terminate().

Thanks for it — and for citing #11804, which is the same class.