fix(v1,serve): discard delivered intercepts; trim worker arenas at heartbeat cadence

[hubert-marek]

Completes the env-worker memory series: #1608 merged the first two fixes (step slimming + registry eviction) just before the commit carrying these last two was pushed to its branch — this PR is that stranded delta, cherry-picked onto the current base.

3. Discard delivered intercepts (dominant term: −74% worker peak)

InterceptionServer.intercepts retained every request's raw body — the full message history including in-sandbox base64 screenshots — until rollout unregister, so a long browser rollout held every turn's request simultaneously. (The renderer's image offload rewrites a normalized copy; the intercept's base64 never left.) forward_request now discards the intercept after delivery via a new Endpoint.discard_request: the HTTP handler keeps its own local reference so delivery is unaffected, discard is idempotent, and rollout-unregister still sweeps undelivered entries.

4. malloc_trim(0) at the worker stats cadence (resting RSS ÷3)

Env workers had no trim anywhere (the per-batch trim added on prime-rl's side is orchestrator-only); freed arena pages ratcheted worker RSS to ~3× the live set. Trim every 10s in the existing stats loop via ctypes (releases the GIL — never stalls the loop). Deliberately not gc.collect: full collections on fat heaps are what caused the worker heartbeat-timeout kills.

Measurement (churn simulation: 4 concurrent 45-turn browser rollouts/worker, real subprocess RSS)

scenario	worker peak	worker resting
#1608 fixes only	1053 MB	943–988 MB
+ intercept discard	275 MB	179–218 MB
+ worker trim (both)	275 MB	119–155 MB

Production: without these two, the worldsims perplexity env pod climbed to ~90GB and failed even with #1608's fixes; with all four (worldsims pin e522d36e), run oh7jsedjxix7rbk23sdvsujd is training now. py_compile + ruff clean; discard behavior unit-checked (local ref intact post-discard, idempotent).

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches request-forwarding lifecycle and periodic native memory calls in serve workers; behavior is localized but affects long-running rollout memory and timing under load.

Overview
Env-worker memory: The stats heartbeat loop now calls malloc_trim(0) on Linux (via ctypes + libc.so.6) so freed allocator pages are returned to the OS on the same ~10s cadence as worker stats. The change deliberately avoids gc.collect() in that loop to prevent heartbeat stalls on large heaps.

Intercept retention: Endpoint.discard_request removes each delivered intercept from InterceptionServer.intercepts after streaming or non-streaming delivery in forward_request. Raw request bodies (including full multimodal histories) no longer accumulate for every turn until rollout unregister; the handler’s local request reference still covers delivery, and rollout unregister continues to sweep anything undelivered.

^{Reviewed by Cursor Bugbot for commit 868124c. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Discard delivered intercepts and trim worker heap memory at heartbeat cadence

• Adds Endpoint.discard_request in endpoint_utils.py to delete completed intercept entries from server.intercepts, reducing retained memory after request delivery.
• Wraps the response delivery block in forward_request with try/finally so discard_request is always called, including on error paths.
• Adds a malloc_trim(0) call via ctypes in EnvWorker.stats_loop to return freed heap pages to the OS on each stats interval when libc is available.

^{Macroscope summarized 868124c.}

[macroscopeapp]

Approvability

Verdict: Approved

Memory optimization fix that cleans up intercept data after delivery and trims malloc arenas at heartbeat cadence. Both changes are self-contained, well-documented, and don't alter functional behavior - they only release memory earlier to prevent RSS growth during long rollouts.

^{You can customize Macroscope's approvability policy. Learn more.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 868124c013

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Keep failed streaming intercepts until cleanup can unblock them

When an intercepted request has stream: true for any non-chat protocol, this block raises before synthesize_stream() can put an EOF sentinel or resolve the response_future, but the new unconditional discard removes the only entry that InterceptionServer.unregister_rollout() uses to cancel that future and signal the chunk queue. In that failure path (and any synthesize_stream() error before delivery), the aiohttp handler is left waiting/keepalive-looping instead of being unblocked during rollout cleanup; only discard after successful delivery or explicitly unblock the intercept before popping it.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Move malloc_trim off the event loop

Every EnvWorker already starts the configure_runtime_native_threads() daemon trim loop from _cap_native_threads() during construction, so this adds a second trim path that runs synchronously on the asyncio event-loop thread. Releasing the GIL does not let this same event loop continue processing requests or heartbeats while malloc_trim(0) is executing (or waiting on allocator locks held by the daemon trim), so on the large fragmented heaps this targets it can introduce exactly the worker lag/heartbeat delays the stats loop is meant to report; rely on the existing background trim or offload this call instead.

Useful? React with 👍 / 👎.