fix(gateway): reap only the background processes an abandoned turn created

[JoaoMarcos44]

What does this PR do?

Fixes #76115. When a gateway turn spawns a background subprocess (background=true, e.g. next build) and that turn is later abandoned — inactivity timeout, /stop, /new, or a client disconnect — the subprocess was never reaped. It kept running inside the gateway's cgroup indefinitely. Enough memory growth pushes the cgroup past MemoryHigh, the kernel throttles/swap-thrashes the gateway process, its asyncio event loop starves, and every platform connection and cron job looks hung even though nothing upstream is actually broken.

The process registry already had a working tree-kill primitive. What was missing was per-turn ownership: nothing distinguished a process that predates the turn (must survive), a process the turn started and finished successfully (must survive), and a process an abandoned turn left running (must be reaped).

Root cause → fix

%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#00f0ff', 'mainBkg': '#0a0a16', 'primaryTextColor': '#ffffff', 'primaryBorderColor': '#ff007f', 'lineColor': '#00f0ff'}}}%%
flowchart TD
    A[🔒 Turn Starts] -->|Snapshot Baseline IDs| B[⚡ process_registry.snapshot_running_ids]
    B --> C{Agent Runs Tool}
    C -->|background=true| D[🚀 Subprocess Spawned]
    C -->|Turn Finishes Normally| E[✅ Ownership Markers Cleared]
    D --> F{Turn Abandoned?}
    F -->|Inactivity Timeout| G[🧵 Daemon Watchdog Thread]
    F -->|/stop, /new, Disconnect| H[⚔️ Explicit Interrupt Path]
    F -->|No — Completed| E
    G --> I[🔥 kill_started_since: Reap IDs Not in Baseline]
    H --> I
    I --> J[🩸 Prior + Legit Processes Preserved]
    E -.->|Stale Reference Guard| H

Loading

Infographic :

Timeline: everything before the Turn Baseline Snapshot is preserved; the selective reaper only terminates processes created after that snapshot by a turn that was abandoned (timeout / /stop / /new / disconnect). Matches snapshot_running_ids() → kill_started_since() in tools/process_registry.py.

• tools/process_registry.py: snapshot_running_ids(task_id) captures the IDs already running for a turn's task_id at start; kill_started_since(task_id, baseline, source=...) reaps only the IDs created after that baseline. Pure additive methods — no existing method touched.
• gateway/turn_context.py: TurnContext gains process_task_id + process_baseline so both cleanup paths can reach the same ownership data.
• gateway/run.py:
  • Baseline snapshotted right before the turn's executor task starts (_turn_task_id = session_id, matching the same task_id the terminal tool already threads through effective_task_id when it spawns a process — verified end to end, not assumed).
  • Inactivity-timeout path and the explicit /stop//new/disconnect interrupt path both reap through the same _reap_gateway_turn_processes helper, so the two call sites can't drift.
  • A daemon-thread watchdog (_watch_gateway_turn_inactivity) backs up the asyncio-based timeout poll — the failure mode this bug causes is exactly "event loop too starved to run its own timeout check," so the detector can't depend solely on that loop.
  • Race fix beyond the original patch: the turn's own worker clears its ownership markers (_gateway_turn_process_task_id / _gateway_turn_process_baseline) on the agent instance the instant run_sync() returns. Without this, .turn.agent stays reachable until the next turn is claimed, so a /stop landing right after a turn finished normally could still reap a background process that turn deliberately left running — violating the exact invariant this fix exists to protect.

What it does NOT change

No new core tool, no new HERMES_* env var, no prompt change, no cache-key change, no change to any pre-existing ProcessRegistry method. Processes that predate a turn, and processes left running by a turn that completed successfully, are untouched — verified by dedicated regression tests, not just claimed.

Related issues — checked, not assumed related

Issue/PR	State	Relation	Does it solve #76115?
#37454	open	ExecStopPost cgroup reaper	No — only fires when systemd actually restarts the unit; a throttled-but-alive gateway never triggers it
#68915	open	Orphaned-grandchild-holds-pipe registry bug	No — fixes exit detection, not killing a still-running process whose owning turn was abandoned
#71148 / #71506	open	Cron script timeout, same "leaked tree" family	Different lifecycle (cron job, not gateway turn)
#48339	open	Stale "running" status in the process tracker	Registry state bug, not ownership
#69033 / #69076	open	Windows job-object detachment	Platform-specific orphaning, not turn-abandonment
#62101 / #62112	open	Codex app-server leak per cron run	Cron lifecycle, different owner
#66671 / #66904	issue open, PR closed unmerged	AIAgent.close() never closes _codex_session	Session-close lifecycle, not turn-abandonment — and the fix PR itself never landed

None of the above ties process cleanup to gateway turn abandonment specifically. This PR is scoped to that one lifecycle.

Test plan

• tests/gateway/test_abandoned_turn_process_cleanup.py (new) — watchdog reaps only baseline-diff IDs; completed-worker wins the race; timeout cleanup is idempotent
• tests/tools/test_process_registry.py — snapshot_running_ids / kill_started_since contracts
• tests/gateway/test_tool_response_drop_recovery.py — existing interrupt/recovery behavior unaffected
• tests/gateway/test_turn_context.py — TurnContext field additions
• ruff check, python -m py_compile — clean
• Linux host with real cgroup limits (not verifiable in this Windows dev environment) — the 3 pre-existing test_process_registry.py failures on Windows (os.getpgid unavailable, PTY EOF semantics) are unrelated to this patch

[alt-glitch]

This was generated by AI during triage.

Related: #76172/#76183 use a global age-based sweep. This PR instead reaps only processes created by an abandoned gateway turn and explicitly preserves successful-turn background work. Both address #76115, but the lifecycle policies differ and need a maintainer choice.

[teknium1]

Thanks for preserving successful-turn background work rather than using a global sweep. The abandoned-turn cleanup gap is present on current main: gateway/run.py:22077-22080 and gateway/run.py:24159-24186 only interrupt the agent.

Problems

• The reaper is asynchronous but its ownership scope is session-wide. Gateway turns pass task_id=ctx.session_id (gateway/run.py:5096-5108), and the PR starts the reaper before _interrupt_and_clear_session releases the session. A replacement turn can create a process before the old reaper enumerates targets; that process is absent from the old baseline and is killed incorrectly.
• The stated client-disconnect coverage is incomplete: API-server SSE disconnect handlers call agent.interrupt() directly at gateway/platforms/api_server.py:4231-4238 and gateway/platforms/api_server.py:4810-4818, outside this diff.

Suggested changes

• Associate background processes with a unique logical-turn ownership token, not a session-wide task ID, and reap by that token.
• Cover the API-server disconnect path or narrow the claimed lifecycle scope.

This is an automated hermes-sweeper review.

[JoaoMarcos44]

Addressed both points from the hermes-sweeper review in dbbb10d:

1. Cross-turn race — reap is now gated on run_generation (existing _is_session_run_current mechanism, not a new ownership token): if a replacement turn has already claimed the session by the time a stale reaper thread actually runs, the reap is skipped instead of risking a kill on the new turn's freshly-spawned process. The new turn owns its own baseline, so nothing is left permanently unreaped.
2. API-server disconnect coverage — gateway/platforms/api_server.py's two SSE disconnect handlers (_run_agent() callers for chat-completions and the /api/sessions responses endpoint) now snapshot a baseline and reap on disconnect too, via a small shared _reap_disconnected_agent_processes() helper. That surface runs its own agent lifecycle outside TurnRunner, so it needed its own wiring rather than inheriting the gateway-chat path's.

Tests, ruff, and py_compile all still green; no regressions in the existing suite.

[JoaoMarcos44]

Added direct test coverage for the run_generation guard and API-server disconnect reap shipped in dbbb10d (a626d56) — same gap teknium flagged on the competing PR, closing it here too before it's asked for.

[kshitijk4poor]

Merged via #76687 — your three commits were cherry-picked with authorship preserved (rebase-merge), so they land on main under your name: 80e4fb5, a356917, 1b88682.

Excellent work on this one: the baseline-diff ownership model, the daemon watchdog rationale (the failure mode starves the exact event loop the normal timeout poll depends on), and the run_generation race fix in your second commit were all verified end-to-end with real subprocesses during review.

We added three follow-up commits on top: an epoch gate for the API-server disconnect reap (concurrent runs can share a client session_id, so a stale reaper needed the same gate you added on the gateway path), coverage for the /v1/runs sibling surface, an empty-task_id guard, and a dedup of kill_started_since into kill_all.

Closing this PR in favor of the merged salvage. Thanks for the contribution!