fix: reap stale background processes to prevent gateway starvation (#76115)

[JonthanaHanh]

Summary

Add a background sweep thread to ProcessRegistry that kills running background processes exceeding MAX_ACTIVE_PROCESS_AGE (default 24h).

Problem

When an agent turn spawns a background tool subprocess (e.g. pnpm build) and that turn gets stuck or abandoned, the subprocess is never reaped and has no lifetime cap. It keeps running indefinitely. A single runaway build can push the gateway cgroup past its MemoryHigh limit, starving the asyncio event loop — causing every messaging platform to disconnect and every cron to time out (see #76115 for full analysis).

Root Cause

MAX_ACTIVE_PROCESS_AGE (86400s / 24h) is defined in process_registry.py and used by has_active_for_session() to decide whether a process blocks session reset, but the process itself is never killed — only ignored by the reset guard.

Changes

What	Where
STALE_SWEEP_INTERVAL constant (300s)	tools/process_registry.py:63
_stale_sweep_loop() daemon thread	Started in ProcessRegistry.__init__()
_reap_stale_running() method	Uses existing kill_process() for process-tree teardown

• The sweep thread runs every 5 minutes and kills any running background process older than MAX_ACTIVE_PROCESS_AGE
• Uses existing kill_process() infrastructure which handles tree teardown (SIGKILL/SIGTERM on POSIX, taskkill /T /F on Windows) and moves the session to the finished dict
• Logs a warning for each reaped process with age and command

Test Plan

• Syntax check passes
• 33/34 existing tests pass (1 pre-existing failure due to missing psutil)
• Verify sweep thread starts on gateway launch
• Verify stale process is killed after MAX_ACTIVE_PROCESS_AGE

Fixes #76115

[teknium1]

Thanks for addressing a real process-lifecycle gap. The sweep needs policy and lifecycle changes before it can safely land.

Problems

• tools/process_registry.py:62 currently supplies the reset guard's default only; gateway/config.py:506-511 and website/docs/user-guide/messaging/index.md:274-281 explicitly permit 0 or larger values for legitimate long-lived processes. The new hard 24-hour sweep ignores that contract and would kill those processes anyway.
• #76115 reports gateway starvation after an approximately 11-hour build, while this diff waits 24 hours before reaping.
• The added ProcessRegistry.__init__() worker has no stop path. _stale_sweep_loop() only exits when its event is set, but the diff never sets it; tests/tools/test_process_registry.py:23-26 constructs a registry per test.
• No regression tests cover reaping, policy overrides, or worker lifecycle.

Suggested changes

• Define a separate, documented lifecycle policy (or wire the existing configured policy deliberately), make it cover the abandoned-turn failure path, and add a shutdown-safe execution mechanism plus focused regression tests.

Automated hermes-sweeper review.

[teknium1]

This starts one daemon thread per ProcessRegistry instance, but _stale_sweep_loop() only exits when _stale_sweep_stop is set and this diff adds no shutdown path. The per-test ProcessRegistry fixture in tests/tools/test_process_registry.py:23-26 would accumulate retained workers; please use an existing gateway lifecycle scheduler or add explicit stop/join ownership.

[kshitijk4poor]

Closing as duplicate of #76188 (@JoaoMarcos44) — that PR implements turn-scoped reaping (only kills processes the abandoned turn created) with 4 test files and gateway integration, vs the age-based blind sweep here which would kill legitimate long-running background processes older than 24h. You were first to submit (by ~18 min) but #76183 was an identical submission. Thanks for the contribution!