fix(gateway): recover stale pid files and release evicted clients

[LeonSGP43]

Summary

• recover startup when a stale gateway.pid is still present after the runtime lock is acquired
• release provider clients when _evict_cached_agent() drops a cached agent
• add regression coverage for both recovery paths

Testing

• python3 -m pytest -o addopts='' -q tests/gateway/test_agent_cache.py -k 'evict' tests/gateway/test_runner_startup_failures.py -k 'stale_pid or start_gateway'
• python3 -m pytest -o addopts='' -q tests/tools/test_zombie_process_cleanup.py -k 'evict_does_not_call_close' tests/gateway/test_session_model_reset.py -k 'evict'

Closes #14598

[teknium1]

Thanks for this, @LeonSGP43 — both issues you identified were real, and the analysis in #14598 was solid. Closing because both halves have since been fixed independently on main, so there's nothing left to merge.

Eviction httpx leak — fixed by #29298 / #41974. _evict_cached_agent() now spawns _release_evicted_agent_soft() on a daemon thread, which calls agent.release_clients() exactly as you proposed. The main version additionally guards agents that are mid-turn (and the pending-agent sentinel), so cherry-picking the PR's version would actually regress that.

Stale PID file on startup — fixed by 402d048 ("also unlink stale PID + lock files on cleanup"), which landed 2026-04-22, the day before this PR was opened. That commit made _cleanup_invalid_pid_path() force-unlink the stale file, and it runs inside get_running_pid() at the top of start_gateway() — before write_pid_file() is ever called. So a crashed/SIGKILLed gateway's leftover gateway.pid is removed during the liveness check, and write_pid_file() no longer hits FileExistsError.

I verified this end-to-end against a temp HERMES_HOME on current main (no patch): with a stale PID file present — both with and without a leftover lock file — get_running_pid() returns None and unlinks the stale file, then write_pid_file() succeeds cleanly. The retry-once recovery path this PR adds is unreachable in practice.

Your fix was correct for the state of the tree when you filed it — the auto-cleanup commit just beat it by a few hours. Appreciate the careful write-up.