fix(gateway): prevent stuck sessions with agent timeout and staleness eviction

[kshitijk4poor]

Summary

Fixes sessions getting permanently locked when an agent hangs, requiring a gateway restart. Also fixes hung cron jobs blocking all subsequent cron execution.

Problem

1. Stuck message sessions: run_in_executor(None, run_sync) in _run_agent had no timeout. A hung API call (httpx timeout is 30 minutes) or runaway tool locked the session indefinitely — the finally block that cleans up _running_agents never ran. All subsequent messages for that session were routed to the interrupt/queue path and never processed.

2. Stuck cron jobs: agent.run_conversation(prompt) is synchronous and blocked the cron ticker thread. One hung cron job prevented all subsequent cron jobs from firing until gateway restart.

Fix

• Agent timeout (gateway/run.py): Wrap run_in_executor with asyncio.wait_for(timeout=HERMES_AGENT_TIMEOUT) (default 10min). On timeout, the agent is interrupted and the user gets an actionable error message. The finally block runs normally and unlocks the session.

• Staleness eviction (gateway/run.py): _running_agents_ts tracks start timestamps. When a new message arrives and finds an entry older than timeout + 1min grace, it's auto-evicted. Safety net for any cleanup path that fails.

• Cron timeout (cron/scheduler.py): Run run_conversation in a 1-worker ThreadPoolExecutor with future.result(timeout=HERMES_CRON_TIMEOUT) (default 10min). On timeout, the agent is interrupted and the pool is shut down with wait=False, cancel_futures=True so the ticker thread isn't blocked.

New env vars

Variable	Default	Purpose
HERMES_AGENT_TIMEOUT	600 (10min)	Max agent execution time per message
HERMES_CRON_TIMEOUT	600 (10min)	Max cron job execution time

Files changed

File	Changes
gateway/run.py	asyncio.wait_for timeout, _running_agents_ts staleness tracking
cron/scheduler.py	ThreadPoolExecutor with timeout and non-blocking shutdown

Test plan

• pytest tests/gateway/ tests/cron/ — 1932 passed, 6 failed (all pre-existing: approval flaky + whatsapp + progress emoji)
• Let agent run a tool that hangs >10min → should timeout, user gets error, session unlocks
• Schedule a cron job with a prompt that causes a hang → should timeout after 10min, ticker continues
• Normal messages and cron jobs under 10min → no change in behavior

[teknium1]

Superseded by #4727 which includes this fix alongside the full reliability cluster from #4577. Credit to @kshitijk4poor.