feat(agent): unified deadline layer — bounded execution primitive + timeout resolver (#85125 Phase 1)

[kshitijk4poor]

Summary

Tracking issue: #85125 (Phase 1 of 4). Draft until the plan on the tracking issue is signed off.

Phase 1 of #85125: one shared deadline layer (agent/deadline.py) so the timeout/hang bug classes get fixed once at a primitive instead of once per incident — plus the first call-site migration proving the pattern.

Changes

• agent/deadline.py (new):
  • run_bounded_async — wall-clock deadline for awaitables driven by a daemon threading.Timer, so a blocked event loop cannot disable the deadline (family A of the Triage: the seven mechanisms behind the 77 open stall/hang reports #84047 stall triage; generalizes the proven telegram-adapter _await_with_thread_deadline primitive, including the blocked-loop stack-dump watchdog and abandonment of cancellation-shielded tasks)
  • run_bounded_sync — same contract for sync callables (daemon worker, abandoned on expiry, on_timeout hook for marking backends suspect)
  • clamp_timeout — platform-safe clamping; kills the OverflowError: timestamp out of range for platform time_t class ([Bug] Large approvals.timeout crashes all parallel tool calls on macOS (OverflowError: timestamp out of range for platform time_t) #83220) at the shared boundary
  • resolve_timeout — one resolution path: timeouts: section in config.yaml > legacy HERMES_* env bridge > default. No new user-facing env vars (".env is for secrets only")
  • kill_process_tree — portable whole-tree termination (POSIX killpg for session leaders / Windows taskkill /F /T) for the orphaned-process-tree class (Cron script timeout leaves orphaned process trees (timeout kill doesn't reach descendants) #71148, [Bug]: Cron script timeouts leave orphaned background processes and get reported as provider timeouts #59549, [Bug]: Docker terminal timeout leaves in-container process trees running #84967, browser_tool: command timeout leaks the spawned agent-browser daemon + Chromium tree (long-lived gateway accumulates orphans until host OOM) #68139)
  • DeadlineExpired — our deadline, mechanically distinct from provider/transport timeouts (the [Bug]: Cron script timeouts leave orphaned background processes and get reported as provider timeouts #59549/Cron no_agent script failure reported as 'provider timeout / fallback chain exhausted' — misleading error message #80323 misattribution class)
• agent/tool_executor.py — _resolve_concurrent_tool_timeout() migrates onto resolve_timeout("tools.concurrent_batch"); exact legacy contract preserved (default 420s, HERMES_CONCURRENT_TOOL_TIMEOUT_S bridge, 0 disables)
• hermes_cli/config.py — timeouts registered as a known config root
• cli-config.yaml.example — documents the new section
• tests/agent/test_deadline.py — 40 tests: clamp normalization (including a real time_t regression proof for the [Bug] Large approvals.timeout crashes all parallel tool calls on macOS (OverflowError: timestamp out of range for platform time_t) #83220 class), resolver precedence + bool/NaN rejection, sync/async bounding (cancellation-shielded hung task, external-cancellation cleanup), grandchild and own-session-descendant process-tree kills, and the tool_executor back-compat contract

Validation

Check	Result
tests/agent/test_deadline.py	40 passed
tests/run_agent/ executor-adjacent (-k 'concurrent or batch or timeout or executor')	100 passed, 1 skipped
tests/run_agent/test_start_order_gate.py + tests/hermes_cli/test_config.py	77 passed
E2E (isolated HERMES_HOME, real config.yaml)	config-wins / env-bridge / oversized-clamp / shielded-task bound / grandchild + own-session descendant killed — all pass
Clean-import probe (model_tools + agent.tool_executor with worktree on sys.path)	imports ok
ruff on changed files	clean

Self-review (3-angle + Hermes-specific) applied before push: caller-cancellation task leak fixed, taskkill exit-code contract, psutil descendant sweep for own-session grandchildren, bool/NaN config rejection, kw_only BoundedResult. Known deliberate deferrals, tracked in #85125: telegram's private _await_with_thread_deadline copy migrates in Phase 2; gateway/status.py and tools/code_execution_tool.py tree-kill sites migrate in Phase 4 (both named in the module docstring).

Pure addition otherwise: no behavior change under default config, no cache impact, no message-flow changes. Later phases of #85125 migrate the sequential tool path, MCP handlers, and subprocess kill sites onto these primitives (aligning in-flight PRs #84795 / #84125 / #76822).

Part of #85125.

Summary by CodeRabbit

• New Features

  • Added unified timeout controls for internal operations, configurable through the application configuration.
  • Added support for disabling time limits and safely clamping excessively large values.
  • Improved timeout handling for synchronous and asynchronous operations, including cleanup and clearer timeout reporting.
  • Added process-tree termination to better stop timed-out operations.

• Documentation

  • Documented timeout configuration and precedence in the example configuration file.

• Bug Fixes

  • Preserved compatibility with the existing environment-variable timeout setting.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 2e8e53ac-13f1-436b-9a8a-2e3f86e09732

📥 Commits

Reviewing files that changed from the base of the PR and between 90c7180 and 8b38796.

📒 Files selected for processing (5)

• agent/deadline.py
• agent/tool_executor.py
• cli-config.yaml.example
• hermes_cli/config.py
• tests/agent/test_deadline.py

---

📝 Walkthrough

Walkthrough

Added a shared deadline module for timeout resolution, bounded async and sync execution, diagnostics, and process-tree termination. Registered timeout configuration, documented precedence rules, migrated concurrent tool timeout handling, and added comprehensive tests.

Changes

Unified deadline infrastructure

Layer / File(s)	Summary
Deadline contracts and configuration agent/deadline.py, hermes_cli/config.py, cli-config.yaml.example, tests/agent/test_deadline.py	Added timeout clamping, configuration/environment/default precedence, DeadlineExpired, BoundedResult, and timeout configuration documentation. Tests cover normalization, precedence, invalid values, and configuration failures.
Bounded asynchronous and synchronous execution agent/deadline.py, tests/agent/test_deadline.py	Added daemon-timer async execution, task abandonment, blocked-loop diagnostics, cleanup callbacks, daemon-thread sync execution, timeout callbacks, and exception preservation. Tests cover completion, timeout, cancellation, cleanup, and callback behavior.
Portable process-tree termination agent/deadline.py, tests/agent/test_deadline.py	Added Windows taskkill handling and POSIX descendant, process-group, and fallback signaling. Tests cover process-tree termination scenarios.
Concurrent tool timeout integration agent/tool_executor.py, tests/agent/test_deadline.py	Migrated concurrent tool timeout resolution to resolve_timeout with configuration precedence, legacy environment compatibility, disabled deadlines, defaults, and invalid-value handling.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant run_bounded_async
  participant DaemonTimer
  participant asyncio.Task
  Caller->>run_bounded_async: Start awaitable with timeout
  run_bounded_async->>DaemonTimer: Schedule independent deadline
  DaemonTimer->>run_bounded_async: Signal timeout
  run_bounded_async->>asyncio.Task: Cancel or abandon task
  run_bounded_async-->>Caller: Return BoundedResult

Loading

Suggested reviewers: teknium1

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}