Skip to content

fix(delegate_task): add hard-timeout protection for subagent execution - #11591

Closed
ronatube wants to merge 1 commit into
NousResearch:mainfrom
ronatube:fix/delegate-task-hard-timeout
Closed

fix(delegate_task): add hard-timeout protection for subagent execution#11591
ronatube wants to merge 1 commit into
NousResearch:mainfrom
ronatube:fix/delegate-task-hard-timeout

Conversation

@ronatube

Copy link
Copy Markdown

Summary

The single-task path in delegate_task() had no timeout, no interrupt propagation, and no exception handling — a bare call to _run_single_child() that could block the parent agent indefinitely. The batch path had interrupt-flag checking but did not actually call child.interrupt() on pending children.

Changes

tools/delegate_tool.py

1. Single-task path

Wrapped the bare _run_single_child() call in a threading.Thread with:

  • Hard timeout (default 60 minutes, configurable via HERMES_DELEGATE_TASK_TIMEOUT)
  • child.interrupt() call on timeout + 10s grace period before force-unblocking
  • Exception catching to prevent a crashing subagent from swallowing its error
  • Returns {"status": "timeout", ...} or {"status": "error", ...} instead of hanging

2. Batch path interrupt propagation

  • Added child.interrupt() calls for all pending children when parent is interrupted
  • Added 1-second grace period for interrupt signals to propagate before collecting results
  • Added overall batch hard-timeout (default 120 minutes, configurable via HERMES_DELEGATE_BATCH_TIMEOUT)

3. Cleanup - Removed unused as_completed import

Root Cause

All observed hangs (450+ minutes) were delegate_task calls where:

  • The subagent ran tool=terminal multiple times in a loop
  • Each terminal subprocess completed quickly but the subagent LLM API calls were very slow (just under the 300s stale timeout threshold)
  • The ThreadPoolExecutor thread was blocked on httpx — a blocking I/O call that cannot be interrupted by Python-level signals
  • The parent agent /stop command set _interrupt_requested but could not break through

Configurability

Env Var Default Description
HERMES_DELEGATE_TASK_TIMEOUT 3600 (60 min) Max runtime per subagent (single-task)
HERMES_DELEGATE_BATCH_TIMEOUT 7200 (120 min) Max runtime for entire batch

Backwards Compatibility

  • Normal (non-timeout) completions return identical result structure
  • Only adds new status values: "timeout" and "error"
  • No changes to tool definitions, API contracts, or dependent code

Files Changed: tools/delegate_tool.py (+118/-15)

Testing: Syntax verified, deployed to production, Gateway restarted successfully.

- Wrap single-task path in Thread with configurable timeout (default 60min)
- Propagate interrupt signal to all pending children on parent interrupt
- Add overall batch hard-timeout (default 120min)
- Return {"status": "timeout"} or {"status": "error"} instead of hanging

Root cause: bare _run_single_child() call had no timeout, no interrupt
propagation, and no exception handling — could block parent indefinitely.
httpx blocking I/O cannot be interrupted by Python-level signals, so a
Thread wrapper with join(timeout) is used instead.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/tools Tool registry, model_tools, toolsets tool/delegate Subagent delegation labels Apr 24, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Likely duplicate of #13770 (merged) — that PR already added hard timeout and stale detection for subagent execution, fixing #13768. This PR addresses the same concern; please check if the merged fix is sufficient.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for this contribution, @ronatube! The protections you've described here were already shipped in PR #13770 (merged 2026-04-22), which landed in v2026.4.23.

This is an automated hermes-sweeper review.

What's already in main / v2026.4.23 (commit dd8ab4055):

  • _get_child_timeout() — reads delegation.child_timeout_seconds (config) or DELEGATION_CHILD_TIMEOUT_SECONDS (env var); default 300s, later bumped to 600s in feat(delegation): bump default child_timeout_seconds to 600s #14809 (tools/delegate_tool.py:363)
  • future.result(timeout=child_timeout) — hard timeout on the single-task path (tools/delegate_tool.py:1428)
  • exit_reason: "timeout" — new status value returned on timeout (tools/delegate_tool.py:1514)
  • _timeout_executor.shutdown(wait=False) — avoids ThreadPoolExecutor.__exit__ deadlock on stuck blocking I/O (tools/delegate_tool.py:1523)
  • Heartbeat stale detection via api_call_count advancement tracking (tools/delegate_tool.py:1277)

Closing as implemented. If you believe there are gaps not covered by #13770, please reopen with a specific reproduction case against v2026.4.23.

@teknium1 teknium1 closed this Apr 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists tool/delegate Subagent delegation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants