fix(delegate): declare stateless channel in one-shot and cron so delegate_task returns results - #63866
Conversation
…gate_task returns results
run_agent._dispatch_delegate_task forces background=True for every top-level
delegation, and async_delivery_supported() returns True for any session that
never binds the capability. On runners that cannot receive a completion after
their turn ends, that combination silently discards every subagent result: the
model gets a dispatch handle, ends its turn, and reports 'waiting for results'.
Two such runners never bind the capability:
* hermes -z (one-shot) prints one final response and exits. It bypasses cli.py,
so nothing drains process_registry.completion_queue (only the interactive
process_loop and the gateway watchers do).
* cron run_job clears the HERMES_SESSION_* routing keys, so a completion event
carries session_key="" — _enrich_async_delegation_routing cannot resolve it
and _inject_watch_notification drops it ("no routing metadata"). By then
run_job has already shipped the job's final response via _deliver_result;
there is no turn left to re-enter. Worse, get_current_session_key() can fall
back to the ambient os.environ HERMES_SESSION_KEY, so a cron subagent's output
can be routed into an unrelated user chat rather than merely dropped.
Add declare_stateless_channel() and bind it in both runners, routing
delegate_task to its existing inline/synchronous path — the same fallback the
stateless HTTP adapter already relies on, and the fix suggested in NousResearch#63142. The
helper binds only the capability: set_session_vars() would also latch
_session_context_engaged, which a pure single-process one-shot must not trigger.
Also correct two agent-facing strings that hardcoded 'stateless HTTP API' as the
only channel without async delivery (delegate_tool, terminal_tool); they now name
the actual condition.
Repro (before): hermes -z 'Use delegate_task to spawn a subagent that replies
BANANA. Report its reply.' -> "Waiting for the subagent's response...", exit 0,
no BANANA. After: BANANA is returned in-turn.
Fixes NousResearch#53027
Fixes NousResearch#63142
|
Thanks for tracing this to the delivery-capability gate. Current main forces top-level delegation into background mode ( Problems
Suggested changes
Automated hermes-sweeper review. |
|
Merged as #66617 (rebased so your commit keeps authorship). Yours was the earliest and the only complete fix in the cluster — it covers one-shot AND cron AND the terminal watcher path, and it correctly uses a dedicated |
…ainer) Bug: on the ACP surface, delegate_task silently loses every subagent result. The dispatch succeeds and the child runs to completion, but its summary never re-enters the conversation; the model tells the user to wait for a result that cannot arrive. Root cause: async delegation completions are pushed onto the process- global process_registry.completion_queue, and delivery is each surface's job — CLI drains between prompts, TUI/desktop run a notification poller, the gateway has drain loops. acp_adapter/ has NO consumer for that queue, and additionally binds its session context via set_session_vars() with the default async_delivery=True, so delegate_task's capability gate (async_delivery_supported()) wrongly permits background dispatch on a channel that can never deliver. The completion event rots in the in- memory queue until the hermes acp process exits. Fix: bind ACP sessions with async_delivery=False, the same contract the stateless API server declares (supports_async_delivery=False; cf. upstream PRs NousResearch#50319, NousResearch#63866 for the api-server and cron surfaces). delegate_task then takes its existing synchronous fallback: children run inside the tool call and their summaries return in the tool result, within the same turn. Behavior change: ACP delegations now block the prompt until the subagents finish (batch tasks still run in parallel; the call waits for all). Cancelling the turn interrupts attached children via the sync path's normal lifecycle. Upstream: tracked as issue NousResearch#62548; open PR NousResearch#62558 adds a real drainer but delivers results as display-only session/update chunks that never re-enter the model's context, so this local override remains preferable for agent-consumable delegation results even if that PR lands. Bug report: ~/config/hermes/bug_reports/acp-delegate-task-results-never-delivered.md Regression test: tests/acp/test_server.py::TestPrompt:: test_prompt_binds_async_delivery_unsupported captures async_delivery_supported() from inside run_conversation (the exact context where delegate_task reads the flag); verified to FAIL against the pre-fix binding and PASS with the fix.
What does this PR do?
run_agent._dispatch_delegate_taskforcesbackground=Truefor every top-level delegation, andasync_delivery_supported()returnsTruefor any session that never binds the capability. On a runner that cannot receive a completion after its turn ends, that combination silently discards every subagent result — the model gets a dispatch handle, ends its turn, and reports "waiting for results".Two runners never bind the capability:
hermes -z(one-shot) prints one final response and exits. It bypassescli.py, so nothing drainsprocess_registry.completion_queue(only the interactiveprocess_loopand the gateway watchers do).run_jobdeliberately clears theHERMES_SESSION_*routing keys, so a completion event carriessession_key=""—_enrich_async_delegation_routingcannot resolve it and_inject_watch_notificationdrops it ("no routing metadata"). By thenrun_jobhas already shipped the job's final response via_deliver_result; there is no turn left to re-enter. Worse,get_current_session_key()can fall back to the ambientos.environ["HERMES_SESSION_KEY"], so a cron subagent's output can be routed into an unrelated user chat rather than merely dropped.This adds
declare_stateless_channel()and binds it in both runners, routingdelegate_taskto its existing inline/synchronous path — the same fallback the stateless HTTP adapter already relies on, and the fix suggested in #63142.This is not new behavior. It restores the documented default. From
website/docs/user-guide/features/delegation.md(added in af250d8):For one-shot the process does not remain alive, and for cron there is no resolvable owning session — so
background=trueis a promise neither channel can keep.Note: this reproduces on current
main(af250d8), after the durable-completion work in 67f4e1b / d0e9a42. Those commits correctly store the completion; nothing on these two paths ever consumes it.Related Issue
Fixes #53027
Fixes #63142
Type of Change
Changes Made
gateway/session_context.py— adddeclare_stateless_channel(). Binds only the capability;set_session_vars()would also latch_session_context_engaged, which switches the subprocess env bridge to ContextVar-authoritative — a side effect a pure single-process one-shot must not trigger. Updatedasync_delivery_supported()'s docstring, which claimed cron was always supported.hermes_cli/oneshot.py— call it alongside the other process-wide setup inrun_oneshot().cron/scheduler.py—run_jobpassesasync_delivery=Falseto its existingset_session_vars()call.tools/delegate_tool.py,tools/terminal_tool.py— the two agent-facing strings hardcoded "stateless HTTP API" as the only channel without async delivery. They now name the actual condition, so a cron job is no longer told it is an HTTP endpoint.tests/gateway/test_async_delivery_capability.py— new tests (below).How to Test
Repro on
main:With this PR:
BANANAis returned in-turn.Tests added to
tests/gateway/test_async_delivery_capability.py:test_declare_stateless_channel_disables_async_deliverytest_declare_does_not_engage_full_session_context— pins the reason the helper exists rather than reusingset_session_varstest_background_delegation_runs_inline_when_channel_is_stateless— the behavioral contract: assertsdispatch_async_delegation_batchis not called and the child's result comes back in the payloadBehavior changes worth calling out
Cron delegations now block.
delegation.child_timeout_secondsdefaults to no timeout, and the delegation heartbeat refreshes the parent's activity timestamp, so cron's inactivity watchdog cannot fire until the child's heartbeat goes stale. Worst case is bounded but long. Aworkdirjob also holds_terminal_cwd_lockas a writer for its whole run, so one delegating workdir job blocks other cron jobs for that window. This is the pre-0.18.2 behavior described in #63142 ("delegate_task completed (402.71s, 15,075 chars)✓ sync, real results"), and it is strictly better than returning placeholder text — but it is a change, and installs relying on cron jobs returning promptly should setchild_timeout_seconds.This avoids #63769 rather than aggravating it. The stateless branch returns before
dispatch_async_delegation_batch, so these runners never touch the shared async pool and never reach the pool-at-capacity fallback where that bug lives.Known residual (out of scope).
_run_single_childsubmits without a context copy, so subagents start with the capability unset. A subagent callingterminal(notify_on_complete=True)inside a cron/one-shot run is still told async works and its notification is still dropped. Happy to address in a follow-up.Checklist
Code
scripts/run_tests.sh— 40,646 passed, 4 failed. The 4 (test_bedrock_integration,test_startup_restart_race,test_gateway_runtime_health,test_resolve_provider_openrouter_pool) fail identically on unmodifiedmainand are unrelated to this change.Documentation & Housekeeping
declare_stateless_channel,async_delivery_supported) — the latter previously documented cron as async-capable, which is what made this bug easy to miss