feat(delegation): add same-turn background result injection

[Xipong]

Summary

• add result_delivery: "inject" | "after_turn" to delegate_task, preserving after_turn as the compatibility default
• inject completed child results at the next safe boundary of the originating turn without polling or history rewrites
• persist child-level event identities and claims for exactly-once recovery, including per-child batch delivery
• reconcile provisional finals with at most one bounded grace call per parent turn when an inject arrives at the iteration limit
• keep late results bound to their originating parent turn and retain existing /background behavior
• forward result_delivery through both the direct and ToolRegistry fallback dispatch paths
• persist inject child events before atomically acknowledging the aggregate parent, preventing duplicate aggregate recovery after a crash

Verification

• focused delivery/async/gateway/CLI contract suite: 81 passed
• repository-wide Ruff: passed
• Python compile for all changed Python modules: passed
• Windows footgun scan over origin/main..HEAD: passed (9 files scanned)
• git diff --check: passed
• dependency-complete full runner reached 100%: 48,809 passed, 67 failed assertions; it is therefore not claimed green
  • 60/67 assertion failures were reproduced on clean origin/main under the same sandbox (cua_no_overlay, container boot/chown, XAI migration, service manager/chown, and macOS launcher)
  • 2 more TUI gateway profile failures reproduced on clean origin/main in the focused comparison run
  • the remaining 5 assertion markers (zeroed_state_db, lazy-deps target, search error guard, Termux API detection) passed when rerun in isolation; zeroed_state_db passed 4/4 on both PR and origin/main
  • 9 additional files hit the runner's 600-second per-file timeout. One relevant agent file passed 4/4 in isolation on both PR and origin/main; one Kanban file timed out on both. The other timeout files remain unclassified, so the full-suite result is reported as partial RED evidence rather than waived
• a startup-race retry initially appeared PR-only, but the environments were not equivalent: the PR venv had lazy-installed platform SDKs while baseline did not. Installing the same Feishu SDK made unchanged origin/main reproduce the cold-import timeout; restoring the lock-defined environment produced the final 81/81 focused result

Crash/recovery audit hardening

The follow-up audit found and fixed three durable-delivery defects:

• Two-phase inject acknowledgement: an event now remains durably claimed after its synthetic user message is appended and is acknowledged only after a provider response consumes it. Interrupts and pre-response exits release unpersisted claims; a persisted active-history copy is deduplicated by durable event identity after restart.
• Atomic inject-batch finalization: missing child safety-net inserts and the aggregate bookkeeping update commit in one SQLite transaction. Recovery preserves child-scoped wire shape: a complete set suppresses the aggregate; a partial set adds one terminal gap event rather than a contradictory parent unknown result.
• Formatter failures do not burn delivery attempts: formatting now runs before the durable claim, so repeated local spill/format errors cannot exhaust the eight-attempt delivery budget without presenting anything.

Deterministic regressions cover crash-before-parent-update rollback, complete and partial batch restart recovery, pre-response interrupt release/requeue, persisted-history restart deduplication, provider acknowledgement, duplicate queue entries, and repeated formatter failure.

Additional verification

• tests/agent/test_delegation_inject.py: 18 passed
• Expanded inject/API-server/gateway/CLI/recovery/session-lifecycle matrix: 70 passed
• Conversation-loop, background-review, compression-persistence, prompt-cache, and steer matrix (with declared optional extras): 277 passed
• Ruff and py_compile on all changed Python files: passed

Current head: e37a042debf72d6ae3dc627e680d974c83b09497.

Lease/lifecycle audit follow-up

A second adversarial review identified and fixed the remaining long-provider and rollback windows:

• Claim heartbeat: same-turn durable claims are renewed every 60 seconds while the originating conversation loop still owns them, including pre-API compression, slow provider streams, and retry backoff. A crashed process stops renewing, so the existing 300-second stale-claim recovery remains intact.
• Claim-checked settlement: complete_event_delivery() / release_event_delivery() now return the actual SQLite transition result. RAM claims are removed only after a successful completion or an explicitly observed delivered row; events are requeued only after a successful release that is confirmed pending.
• Compression-safe rollback: unconsumed synthetic messages are removed by durable delegation_id:event_key, not Python object identity, so copied message dicts produced by compression cannot survive rollback.
• Terminal-error ordering: provider-error exits release unconsumed claims before terminal persistence, preventing an unused marker from being mistaken for a durable consumed handoff.
• Exact batch recovery: recovery requires the expected task:0..N-1 key set; extra/malformed task rows cannot make a partial batch appear complete.
• Real loop coverage: regressions exercise actual AIAgent.run_conversation transport assembly, normalize/ack success, and inject -> copied pre-API compression -> provider error -> release/requeue with heartbeat shutdown.

Follow-up verification

• tests/agent/test_delegation_inject.py: 26 passed
• Final inject/API-server/gateway/CLI/compression/abort/async-registry/session-binding matrix: 110 passed
• git diff --check, focused Ruff, and py_compile: passed

One delegation rail, model-selected delivery

This is not a second delegation implementation. The top-level model sees the explicit result_delivery: inject | after_turn enum and chooses only when an already-completed result should become model-visible. Child construction, isolated execution, async registry, durable event ledger, completion queue, formatting, claims/leases, recovery, and late delivery are shared.

• after_turn remains the backward-compatible default for independent work. At each available turn boundary it groups every currently ready, undelivered child into one synthetic result turn; unfinished siblings never block that ready-set and arrive in a later grouped turn.
• inject is for auditors, reviewers, and dependencies that can change the work in progress. A single result re-enters after a complete tool-call/result block and before the next model request. Batch children may re-enter independently as they finish.
• Hermes never waits or polls to keep the foreground turn alive. An inject that misses its bounded same-turn window automatically follows the normal late after_turn path.

This gives frontier models a per-delegation dependency choice without forking execution or durability semantics.

Reviewer follow-up: TUI inject reservation

• Rebased onto current main (4a798f4bce29).
• Added an atomic completion-routing boundary shared by the TUI poller and active same-turn drain, including shutdown routing.
• A TUI durable-claim loss no longer leaves session.running=True.
• Added deterministic dequeue-race and claim-loss regressions.
• Real two-provider-call lifecycle test now asserts byte-identical prior message prefix and unchanged cached system prompt.
• Relevant Python/TUI matrix: 561 passed; Ruff, py_compile, and git diff --check green.

After-turn ready-set invariant

after_turn batch delivery no longer joins every sibling before publishing. At each available turn boundary, the consumer snapshots one batch's completed-but-undelivered child rows under the shared routing lock and produces exactly one transient envelope for that ready-set:

• if only task:0 of 3 is ready, task:0 is delivered immediately after the foreground turn;
• if task:1 and task:2 become ready before a later boundary, they are coalesced into the next result turn;
• if all 3 are ready at one boundary, all 3 are delivered together;
• unfinished siblings are never waited on merely to form an aggregate.

The durable ledger remains child-scoped (delegation_id + task:N). Group claim, renewal, acknowledgement, release, and drop are transactional: a consumer owns the whole ready-set or none of it. Busy TUI/gateway sessions requeue a re-coalescible envelope, so children that finish meanwhile join the next boundary. Finalization publishes a fully-ready set under the same routing lock; crash recovery restores exact pending child keys without reviving the aggregate parent or duplicating delivered children. Legacy aggregate rows already on disk retain their existing delivery path.

Deterministic regressions cover 2/3 now + 1/3 later, all-ready coalescing, atomic group claim/ack, attempt-cap pruning without poisoning a pending sibling, composite requeue plus a newly-ready child, inject/after-turn partial recovery parity, finalizer enqueue locking, idle/busy TUI, idle/busy gateway, CLI drain, and forced-sync fallback isolation.

• Fresh isolated delivery/TUI/gateway/CLI matrix: 575 passed
• Focused Ruff, py_compile, and git diff --check: passed
• Current head: e37a042debf72d6ae3dc627e680d974c83b09497

Gateway routing and live-lease restart hardening

Commit e37a042de closes the two remaining delivery races without changing the intentional ready-set contract:

• The regular gateway watcher now holds completion_routing_lock across the complete bounded routing reservation: dequeue → ready-set coalesce → active-parent classification → requeue. Formatting and adapter/network delivery remain outside the lock. A deterministic post-dequeue pause test proves the active conversation-loop drain cannot observe a temporary-empty queue.
• A quick restart inside another process's still-live 300-second durable claim no longer consumes the only RAM copy. ProcessRegistry owns one lazy, deduplicated heap/condition scheduler. Terminal duplicates are discarded; pending rows wake after the remaining lease plus a small strict-comparison guard, with no immediate-requeue spin. Heartbeat-renewed leases are reclassified and deferred again if still live.
• Every durable claim-loss consumer uses the same seam: active conversation loop, classic CLI, gateway, TUI poller, TUI shutdown drain, and TUI post-turn drain. The TUI marks a session busy only after a successful claim.
• Grouped after_turn envelopes prune already delivered/dropped siblings before retry and retain only pending child rows.
• Documentation now states the actual storage model: both delivery policies use independently claimable child rows; after_turn creates only a transient ready-set envelope. Legacy aggregate rows remain readable.

Verification after rebasing onto 524ab539947aa7a092d749921e0e93913cb683de:

• isolated delivery/async/gateway/CLI/TUI matrix: 581 passed;
• tests/agent/test_delegation_inject.py: 36 passed;
• deterministic gateway routing race, quick-restart live-lease wakeup, terminal-duplicate pruning, grouped mixed-state pruning, and claim-conflict wiring tests: GREEN;
• Ruff, py_compile, and git diff --check: GREEN;
• independent adversarial post-fix audit: 0 High / 0 Medium findings.

[Xipong]

CI is currently blocked at action_required because this PR comes from a fork; GitHub created no jobs.

Could a maintainer approve the workflow run? The focused delivery/async/gateway/CLI suite passes locally (75 tests), and the remaining repository-wide limitations are documented transparently in the PR body.

[teknium1]

Thanks for the substantial durability and recovery work. The feature is still absent from current main, so this is not stale.

Problems

• agent/delegation_inject.py:273-285 appends a synthetic role="user" message during an active turn. That directly conflicts with the repository's stated invariant in AGENTS.md:88-91: never inject a synthetic user message mid-loop. This needs an explicit maintainer-approved cache/alternation design before this path can merge.
• The gateway receives a busy-parent deferral, but the TUI completion poller remains unchanged. tui_gateway/server.py:8726-8737 concurrently removes and requeues completion events while its session runs, so the loop has no deterministic reservation of an inject event. No TUI regression is included.

Suggested changes

• Resolve the mid-loop user-message invariant with maintainers, then encode the approved contract in tests.
• Add TUI-aware inject deferral/reservation and a deterministic TUI delivery test.

Automated hermes-sweeper review.

[teknium1]

This injects a synthetic user message into an active parent turn. AGENTS.md:88-91 explicitly requires strict alternation and says never to inject a synthetic user message mid-loop; this needs an approved design exception or a different delivery mechanism before merge.

[Xipong]

Thanks — the current wording of the invariant is clear. Before replacing this transport, could you advise which representation you want if same-turn delivery itself is acceptable?

I chose a typed user tail because, at an established assistant/tool boundary, it is the only portable cross-provider role that represents new external information. An assistant message would impersonate model output; a tool message requires a matching unresolved assistant tool call; and injecting system/developer content mid-loop would change the stable system/prompt-cache contract. The message is explicitly tagged _synthetic_delegation_inject / display_kind=delegation_inject and is only appended at guarded safe boundaries.

I also traced the cache behavior rather than assuming it: the follow-up at 1bd9aba61 now has a real two-provider-call regression asserting that request 2 starts with request 1 as a byte-for-byte identical message prefix and that _cached_system_prompt is unchanged. The inject appears only in the new tail; existing system/history bytes are not rewritten. Unconsumed tails are removed by durable event identity across compression copies.

If that append-only/cache-safe contract is acceptable, would you prefer (1) a narrow documented exception to the invariant for typed internal delegation input, plus a test forbidding any other synthetic-user mid-loop path, or (2) a specific existing internal event/input representation? The current /steer seam only attaches input to a tool result and does not cover provisional-final or no-tool boundaries, so I do not want to guess an alternative that silently breaks provider role validity or cache stability.

[Xipong]

One product-level point I undersold above: this is not a second delegation implementation or a workaround around the legacy path. It is one delegation rail with a model-selected delivery policy.

The top-level model sees the explicit result_delivery: inject | after_turn enum and chooses only the result timing; top-level child execution remains asynchronous regardless of the deprecated background argument. Both modes use the same child construction, executor, async registry, durable ledger, completion queue, formatter, claim/lease, recovery, and late-delivery machinery. after_turn remains the backward-compatible default.

For a single child, the semantic difference is deliberately narrow: if the result is ready during the originating turn, inject makes it model-visible after a complete tool-call/result block and before the next model request; after_turn makes the same completed result visible in a separate synthetic turn. If the inject misses the bounded same-turn window, it automatically follows that same late after_turn path — no wait, no dropped result, and no unrelated future turn contamination. For a batch there is one intentional additional benefit: inject children publish independently as they finish, while after_turn preserves the consolidated all-siblings result.

That gives a frontier model a useful per-delegation choice: independent work can stay after_turn; an audit, reviewer, or dependency that can invalidate the work in progress can use inject and influence the current answer. The execution and durability semantics do not fork. The exact schema/forwarding, incremental-inject batch, and consolidated-after-turn tests all pass.

Given those properties plus the append-only/cache-prefix proof, I think this is a strong architecture and that a narrow documented invariant exception for typed internal delegation input would be preferable to replacing it with a less valid provider role or a cache-sensitive system mutation. Does that direction make sense to you? If yes, I can codify the exception and its guard test rather than redesigning a working delivery rail.

[Xipong]

Follow-up lifecycle audit fixes pushed in b88a7e583201e14481fcbacd5f2d3c95b83c24d0:

• renew active inject claims every 60s across pre-API compression, slow provider calls, and retry backoff
• keep RAM claims/requeue behavior conditional on the actual SQLite complete/release transition
• roll copied compression messages back by durable event identity instead of Python id()
• release unconsumed injects before terminal provider-error persistence
• require exact expected task:N keys for recovered batches
• add real AIAgent.run_conversation success and compression-copy/provider-error lifecycle regressions

Verification: tests/agent/test_delegation_inject.py 26 passed; final focused inject/API-server/gateway/CLI/compression/abort/async-registry/session-binding matrix 110 passed; Ruff, py_compile, and git diff --check passed.

The fork workflow is still expected to require maintainer approval; could a maintainer approve the new run when available?

[Xipong]

Addressed the TUI inject race in 1bd9aba61 (rebased onto current main at c9de69c6d).

The TUI poller and active conversation loop now share a bounded completion-routing lock covering only dequeue → ownership decision → requeue/claim. The lock is released before any agent/model turn. This means an inject event cannot be temporarily held outside the queue while drain_ready_injects() snapshots it. The same coordination now covers the poller's shutdown drain, since teardown can signal the poller while the active turn is still unwinding.

I also moved session["running"] = True after a successful durable claim. Previously, losing the claim to the active loop could leave the TUI session permanently marked busy.

Regression evidence:

• RED before the fix: a deterministic concurrency test pauses the TUI immediately after dequeue; the active drain returned 0 while the ready event was temporarily absent.
• GREEN after the fix: the active drain blocks through the atomic route/requeue section, then claims and appends the event exactly once.
• Added a claim-loss regression asserting the TUI remains idle when another durable consumer wins.
• Added append-only cache evidence to the real two-provider-call lifecycle test: request 2 preserves request 1 as a byte-identical prefix and _cached_system_prompt remains unchanged.
• Full relevant matrix: 561 passed; Ruff, py_compile, and git diff --check are green.

The new fork CI run is blocked at action_required with no jobs created: https://github.com/NousResearch/hermes-agent/actions/runs/30578032085. Could a maintainer approve this workflow run?

[Xipong]

Fixed a blocking after_turn batch semantic bug in d1a2f47e3 (rebased onto current main).

The previous path emitted only the final parent aggregate, so one fast child could be hidden behind a slow sibling indefinitely. The invariant is now: at each available turn boundary, deliver exactly all completed-but-undelivered children from one batch in one coalesced envelope; never wait for unfinished siblings. Thus 1/3 may arrive now, and the remaining 2/3 later as one grouped turn; if all 3 are ready at the same boundary, all 3 arrive together.

Durability remains child-scoped (delegation_id + task:N). Ready-set claim/renew/ack/release/drop are transactional, parent finalization is bookkeeping-only, final multi-child enqueue is protected by the shared routing lock, busy TUI/gateway envelopes are re-coalescible, and recovery restores exact pending keys without aggregate duplication. Legacy aggregate rows already on disk remain readable through the old path. inject keeps its same-turn semantics and shares the child ledger.

Verification on a fresh isolated HERMES_HOME: 575 relevant delivery/TUI/gateway/CLI tests passed; focused Ruff, py_compile, and diff-check passed.

[Xipong]

Closed the two remaining delivery races in e37a042de (rebased onto current main 524ab5399).

1. Regular gateway watcher: completion_routing_lock now covers the complete bounded reservation — dequeue -> ready-set coalesce -> active-parent classification -> requeue. Only events classified idle leave the lock; formatting and adapter/network delivery stay outside it. test_gateway_busy_route_is_atomic_with_same_turn_inject_drain deterministically pauses after dequeue and proves the active conversation-loop drain cannot observe a temporary-empty queue.

2. Quick restart inside a live delivery lease: a failed durable claim is now classified as terminal duplicate vs pending live-claimed row. Pending rows enter one lazy deduplicated ProcessRegistry heap/condition scheduler and return after the remaining lease plus a strict-comparison guard, so there is no hot immediate-requeue loop and no dependency on another restart/rescan. All claim-loss consumers use this seam (conversation loop, CLI, gateway, TUI poller/shutdown/post-turn). Grouped envelopes prune terminal siblings before retry.

Also corrected the stale docs: both inject and after_turn use independently claimable child rows; after_turn builds a transient ready-set envelope, while legacy aggregate rows remain readable. The intentional ready-set delivery contract is unchanged.

Post-rebase verification: 581 passed in the isolated delivery/async/gateway/CLI/TUI matrix; test_delegation_inject.py 36 passed; Ruff, py_compile, and diff-check GREEN. Independent adversarial post-fix audit: 0 High / 0 Medium.

[Xipong]

CI for head e37a042debf72d6ae3dc627e680d974c83b09497 is blocked at action_required (run 30587846896; no jobs were created for the fork PR). Could a maintainer approve the workflow run? The post-rebase isolated delivery/async/gateway/CLI/TUI matrix passes locally: 581/581, with Ruff, py_compile, and diff-check GREEN.

[teknium1]

Thanks for the enormous amount of engineering here — the ledger/lease/two-phase-ack machinery and the test coverage are genuinely impressive work, and this was reviewed carefully against current origin/main.

Closing on architectural grounds, not quality: inject mode appends synthetic role:"user" messages inside a still-running parent turn. Strict message-role alternation (no synthetic user messages mid-loop) is a hard, deliberate invariant in Hermes — the docstring this PR removes from tools/async_delegation.py documents exactly why background delegation results re-enter only between turns (alternation legality + prompt-cache integrity). Some providers handle assistant(tool_calls)→tool→user→assistant sequences inconsistently, and the ~10 hook sites plus lease/heartbeat/rollback paths in conversation_loop.py's hot path are a large permanent cost for a benefit the existing after_turn queue already delivers at the next turn boundary.

Two pieces of this are worth extracting into their own PRs, and we'd welcome them:

1. The after_turn ready-set batching (a fast child no longer blocked behind a slow sibling) — that's a clean improvement to the existing rail.
2. The durable child-level event ledger / crash recovery, if it can be framed for the existing between-turns path.

If maintainers ever decide to relax the alternation invariant, this PR is the reference implementation and we'll point back to it. Thanks again — your other delegation fixes (#74858, #66401) are moving forward.

[Xipong]

Implemented the close review as a three-part stack:

• refactor(delegation): persist child-scoped delivery state #76228 — child-scoped durable settlement for the existing delegation rail
• feat(delegation): route ready children between turns #76229 — ready-set routing at existing between-turn consumers
• feat(delegation): surface ready dependencies before the next decision #76230 — opt-in visibility of a ready dependency at an existing tool boundary

The central distinction is causal, not textual. after_turn can eventually deliver the same child output, but an auditor or prerequisite that was already ready before the next decision loses its purpose if the parent first edits, commits, or finalizes. #76230 uses that available window without waiting; if the window does not occur, the same durable event remains on after_turn.

This does not create a second delivery implementation. A parent may have dependency/review work and independent work in flight as parts of one orchestration graph. result_delivery records which consumer boundary may first claim the event. Execution, child identity, queueing, claims, recovery, formatting, settlement, and late delivery remain shared.

The rejected representation has also been removed rather than defended:

#74378 concern	#76230
synthetic role:"user" inside an active turn	no delegation message or user authorship is created
provider-specific tool → user → assistant tail	unchanged assistant(tool_calls) → tool → assistant continuation
prompt/history tail complexity	only the new unsent tool result is finalized before first persistence; prior messages are untouched
provisional-final/grace lifecycle and ~10 hooks	removed; preparation/settlement is localized to the existing post-tool persistence checkpoint with common rollback
TUI/loop competition	one routing reservation and durable claim API from #76229
after_turn already supplies the benefit	it supplies eventual output, not the already-available opportunity to revise unfinished work

The new carrier appends a clearly delimited DELEGATION RESULT READY framework block to the last result of a complete tool batch—the same established role-preserving boundary used by /steer. The marker explicitly identifies background evidence and says it is not a new user request. If there is no complete tool batch, no carrier is fabricated.

This is orchestration semantics, not new user guidance or a model-preset project. Weaker models may use same-turn evidence poorly or drift toward the child topic; that is a bounded model-quality risk. The safe baseline remains unchanged: after_turn is the default and late fallback, and the stack adds no settings, required /goal, waiting, grace call, or budget extension.

Each PR body now states its owned commit and standalone effect. All three exact remote heads remain fully green; #76230 contains the complete carrier, failure-ordering, and compatibility rationale.