fix(delegation): preserve redacted output on timeout

[ypwcharles]

What does this PR do?

When a delegate_task child times out after completing real work, the parent currently receives a timeout entry with no summary and loses every completed tool result. A research child can finish several web_search, x_search, or terminal calls, hang on a final step, and leave the parent with nothing to continue from even though useful work existed.

This PR preserves a bounded, redacted snapshot of completed child tool outputs on the timeout path and returns it as partial: true plus partial_output_tail[]. It also renders that evidence through the async/background notification path used by delegate_task(background=true).

Attribution and consolidation

• Original implementation: Charles Cha (@ypwcharles), originally published in this PR.
• Follow-up hardening: Matvii Sakhnenko (@sakhnenkoff), contributed through fix(delegation): preserve redacted partial output when a subagent times out (rebase of #65824 + background path) #84085 and preserved as a separate authored commit here.
• This branch is the canonical consolidation on current main; it keeps both contributors' git authorship intact.
• Independent pre-push review then found and fixed a single-task async transport gap as a third Charles-authored commit.

What changed

Timeout evidence

• Snapshot the child's completed in-memory tool results when a real child timeout fires.
• Attach at most 8 entries, each capped at 600 characters.
• Add the fields only when a safe non-empty tail exists:
  • partial: true
  • partial_output_tail: [{tool, preview, is_error}, ...]
• Preserve the legacy schema when there is no completed output or extraction cannot be performed safely.

Security boundary

• Apply redact_sensitive_text(..., force=True) to the complete output before truncation. This prevents truncation from splitting a credential into a fragment that no longer matches the redactor.
• Timeout evidence uses fail_closed=True: any redaction failure drops the entire tail.
• The shared ordinary delegation Output overlay remains best-effort: one unredactable entry is skipped without blanking all healthy output. This scopes fail-closed behavior to the failed-child evidence boundary instead of regressing the normal display path.

Background delivery

• Render validated partial_output_tail entries in async batch and single-result completion formatting.
• Preserve the validated additive fields through the single-task async_delegation completion-event boundary.
• Reject malformed tail entries at that transport boundary and retain the legacy event schema.
• Persist only the sanitized event evidence; omit the untrusted pre-sanitized tail from durable result_json / status reads.
• Cover the real dispatch → completion queue → durable state → public format_process_notification() route used to re-inject background delegation results into the parent.

Scope guards

• Only the child-timeout path exposes partial evidence.
• Ordinary non-timeout exceptions retain their existing error schema.
• Zero-API-call (before_first_llm_call) diagnostic behavior is unchanged.
• Successful child result entries remain unchanged.
• Existing consumers can ignore the additive timeout fields.

Schema

{
  "status": "timeout",
  "summary": null,
  "error": "...",
  "diagnostic_path": "...",
  "partial": true,
  "partial_output_tail": [
    {
      "tool": "terminal",
      "preview": "Build succeeded...",
      "is_error": false
    }
  ]
}

How to test

Validated on Ubuntu 24.04 / WSL2 with Python 3.11.15.

Focused canonical runner on the final three-commit rebased head:

scripts/run_tests.sh \
  tests/tools/test_delegate.py \
  tests/tools/test_delegate_subagent_timeout_diagnostic.py \
  tests/tools/test_process_registry.py \
  tests/tools/test_async_delegation.py \
  tests/tools/test_delegate_composite_toolsets.py \
  tests/tools/test_delegate_toolset_scope.py \
  tests/tools/test_delegate_summary_budget.py -q

Result: 205 passed, 0 failed, 4 skipped. The skipped cases are Windows-only and run in the Windows CI lane.

The suite includes real single-background dispatch → queue → durable-state → notification coverage, malformed-tail and non-timeout legacy-schema coverage, forced redaction/bounds coverage, and redactor-failure fail-closed coverage. All new transport/security regressions were observed failing for the expected reason before the production fix, then passing after it.

Static gates:

python scripts/check-windows-footguns.py --diff upstream/main
git diff --check upstream/main..HEAD
python -m py_compile \
  tools/delegate_tool.py \
  tools/process_registry.py \
  tools/async_delegation.py \
  tests/tools/test_delegate.py \
  tests/tools/test_delegate_subagent_timeout_diagnostic.py \
  tests/tools/test_async_delegation.py

Result: all passed.

Full canonical suite was also executed. It reported 31 files with 92 failures, plus 2 files that did not complete collection/run, all outside this PR's six changed files. This isolated worktree intentionally reused the runtime dependency set and added only pytest tooling, so optional-provider/environment failures were not treated as PR regressions. The related delegation, timeout, and process suites above remained clean.

Related work

• fix(delegate_task): silent timeout kills subagents without notifying parent session #51294 added timeout lifecycle notification but did not preserve completed tool output.
• [Bug]: N-API-call subagent timeout lacks tool_trace diagnostics — cannot identify last stuck tool #17308 and its related PR cluster concern structured last-tool state classification; this PR is complementary and intentionally limited to bounded redacted partial-output preservation.
• fix(delegation): preserve redacted partial output when a subagent times out (rebase of #65824 + background path) #84085 supplied the follow-up fail-closed scoping and boundary-regression test now consolidated here.

Checklist

• Read the contribution guide
• Conventional commit messages
• Narrow bug-fix scope
• Tests added for synchronous, batch, single, and background paths
• Redact-before-truncate boundary tested non-vacuously
• Linux/WSL focused canonical suite passed
• Windows footgun scan passed; Windows-only tests left to CI
• Contributor authorship preserved

[tonydwb]

Code Review Summary

Verdict: Approved

Preserves redacted output on delegation timeout. Note: The "secret" in diff is a test fixture (sk-... pattern in test code), not a real credential. No security concerns.

---

Reviewed by Hermes Agent

[teknium1]

Thanks for preserving bounded, force-redacted evidence from timed-out children. The synchronous timeout record and subagent.complete progress payload are covered, but the background completion path still drops the new data.

Problems

• delegate_task(background=true) uses the batch completion route (tools/delegate_tool.py:2803-2812). That route retains task results, but _format_async_delegation() renders timeout/error tasks solely from summary and error (tools/process_registry.py:2091-2121); it never reads partial_output_tail. The new field added at tools/delegate_tool.py:2110 therefore is not reinjected into the parent session for the background workflow.

Suggested changes

• Render validated partial_output_tail entries in both async batch and single-result formatting, and add a regression test through format_process_notification() using a timeout result with a redacted preview.

This is an automated hermes-sweeper review.

[ypwcharles]

Addressed review feedback: _format_async_delegation now renders partial_output_tail for timeout results in both batch and single-task paths.

Changes:

• tools/process_registry.py: added _render_partial_output_tail() helper; batch path renders tail per-task after status line; single-task path renders tail before return. Only activates when partial_output_tail is present and non-empty — success/interrupted paths unaffected.
• tests/tools/test_delegate.py: added 12 regression tests covering batch/single × timeout/success × with/without tail × mixed states.

Post-fix validation: 188 passed, 0 failed (canonical focused suite).

[ypwcharles]

Added format_process_notification() regression tests per sweeper review.

New tests in TestFormatAsyncDelegationPartialTail:

• test_format_process_notification_timeout_with_tail: single-task timeout → renders Partial output (redacted) + tool preview
• test_format_process_notification_batch_timeout_with_tail: batch timeout → same

Validation: 174 + 16 = 190 passed, 0 failed. New head: 37d99f012.

[chrisyoung2005]

Independent production confirmation + E2E verification of this PR.

Production repro (v0.19.0 / tag v2026.7.20, self-hosted: Fedora/podman, local llama.cpp model, gateway mode): a 3-agent research pipeline had both child subagents killed by a 600 s wall-clock cap while making steady progress — one child had ~15 completed web_search/extract calls when it was cut off. The parent received only {status: "timeout", error, diagnostic_path}; every completed tool result was discarded and the whole stage had to re-run from scratch. Exactly the loss mode this PR addresses.

Worth noting for triage: the code default is already no-cap (DEFAULT_CHILD_TIMEOUT = None, opt-in via delegation.child_timeout_seconds) — our 600 s turned out to be a stale literal written into a long-lived config.yaml by an older install's template and carried across upgrades (the current template writes 0/disabled). So the exposed population is anyone with a configured cap, whether deliberate (we now run child_timeout_seconds: 1800 to bound runaway children on a shared-GPU box) or silently inherited from an old config file — and today they lose all partial work when it fires. This PR is the missing half: cap enforcement without total loss, and it gives the parent enough context to continue instead of restarting.

Verification (head 37d99f0, dev checkout):

• scripts/run_tests.sh tests/tools/test_delegate.py tests/tools/test_delegate_subagent_timeout_diagnostic.py tests/tools/test_process_registry.py → 303 passed, 0 failed.
• Fail-pre-fix proven: reverting tools/delegate_tool.py + tools/process_registry.py to the parent commit fails 14 of the PR's tests (partial-tail extraction, bounded tail count/preview size, redaction of progress events) — the tests bite.

Also +1 on the ordering choice: applying redact_sensitive_text(force=True) before truncation (and dropping the tail entirely on redaction failure) closes the split-credential leak a truncate-then-redact order would open. Happy to re-test after rebases.

---

Correction 2026-07-28: an earlier version of this comment attributed the 600 s to the release default. Verified against v2026.7.7.2 and v2026.7.20 (both ship DEFAULT_CHILD_TIMEOUT = None): it was a legacy value in our own config file, as described above. The observed loss behavior and verification results are unchanged.

[ypwcharles]

Thanks for the independent production confirmation and E2E verification — this validates the remaining opt-in timeout case even after main removed the default cap. No logic change is needed from this report: the current implementation already preserves bounded evidence for configured wall-clock limits, and keeps the security-critical redact-before-truncate / fail-closed ordering you verified.

I have also rebased the PR onto current main (c7dd9e567) and resolved the only conflict by retaining both upstream structured timeout-metadata tests and this PR’s partial-tail safety tests.

Post-rebase validation:

• delegation + timeout diagnostics: 195 passed
• async formatting subset: 5 passed
• diagnostic suite after conflict resolution: 19 passed
• py_compile, diff check, Windows-footgun scan: passed

New head: be54e7e34.

[chrisyoung2005]

Re-verified at the new head be54e7e34 (post-rebase onto c7dd9e567), as offered:

• scripts/run_tests.sh tests/tools/test_delegate.py tests/tools/test_delegate_subagent_timeout_diagnostic.py tests/tools/test_process_registry.py → 316 passed, 0 failed (up from 303 at 37d99f0 — the delta is the upstream structured timeout-metadata tests retained in the conflict resolution, running alongside this PR's partial-tail tests as intended).
• Fail-pre-fix still proven at this head: reverting tools/delegate_tool.py + tools/process_registry.py to the merge-base fails 14 tests (partial-tail extraction/rendering in batch, single-task, and notification paths; bounded preview size; redaction of progress events) — same count as the pre-rebase run, so the rebase didn't dilute test bite.
• Spot-checked the security ordering survived the rebase: redact_sensitive_text(content, force=True) still runs on the complete tool output before the [:max_chars] truncation, with drop-tail-on-redaction-failure intact.

Conflict resolution looks right from here — both test families coexist and pass. No further asks from our side.

[ypwcharles]

Thanks for re-verifying the rebased head and sabotage-checking the regressions. This is confirming evidence rather than a change request: it shows the structured timeout-metadata tests and partial-tail tests coexist, still fail on the pre-fix behavior, and retain redact-before-truncate / fail-closed ordering. No scope expansion is needed.\n\nThe PR later became conflicted again, so I rebased it onto current main (36f885573). During conflict resolution I retained only this PR’s partial-tail tests and dropped stale context tests that current main had intentionally removed.\n\nPost-rebase validation:\n- delegation + timeout diagnostics + process formatting: 136 passed\n- py_compile, Windows-footgun scan, and git diff --check: passed\n\nNew head: dab408e30.

[chrisyoung2005]

Re-verified at the new head dab408e30 (post-rebase onto 36f885573):

• scripts/run_tests.sh tests/tools/test_delegate.py tests/tools/test_delegate_subagent_timeout_diagnostic.py tests/tools/test_process_registry.py → 136 passed, 0 failed — matching your own post-rebase validation count.
• Fail-pre-fix still proven at this head: reverting tools/delegate_tool.py + tools/process_registry.py to the merge-base fails 14 tests, same count as at 37d99f0/be54e7e34 — the conflict resolution that dropped the stale context tests didn't dilute the partial-tail regressions.
• Redact-force-before-truncate survived again: redact_sensitive_text(content, force=True) runs on the complete tool output before the [:max_chars] preview cut, drop-tail-on-redaction-failure intact.

Still merge-ready from our side.

[ypwcharles]

Thanks for re-verifying the current head dab408e30. This confirms the post-rebase focused suite remains green (136 passed), the 14 fail-pre-fix regressions still bite, and the redact-before-truncate / fail-closed ordering is intact. This is confirming evidence rather than a change request, so no code or scope change is needed.

[GottZ]

This was generated by AI during triage.

Summary

Nine PRs address or reference two adjacent delegation issue cores: #17312/#17329/#63379 target structured N-API-call timeout state, #31207/#37724 extend timeout artifacts, #15105 covers zero-call diagnostics, #65824 preserves redacted partial output, and #37368/#38983 isolate child session identity.

Related pull requests

• feat(delegate): diagnostic dump when a subagent times out with 0 API calls #15105 [merged] related — (+490/-14) — n/a: This landed reference implementation adds structured diagnostics only for zero-API-call timeouts and explicitly leaves N>0 timeouts on the generic path, so it establishes adjacent behavior without resolving [Bug]: N-API-call subagent timeout lacks tool_trace diagnostics — cannot identify last stuck tool #17308.
• feat(delegate): extend timeout diagnostics for N-API-call subagents #17312 partial — (+259/-5) — n/a: It attempts to add timeout tool-trace fields, but the contributor keep_open review identifies that the exception path reads undefined result, leaving the trace empty, and notes missing timeout regression tests; its unrelated ACP-provider change also broadens the diff.
• fix(delegate): surface tool_trace on N-API-call subagent timeouts (#17308) #17329 [closed] partial — (+367/-32) — n/a: This closed predecessor reconstructs live child messages and distinguishes completed from in-progress tools, but the later keep_open review identifies its substring error classifier and missing content-block coverage; it remains relevant as the original design superseded by the corrected fix(delegate): surface tool_trace on N-API-call subagent timeouts (#17308) #63379 salvage.
• fix(delegate): write timeout diagnostics for in-flight subagents #31207 [closed] partial — (+29/-25) — n/a: This closed alternative extends diagnostic-file generation to N>0 timeouts and exposes stack/activity evidence, but it does not return the requested tool trace or completed-versus-in-progress classification.
• fix(delegation): keep child sessions out of parent env #37368 [closed] partial — (+54/-4) — n/a: This closed delegate_task subagent overwrites os.environ["HERMES_SESSION_ID"] in parent process #37356 fix prevents direct parent-environment clobbering during child construction, but it remains relevant as the narrower predecessor to fix(delegate): scope HERMES_SESSION_ID to subagent env instead of mutating parent (#37356) #38983, which also handles worker binding and compression rotation or rollback.
• fix(delegate): improve subagent timeout diagnostics #37724 [closed] partial — (+310/-102) — n/a: This closed historical overlay writes redacted artifacts for every timeout and adds pre-interrupt stack capture plus bounded cleanup status, but it does not reconstruct the structured last-tool state requested by [Bug]: N-API-call subagent timeout lacks tool_trace diagnostics — cannot identify last stuck tool #17308.
• fix(delegate): scope HERMES_SESSION_ID to subagent env instead of mutating parent (#37356) #38983 best fix — (+289/-17) — n/a: This delegate_task subagent overwrites os.environ["HERMES_SESSION_ID"] in parent process #37356 fix leaves child construction outside the parent session binding, scopes the child ID to the execution worker, restores it afterward, and protects compression rotation and rollback; its revised diff and execution-path tests explicitly address the contributor keep_open review.
• fix(delegate): surface tool_trace on N-API-call subagent timeouts (#17308) #63379 [closed] best fix — (+504/-32) — n/a: This closed best-fix reference salvages fix(delegate): surface tool_trace on N-API-call subagent timeouts (#17308) #17329 by reading the live transcript and using main's normalized content and conservative error classifier, with focused timeout coverage; despite its keep_open verdict finding no blocker, it was closed as a duplicate of canonical feat(delegate): extend timeout diagnostics for N-API-call subagents #17312.
• fix(delegation): preserve redacted output on timeout #65824 partial — (+640/-11) — n/a: This distinct PR preserves bounded, redact-before-truncate, fail-closed tool-output tails across synchronous, batch, single-task, and notification paths; the diff addresses the contributor keep_open review's missing background rendering, but intentionally does not provide [Bug]: N-API-call subagent timeout lacks tool_trace diagnostics — cannot identify last stuck tool #17308's last-tool state classification.

Duplicates

#17312, #17329, and #63379 address the same structured N-API-call timeout-trace gap, with #63379 the corrected salvage of #17329; #31207 and #37724 overlap on all-timeout diagnostic artifacts, while #37368 is the narrower predecessor of #38983 for #37356. #65824 is complementary partial-output preservation, not a duplicate of the trace cluster.

Suggested consolidation

Keep #65824 open with a salvage path for its independently tested redacted partial-output preservation; its current diff addresses the visible async-rendering review, and it should remain separate from #17308's trace work. For #17312, author action: rebase onto main and replace the non-populating timeout path with the live-transcript, conservative-classifier implementation demonstrated by closed best-fix reference #63379, including its timeout tests; retain #17329 as its superseded predecessor and leave #31207/#37724 closed. Keep #38983 open with its worker-scoped session-isolation path because the revised diff explicitly addresses the contributor review, while #37368 remains closed as its narrower predecessor.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I17308(["issue #17308 (open)"])
    P65824["PR #65824 (open)"]
    P65824 -.->|partial| I17308
    class I17308 open
    class P65824 open
    class P65824 target
    click I17308 "https://github.com/NousResearch/hermes-agent/issues/17308"
    click P65824 "https://github.com/NousResearch/hermes-agent/pull/65824"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 9 pull requests and 2 issues in this complex. Each diff was read against this issue; Assessment working set: 169 kB of PR diffs, 29 kB of issue/PR text, 22 kB of discussion (29 comments), 10 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[ypwcharles]

Thanks for the cross-PR triage. Agreed: #65824 is complementary to, not a duplicate of, the #17308 structured last-tool trace cluster. Its scoped invariant is preserving a bounded, redact-before-truncate, fail-closed partial-output tail across synchronous, batch, and background-notification paths; last-tool state classification remains out of scope. The current head already includes the reviewed async rendering coverage, so this consolidation note does not require a code change.

[ypwcharles]

Rebased the canonical PR onto current main and consolidated the useful follow-up from #84085 while preserving Matvii Sakhnenko as the author of that commit.

The final three-commit range keeps the original implementation and follow-up hardening separately attributable. Independent pre-push review also found a single-background transport gap; the third commit fixes that path with timeout-only, re-redacted, bounded, fail-closed event transport while preserving the legacy schema for ordinary results.

Validation on the final head:

• related canonical delegation/timeout/process/async suite: 205 passed, 0 failed, 4 Windows-only skipped;
• Windows-footgun scan across all 6 changed files: passed;
• git diff --check: passed;
• py_compile for all changed Python files: passed;
• full canonical suite status and environment limitations are recorded in the updated PR body.

#84085 can now close in favor of this canonical consolidation; its genuine incremental work remains credited in git history.