Skip to content

fix(kanban): salvage partial findings on iteration-budget exhaustion - #52511

Open
DanielMaly wants to merge 1 commit into
NousResearch:mainfrom
DanielMaly:fix/kanban-timeout-salvage
Open

DanielMaly wants to merge 1 commit into
NousResearch:mainfrom
DanielMaly:fix/kanban-timeout-salvage

Conversation

@DanielMaly

Copy link
Copy Markdown

Summary

When a Kanban worker exhausts its iteration budget, the agent loop asks the model for a final toolless summary via _handle_max_iterations(). Previously that summary was discarded — _record_task_failure() was called without forwarding it, so the run row closed with summary=None and the retry worker started from scratch with no partial findings.

This PR forwards the model's last output as partial_summary to _record_task_failure, which writes it to the run row's summary field via _end_run. build_worker_context already surfaces run summaries in the prior-attempts section, so the retry worker sees the partial findings automatically.

Additionally, a [partial_unverified] task comment is posted containing the model's last output, so it appears in the retry worker's comment thread. The tag marks it as emergency-salvaged content from a budget-exhausted run, not a deliberate handoff.

Closes #52510

Root cause

In agent/turn_finalizer.py, the budget-exhaustion path calls _record_task_failure(outcome="timed_out") but never passes the final_response (the model's last output from _handle_max_iterations). The _record_task_failure function had no parameter to accept a partial summary, and _end_run() was called without summary=, leaving the run row's summary field as NULL.

Changes

  • hermes_cli/kanban_db.py: Add partial_summary parameter to _record_task_failure(), forwarded to _end_run(summary=...) in both the breaker-trip (gave_up) and below-threshold paths.
  • agent/turn_finalizer.py: Forward final_response as partial_summary to _record_task_failure; post a [partial_unverified] comment with the model's last output; improve the comment explaining why _record_task_failure is used instead of kanban_block (separating the two concerns: circuit-breaker counting vs. tools being stripped).
  • agent/prompt_builder.py: Add a brief iteration-budget-exhaustion paragraph to KANBAN_GUIDANCE so workers know the salvage mechanism exists and are encouraged to proactively kanban_comment + kanban_block before hitting the cap.
  • Tests: 7 new tests — 3 in test_kanban_db.py covering partial_summary plumbing (timed_out / gave_up / null cases), 4 in test_turn_finalizer_kanban_salvage.py covering the end-to-end salvage flow.

Design choices

  • No fabrication risk: The salvaged text is genuine model output, clearly tagged [partial_unverified].
  • No recursion risk: Tools are stripped before _handle_max_iterations, so the model cannot call kanban tools during the summary API call.
  • No double-recording risk: Status gating prevents the dispatcher's crash detector from re-recording.
  • Why not reserved finalization tool turn: A reserved-tool-turn approach (enforcing a budget so one tool call remains for handoff) is more complex and changes agent loop semantics. The current approach is simpler and sufficient — it salvages what the model already produces. This approach is complementary and could be added later if needed.
  • failure_limit not passed from turn_finalizer: The worker process does not have access to the dispatcher's kanban.failure_limit config. The _record_task_failure default (DEFAULT_FAILURE_LIMIT) is acceptable here — the per-task max_retries override still takes precedence when set.

Test plan

  • test_record_task_failure_salvages_partial_summary — verifies partial_summary lands in run row's summary and is surfaced by build_worker_context
  • test_record_task_failure_without_partial_summary_stays_null — backward-compatible default (existing callers unaffected)
  • test_record_task_failure_partial_summary_on_gave_up — partial summary persists when circuit breaker trips
  • test_budget_exhaustion_salvages_partial_summary_to_run — end-to-end: finalize_turn forwards model output to run summary
  • test_budget_exhaustion_writes_partial_unverified_comment — [partial_unverified] comment is written with correct author
  • test_budget_exhaustion_no_summary_no_comment — no comment when model produces empty summary
  • test_budget_exhaustion_retry_worker_sees_partial_findings — retry worker's build_worker_context contains both the partial summary and the comment
  • 236 existing kanban_db + turn_finalizer tests still pass (no regressions)

When a Kanban worker exhausts its iteration budget, the agent loop
already asks the model for a toolless summary via
_handle_max_iterations.  Previously that summary was discarded:
_record_task_failure was called without it, so the run row closed
with summary=None and the retry worker started from scratch with no
partial findings.

This change forwards the model's last output as partial_summary to
_record_task_failure, which writes it to the run row's summary field
(via _end_run).  build_worker_context already surfaces run summaries
in the prior-attempts section, so the retry worker sees the partial
findings automatically.

Additionally, a [partial_unverified] task comment is posted containing
the model's last output, so it appears in the retry worker's comment
thread.  The tag marks it as emergency-salvaged content from a
budget-exhausted run, not a deliberate handoff.

The prompt_builder KANBAN_GUIDANCE is updated with a brief
iteration-budget-exhaustion paragraph so workers know the salvage
mechanism exists and are encouraged to proactively kanban_comment +
kanban_block before hitting the cap.

Design notes:
- This is the minimal DB-side salvage (Option 1 + Option 2 from the
  design review).  It has no recursion risk (tools are stripped in
  the summary API call), no fabrication risk (genuine model output,
  clearly tagged), and no double-recording risk (status gating).
- The soft-budget-warning (Option 3) and reserved-finalization-tool-turn
  (Option 4) approaches are complementary but deferred — the current
  approach is simpler and sufficient.

Tests: 7 new tests (3 in test_kanban_db.py covering partial_summary
plumbing for timed_out/gave_up/null cases, 4 in
test_turn_finalizer_kanban_salvage.py covering the end-to-end salvage
flow).  236 existing tests still pass.
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cron Cron scheduler and job management P3 Low — cosmetic, nice to have labels Jun 25, 2026
@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for tracing the run-summary loss; current main still closes the Kanban timeout run without a summary, while retry context renders closed-run summaries.

Problems

  • Current main has a second iteration-limit fallback: agent/turn_finalizer.py:72-81 preserves _pending_verification_response without calling _handle_max_iterations. The regression at tests/agent/test_turn_finalizer_iteration_limit_exit.py:135-149 verifies that behavior. The PR's new [partial_unverified] comment calls every saved value the worker's last model output before timeout, which is not true for this preserved-response path.

Suggested changes

  • On salvage, distinguish a generated toolless summary from a preserved verification response, or use origin-neutral wording; add coverage alongside tests/agent/test_turn_finalizer_iteration_limit_exit.py:270-298.
  • The summary plumbing remains useful: current hermes_cli/kanban_db.py:6906-6911 closes the run without a summary, and hermes_cli/kanban_db.py:8242-8275 exposes summaries to retries.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 15, 2026
GravityTone pushed a commit to GravityTone/hermes-agent that referenced this pull request Jul 18, 2026
Builds on timeout-salvage work from NousResearch#52511 and implements the preventive budget-warning contract from NousResearch#54153.

Co-authored-by: Daniel Maly <maly.daniel@protonmail.com>
garadice added a commit to garadice/hermes-agent that referenced this pull request Aug 25, 2026
…inst the parent task

HERMES_KANBAN_TASK stays set in os.environ for in-process children running
inside a dispatcher worker: delegate_task subagents, background-review
forks, and cron agents fired via the cronjob tool. Those children carry
their own much smaller iteration budgets (review-fork cap 16, delegation
cap 50), so when a child exhausted ITS budget,
_record_kanban_budget_exhausted recorded a terminal timed_out failure
against the WORKER's task — advancing the consecutive-failure circuit
breaker and archiving live parent work that was still running fine.

Gate both call sites on is_dispatcher_owned_worker_context() (the
documented single predicate for HERMES_KANBAN_* identity gates — False
for delegated children and non-dispatcher-owned cron execution, from
NousResearch#79657's context machinery) plus the review fork's _persist_disabled
persistence-isolation marker. Zero behavior change for genuine
dispatcher-owned workers.

Complementary to NousResearch#52511 (partial-summary salvage on exhaustion): that PR
forwards what the exhausted worker produced; this PR stops non-worker
exhaustion from being attributed to the worker at all.

Regression test: 5-case matrix — dispatcher-owned fires; delegated
child skips; review fork skips; non-dispatcher-owned cron skips; guard
truth table (mutation-checked: each guard leg independently covered).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cron Cron scheduler and job management P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(kanban): Kanban workers lose partial findings on iteration-budget exhaustion

3 participants