fix(steer): preserve late steer after output truncation

[frizikk]

What does this PR do?

Preserves /steer messages across terminal turn boundaries instead of losing them in the race between ingress and result finalization.

A cached agent now opens a distinct steer-acceptance generation for each run_conversation() call. Terminal result creation seals that generation and drains its pending steer under the same lock used by AIAgent.steer(). This gives the race two lossless outcomes: a steer that wins the lock is returned in result["pending_steer"]; a steer that arrives after sealing is rejected by the completed turn so its ingress can preserve it as next-turn work.

The contract is complete across the classic CLI, TUI RPC, and messaging gateway:

• explicit CLI /steer rejection appends the non-empty payload behind older _pending_input FIFO work;
• TUI session.steer rejection queues the payload for the next turn without hard-interrupting the completed run;
• the messaging gateway consumes accepted terminal steers exactly once and places them behind older transport/FIFO work;
• adapter-local active/pending slot keys remain distinct from profile-qualified durable session-state keys;
• command-shaped accepted steer text (for example /stop) retains steer provenance and runs as user input rather than being redispatched as a command;
• recursion-depth fallback restores the current FIFO head without displacing the terminal steer behind it.

A live overlap search found open PR #67226, but it addresses the TUI busy_input_mode="interrupt" ingress path and bundles unrelated webhook/session changes. It does not implement this terminal-result ownership contract.

The refreshed candidate is the two-commit series headed by b0e128e85f39890737b912a1da42e862039b393d (tree e9aadab974bbd893c560d8fca7b8e8a86fc2a8a9), based directly on frozen upstream main 936dd7346fd7fd8107af1ce7fc019c07c001c1bd on 2026-08-11. The atomic follow-up b0e128e85f39890737b912a1da42e862039b393d repairs the CI fixtures without changing production code.

Hard-interrupt returns remain intentionally excluded because clear_interrupt() discards pending steer by design. codex_app_server also remains excluded because it owns its loop and uses native turn/steer.

Related Issue

Fixes #59084

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• agent/agent_init.py, run_agent.py
  • add per-turn steer-acceptance generations;
  • atomically open, seal, and drain steer acceptance under _pending_steer_lock;
  • reject non-empty steers that arrive after the active turn has sealed.
• agent/conversation_loop.py, agent/turn_finalizer.py
  • route all non-interrupt terminal results through the shared seal-and-drain contract;
  • retain deliberate ownership for hard interrupts, native Codex app-server turns, and normal finalization.
• cli.py, tui_gateway/methods_session.py
  • preserve rejected explicit /steer and session.steer payloads as next-turn work;
  • preserve older queue order and avoid unnecessary hard interrupts.
• gateway/platforms/base.py, gateway/run.py, gateway/slash_commands.py
  • separate adapter-local physical slot keys from durable/profile-qualified session keys;
  • preserve terminal steer FIFO order, exact-once delivery, command-shaped steer provenance, recursion-cap recovery, goal continuation cleanup, and queue-depth reporting across both key domains.
• Tests
  • cover terminal lock ordering, accepted and rejected ingress, provider terminal branches, command-shaped steer text, older FIFO work, named-profile key separation, recursion-depth fallback, queue accounting, and no-interrupt behavior at production seams;
  • align direct tool-batch and gateway fixtures with the production ownership seams: direct _execute_tool_calls() tests explicitly open a steer generation, while adapter-owned active/pending slots use the adapter's physical key and durable runner state keeps its profile-qualified key;
  • make the named-profile routing regression independent of profiles configured on the test host.

How to Test

1. Re-run the three files that failed in CI run 31522209927:

   env -u PYTHONPATH -u VIRTUAL_ENV \
     HERMES_PYTHON=/home/jakub/.cache/hermes-pr-audit-venv/bin/python \
     scripts/run_tests.sh \
       tests/gateway/test_multiplex_busy_input_mode.py \
       tests/gateway/test_telegram_voice_v0_regressions.py \
       tests/run_agent/test_tool_batch_segmentation.py \
       -q

   Exact-tree result: 50 passed, 0 failed, 1 Windows-only skip. The repaired fixtures now exercise the same generation lifecycle and dual-key ownership boundaries as production.

2. Run those three files together with the prior terminal-steer, gateway FIFO, CLI/TUI ingress, provider-exit, and conflict-side regression lanes:

   env -u PYTHONPATH -u VIRTUAL_ENV \
     HERMES_PYTHON=/home/jakub/.cache/hermes-pr-audit-venv/bin/python \
     scripts/run_tests.sh \
       tests/gateway/test_multiplex_busy_input_mode.py \
       tests/gateway/test_telegram_voice_v0_regressions.py \
       tests/run_agent/test_tool_batch_segmentation.py \
       tests/run_agent/test_413_compression.py \
       tests/run_agent/test_partial_stream_finish_reason.py \
       tests/run_agent/test_run_agent.py \
       tests/run_agent/test_run_agent_codex_responses.py \
       tests/run_agent/test_steer.py \
       tests/gateway/test_terminal_steer_fifo.py \
       tests/gateway/test_queue_consumption.py \
       tests/gateway/test_steer_fifo_overwrite.py \
       tests/gateway/test_busy_session_ack.py \
       tests/gateway/test_tts_media_routing.py \
       tests/gateway/test_pending_event_none.py \
       tests/gateway/test_goal_continuation_drain.py \
       tests/gateway/test_goal_max_turns_config.py \
       tests/gateway/test_goal_status_notice.py \
       tests/gateway/test_active_session_text_merge.py \
       tests/gateway/test_run_progress_topics.py \
       tests/gateway/test_busy_session_auth_bypass.py \
       tests/cli/test_cli_init.py \
       tests/test_tui_gateway_server.py \
       tests/agent/test_turn_retry_state.py \
       tests/hermes_cli/test_copilot_token_exchange.py \
       -q

   Exact-tree result: 1071 passed, 0 failed, 2 Windows-only skips across 24 files.

3. Parse run_conversation() and classify every direct return.
   Result: 32 direct returns — 26 shared terminal-helper exits, 4 deliberate hard-interrupt exits, 1 native codex_app_server exit, 1 normal finalizer exit, and 0 unexplained.

4. Run static, diff, and attribution hygiene:

   env -u PYTHONPATH -u VIRTUAL_ENV \
     /home/jakub/.cache/hermes-pr-audit-venv/bin/python -m ruff check \
       tests/gateway/test_multiplex_busy_input_mode.py \
       tests/gateway/test_telegram_voice_v0_regressions.py \
       tests/run_agent/test_tool_batch_segmentation.py
   /home/jakub/.cache/hermes-pr-audit-venv/bin/python -m compileall -q \
     tests/gateway/test_multiplex_busy_input_mode.py \
     tests/gateway/test_telegram_voice_v0_regressions.py \
     tests/run_agent/test_tool_batch_segmentation.py
   git diff --check 936dd7346fd7fd8107af1ce7fc019c07c001c1bd
   scripts/audit_pr_attribution.py

   Exact-tree result: Ruff passed; compileall passed; diff-check passed; all contributor emails mapped.

5. Run the Windows-footgun diff scan. It reports two bare read_text() calls in tests/run_agent/test_run_agent_codex_responses.py; both lines already exist unchanged on exact upstream base 936dd7346fd7fd8107af1ce7fc019c07c001c1bd, and this candidate introduces neither match.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: Linux

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — N/A; no user-facing configuration or API changes
• I've updated cli-config.yaml.example if I added/changed config keys — N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — platform-neutral Python locking, queue, and control flow
• I've updated tool descriptions/schemas if I changed tool behavior — N/A

Screenshots / Logs

Not applicable. This is a control-flow and queue-ownership fix with deterministic automated regression coverage.

Caveats

• The full repository-wide pytest tests/ -q suite was not rerun. The exact committed tree passed the three formerly red CI files (50 passed) and the combined 24-file focused lane (1071 passed), plus Ruff, compileall, AST return classification, diff hygiene, and attribution audit.
• CI run 31522209927 was against parent 5345f8b434efd99c284870609509c7557902c327. Its 10 failures were stale direct-seam fixtures now repaired by follow-up b0e128e85f39890737b912a1da42e862039b393d; remote CI for that follow-up awaits a push.
• Hard interrupts intentionally clear pending steers, and codex_app_server continues to use its native steering protocol.

---

2026-08-21 current-main refresh

This PR remains necessary: current main does not preserve terminal /steer ownership across the reported truncation/finalization race. The series was rebased onto exact upstream main fc9cbc872d8050c22f1192b16bc5ff4aed471e10; the refreshed candidate is e426eb5e460ea9a262bcbf64feedd6bf68edc8f2. The port preserves current-main profile-scoped adapter busy lanes and makes every producer and consumer derive the adapter slot through the same profile-aware key helper. Final review follow-ups also route /goal resume through the transport-owning profile adapter and its physical pending slot, instead of the default adapter namespace.

Verified locally on the exact candidate: the 18 changed test files (611 passed, two Windows skips), the focused goal-resume regression (5 passed), Ruff across all changed Python files, compileall, attribution audit, and git diff --check. The same lane on the parent before the final focused follow-up passed 1192 tests with two Windows skips. Pytest reports one existing non-fatal _BarrierDB.flush_token_counts thread warning from a test already present unchanged on the pinned base.

[tonydwb]

Code Review Summary

Looks good! No obvious issues found.

---

Reviewed by Hermes Agent

[tonydwb]

Code Review Summary\n\nLooks good! No obvious TODO/FIXME comments found.\n\n---\nReviewed by Hermes Agent

[teknium1]

Thanks for the focused regression fix. Current main's output-limit terminal returns bypass the TurnFinalizer drain at agent/conversation_loop.py:1880, 1991, 2057, 2074, 2087, 3629, 4465, 4574, and 4713, while agent/turn_finalizer.py:475-480 establishes the pending_steer result contract consumed by cli.py:12862-12868 and gateway/run.py:20262-20270.

The shared helper restores that contract across the targeted paths without mutating conversation history, toolsets, or the system prompt. The added tests cover representative text truncation, truncated tool calls, Codex incomplete-output exhaustion, and provider output-cap rejection. The complete patch also passed git apply --check against the inspected checkout.

Automated hermes-sweeper review.

[frizikk]

Prepared a provenance-only refresh of the exact two-commit series. The previous head dac7d53b856946cf78e5acfc0cd0415e1457ae5c maps to 0c02e0d3748be3440bd695cb40fb88d94a854578 (41308fe5923850dea0c82252ba78903b8377cd6d → 1082eb0b275c50d5cee518728cda0d73fcba1dcf; dac7d53b856946cf78e5acfc0cd0415e1457ae5c → 0c02e0d3748be3440bd695cb40fb88d94a854578). Only the committer name/email and the resulting second-commit parent OID changed; trees, author headers, message bytes, committer timestamps/timezones, order, and patches are identical. range-diff is = for both commits and the old→new head diff is empty. Exact-head verification: 300 focused tests passed, the 9-test steer contract canary passed, Ruff passed, and git diff --check passed.

[frizikk]

Superseding my earlier provenance-only refresh note: PR #63764 is now rebased and conflict-free at exact head d8194b2f85750ab0c05b77b446ad539628e391d7 (tree 890aeaea375bd1394d22ff4a586b7089047e233f).

The only textual conflict was the helper insertion point in agent/conversation_loop.py; the resolution preserves both current upstream Copilot credential-recovery helpers and this PR's _terminal_result_with_pending_steer contract. Final freshness preflight against main 40e0e7ad56f7faac24c757b11d3ef6f0f9b83de4 found zero overlap on the five candidate paths and a clean merge-tree.

Verification on the published SHA:

• focused four-file lane: 308 passed;
• upstream conflict-side Copilot lane: 14 passed;
• steer contract canary: 9 passed;
• exact-base overlay and helper-disabled mutation: both failed all 5 expected regressions; restored candidate passed all 5;
• Ruff, Windows-footgun CI, contributor attribution, supply-chain checks, Docker builds, and all Python CI slices passed;
• independent exact-SHA Spec and Standards audits passed.

GitHub final state: MERGEABLE / CLEAN, with 27 successful, 10 skipped, 1 neutral, 0 pending, and 0 failed checks. No unresolved review threads.

[GottZ]

This was generated by AI during triage.

Summary

Fourteen PRs address or reference four related failure-accounting and truncation/stream-recovery issues: #38460/#38553 account for truncated-response usage, #38479/#38553/#39115 refund optimistic API-call counts, #45919/#45920/#45922/#45971/#46003/#46016 repair Anthropic partial-stream recovery, and #63764 preserves late steering across terminal exits. #47280/#68041/#73028 address the adjacent empty-stub persistence failure, with #73028 providing the merged implementation.

Related pull requests

• fix(conversation): count tokens from length-truncated responses #38460 best fix — (+64/-0) — n/a: Counts usage from the truncated leg before length continuation and tests in-memory totals; the keep_open review requires salvaging this through the current canonical accounting path with persisted SessionDB coverage.
• fix(agent): refund api_call_count when all retries are exhausted #38479 best fix — (+71/-0) — n/a: Narrowly refunds api_call_count on max-retry exhaustion and adds a zero-success regression; consistent with the keep_open verdict, although the latest contributor note corrects the bot's broader-path description because the current head covers only this reported terminal path.
• fix(agent): track tokens on truncated responses and refund api_call_count on retry exhaustion #38553 best fix — (+210/-0) — n/a: Combines truncated-response accounting with refunds on several no-success exits, but its keep_open review identifies incomplete parity with canonical accounting and requests a current-main rework with both regression classes.
• fix(agent): refund api_call_count when all retries are exhausted #39115 [closed] partial — (+162/-0) — n/a: Closed duplicate of the fix(agent): refund api_call_count when all retries are exhausted #38479 retry-exhaustion refund, with broader tests for zero successes and a prior successful turn, so it remains useful as regression-reference material.
• fix(streaming): return Anthropic-shaped partial stubs #45919 best fix — (+177/-14) — n/a: Extracts provider-shaped partial-stream stub construction and verifies Anthropic validation plus bounded continuation; the keep_open verdict calls for rebasing while preserving newer content-filter and stale-breaker logic.
• fix(agent): return Anthropic-shaped partial stream stub in anthropic_messages mode #45920 [closed] fixes — (+84/-0) — n/a: Adds the same Anthropic content-list/max_tokens stub inline with shape coverage; closed as duplicate of the earlier, more comprehensively tested fix(streaming): return Anthropic-shaped partial stubs #45919.
• fix(agent): shape partial-stream recovery stub for anthropic_messages mode #45922 best fix — (+246/-24) — n/a: Builds an Anthropic-shaped stub while preserving shared post-processing and deliberately uses content=[]/end_turn when no text was recovered; despite the earlier duplicate label, the later contributor correction and diff establish a real empty-recovery policy difference from fix(streaming): return Anthropic-shaped partial stubs #45919, consistent with its keep_open review.
• fix(agent): return anthropic partial stream stubs #45971 [closed] fixes — (+50/-1) — n/a: Returns an Anthropic Message-shaped max_tokens stub after partial delivery and adds an IndexError regression; closed as duplicate of fix(streaming): return Anthropic-shaped partial stubs #45919.
• fix(agent): shape partial-stream stub to match active api_mode #46003 [closed] fixes — (+319/-0) — n/a: Adds Anthropic-shaped recovery plus explicit IndexError stream-parse classification and broad tests; closed as duplicate of fix(streaming): return Anthropic-shaped partial stubs #45919, but its classification tests remain relevant reference coverage.
• fix(streaming): catch IndexError from malformed upstream streams (#45908) #46016 [closed] partial — (+14/-1) — n/a: Augments the recovery stub with Anthropic content blocks while retaining its OpenAI fields; closed in the fix(streaming): return Anthropic-shaped partial stubs #45919 duplicate cluster.
• fix(agent): skip empty partial-stream assistant turns #47280 [closed] related — (+148/-10) — n/a: Skips empty partial-stream assistant turns and safely coalesces continuation prompts with string or multimodal user content; closed after this layer was implemented on main through fix: prevent session poisoning from empty partial-stream-stub assistant turns #73028 at commit 309f06b.
• fix(steer): preserve late steer after output truncation #63764 best fix — (+228/-55) — n/a: Adds one shared terminal-result helper that drains pending steer across output-truncation and other early-return paths, with representative text, tool-call, Codex, invalid-response, and output-cap regressions; this matches the visible keep_open/high-salvage verdict.
• fix: prevent session poisoning from empty partial-stream-stub assistant turns #68041 [closed] related — (+375/-6) — n/a: Adds loop, builder, and send-boundary defenses against persisted empty assistant turns with extensive regressions; closed because its commits were salvaged into fix: prevent session poisoning from empty partial-stream-stub assistant turns #73028.
• fix: prevent session poisoning from empty partial-stream-stub assistant turns #73028 [merged] related — (+437/-6) — n/a: Merged reference implementation of fix: prevent session poisoning from empty partial-stream-stub assistant turns #68041/fix(agent): skip empty partial-stream assistant turns #47280's empty-stub defenses, including current-main multimodal and codex_responses adaptations, and remains relevant as the implemented adjacent truncation fix.

Duplicates

#39115 substantially duplicates #38479; #38553 overlaps both #38479 and #38460 but bundles the two accounting causes. #45920 and #45971 duplicate #45919, while #46003 and #46016 overlap its core fix; #45922 shares that core but differs on empty-recovery semantics. #47280 and #68041 were superseded by merged #73028.

Suggested consolidation

Keep #63764 open with a salvage path: preserve its shared pending-steer terminal-result contract and focused regressions, consistent with the visible keep_open/high-salvage verdict. For the other clusters, retain #38479's narrow refund path, require #38460/#38553 author action to rebase or split canonical persisted accounting, and keep #45919/#45922 open only long enough to resolve the documented empty-recovery policy while rebasing the chosen implementation around current shared post-processing; the already closed duplicate and superseded PRs can remain closed along the chains above.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I59084(["issue #59084 (closed)"])
    P63764["PR #63764 (open)"]
    P63764 -->|best fix| I59084
    class I59084 closed
    class P63764 open
    class P63764 best
    class P63764 target
    click I59084 "https://github.com/NousResearch/hermes-agent/issues/59084"
    click P63764 "https://github.com/NousResearch/hermes-agent/pull/63764"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 14 pull requests and 4 issues in this complex. Each diff was read against this issue; Assessment working set: 167 kB of PR diffs, 56 kB of issue/PR text, 26 kB of discussion (38 comments), 21 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[frizikk]

Refreshed onto current main and resolved the terminal-steer/FIFO ownership conflicts across CLI, TUI, and gateway. The exact candidate and verification evidence are in the updated PR body.

[frizikk]

CI follow-up pushed as b0e128e85f39890737b912a1da42e862039b393d. It repairs the stale direct-seam fixtures without production changes: adapter-owned slots use physical keys, durable runner state keeps profile-qualified keys, direct tool-batch tests open the steer generation, and named-profile routing is host-independent. Exact head: 50/50 formerly-red tests and 1,071/1,071 focused tests passed; Ruff, compileall, diff hygiene, attribution, and synthetic merge against current main pass.