fix(agent): recover from orphaned tool_use blocks with a one-shot retry - #53236
Closed
fsaad1984 wants to merge 5 commits into
Closed
fix(agent): recover from orphaned tool_use blocks with a one-shot retry#53236fsaad1984 wants to merge 5 commits into
fsaad1984 wants to merge 5 commits into
Conversation
Anthropic returns HTTP 400 when a tool_use block is not immediately followed by its tool_result. Two root causes exist: 1. Context compression inserts messages between the pair. _strip_orphaned_tool_blocks (PR NousResearch#52145) already fixes the *wire payload*, but it mutates api_messages — a shallow copy of the canonical messages list. The canonical list is unchanged, so the *next* API call rebuilds the same broken payload and hits the same 400 again. 2. A cron/subagent session is interrupted before the tool_result is appended. Concrete reproduction: the approval guard blocks execute_code inside a cron job (no user present), the tool handler returns an error JSON which the tool_executor normally wraps in a tool_result message. But in this case the gateway reloaded the session transcript from disk AFTER the interruption, finding disk=0 messages vs memory=37. The live (correct) history was preserved, but a prior interrupted turn had left a bare tool_use as the last assistant block with no following user/tool_result turn. _strip_orphaned_tool_blocks never ran against the canonical list, so the next API call sent the broken transcript verbatim. Fix — three-file change: * agent/error_classifier.py: new FailoverReason.orphaned_tool_use + detection pattern in _classify_400. The Anthropic error message always contains both 'tool_use' and 'tool_result', which is distinctive enough for a safe substring match. retryable=True so the retry loop continues rather than aborting. * agent/turn_retry_state.py: orphaned_tool_use_retry_attempted flag so the recovery branch fires at most once per turn (prevents an infinite strip-and-retry loop if stripping somehow fails to fix the issue). * agent/conversation_loop.py: recovery branch that runs _strip_orphaned_tool_blocks against the canonical messages list (not just the wire payload) so the cleaned transcript is persisted and the retry sees a valid conversation. Reproduction: long gateway session → tool call → execute_code blocked by cron approval guard → gateway reload from disk finds stale/empty transcript → live history preserved but contains orphaned tool_use → HTTP 400 crash-loop.
alt-glitch
added
type/bug
…nd OpenAI-style canonical messages The canonical messages list uses OpenAI-style role=tool/tool_calls, not the Anthropic wire format that _strip_orphaned_tool_blocks expects. The original fix stripped 0 entries because it passed the wrong list. Now: (1) detect orphaned IDs from api_messages (Anthropic format), (2) strip api_messages via _strip_orphaned_tool_blocks, (3) also clean the canonical messages list by removing orphaned tool_calls entries and their matching role=tool messages so the next api_messages rebuild produces a valid transcript.
…ssages api_messages at error-handler time is pre-conversion; the Anthropic adapter converts tool_calls→tool_use internally. Detect orphaned IDs from canonical messages (role=tool / tool_calls) instead of api_messages.
…l messages The canonical messages pair IS present but adjacency breaks during Anthropic adapter conversion (context compaction injects synthetic user messages). Parse the IDs directly from the Anthropic 400 error string. Also fix: used 'classified_err' (undefined) instead of 'api_error'.
Contributor
|
Thanks for tracing the Anthropic adjacency failure and documenting the interrupted-session case. Automated hermes-sweeper review found this behavior already implemented on current
Closing as implemented on main. |
teknium1
added
the
sweeper:implemented-on-main
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Anthropic returns HTTP 400 when a
tool_useblock is not immediately followed by itstool_result. Two root causes exist:1. Context compression inserts messages between the pair.
_strip_orphaned_tool_blocks(PR #52145) already fixes the wire payload but mutatesapi_messages— a shallow copy of the canonical list. The canonical list is unchanged, so the next API call rebuilds the same broken payload.2. Interrupted cron/subagent sessions. Concrete reproduction: the approval guard blocks
execute_codeinside a cron job (no user present). The tool handler normally wraps the result in atool_resultmessage, but when the gateway reloads the transcript from disk after an interruption, the live history contains a baretool_useas the last assistant block with no followingtool_result. The HTTP 400 crash-loop follows.Fix
Three-file change:
agent/error_classifier.py: newFailoverReason.orphaned_tool_use+ detection in_classify_400. The Anthropic error message always contains bothtool_useandtool_result, which is distinctive.retryable=Trueso the retry loop continues.agent/turn_retry_state.py:orphaned_tool_use_retry_attemptedflag — fires at most once per turn to prevent an infinite strip-and-retry loop.agent/conversation_loop.py: recovery branch runs_strip_orphaned_tool_blocksagainst the canonicalmessageslist (not just the wire payload) so the cleaned transcript is persisted and the retry sees a valid conversation.Testing
_classify_400unit test with the exact Anthropic error message