fix(anthropic): preserve interleaved thinking/tool_use block order on replay (#17861) - #43943
Conversation
… replay
Interleaved-thinking turns (adaptive thinking, Claude 4.6+/Opus 4.8) emit
content blocks like:
thinking_1(signed) tool_use_1 thinking_2(signed) tool_use_2
Anthropic signs each thinking block against the turn content preceding it
at its position. normalize_response split the turn into two parallel lists
(reasoning_details + tool_calls), discarding cross-type order, and
_convert_assistant_message rebuilt it as [all thinking][text][all tool_use].
That moved thinking_2 ahead of tool_use_1, invalidating its signature, so
Anthropic rejected the latest assistant message with HTTP 400:
messages.N.content.M: `thinking` or `redacted_thinking` blocks in the
latest assistant message cannot be modified.
Observed repeatedly in agent.conversation_loop against api.anthropic.com /
claude-opus-4-8, recurring across sessions on multi-thinking-block turns.
Fix: carry a verbatim, order-preserving copy of the turn's content blocks
(anthropic_content_blocks) end-to-end - capture in normalize_response,
persist/restore through state.db, and replay unchanged for the latest
assistant message. Gated to turns that actually interleave signed thinking
with tool_use, so normal turns are unaffected.
Adds 3 regression tests including a SQLite round-trip covering the
crash-recovery reload path.
…ocks HTTP 400 "messages.N.content.M.text.parsed_output: Extra inputs are not permitted" on the native Anthropic transport. Anthropic SDK 0.87.0 response blocks carry output-only attributes the Messages *input* schema forbids: text blocks get `parsed_output` and `citations=None`, tool_use blocks get `caller`. normalize_response captured blocks verbatim via _to_plain_data and replayed them as request input on the next turn, so the forbidden fields leaked back -> 400. Like the earlier thinking-block bug, one poisoned turn wedges every subsequent request in the session (even the diagnostic turn), recoverable only by switching models or deleting the session. This is a defect in the anthropic_content_blocks channel added for the interleaved-thinking fix: it preserved block ORDER correctly but copied every SDK attribute, including output-only ones. Fix — whitelist input-permitted fields per block type at all three leak points: - agent/transports/anthropic.py normalize_response: sanitize at CAPTURE so the poison never persists to state.db (defence-in-depth). - agent/anthropic_adapter.py _sanitize_replay_block (new): whitelist used on the ordered-blocks replay path; also recovers already-poisoned stored sessions. - agent/anthropic_adapter.py _convert_content_part_to_anthropic: a stored `text` part is rebuilt from whitelisted fields instead of dict(part) verbatim (this was the exact content.N.text.parsed_output failure locus). Whitelist not blacklist, so future SDK output-only fields can't reintroduce it. Block order and thinking-block signatures are preserved (the reason the channel exists). Adds tests/agent/test_anthropic_output_field_leak.py; full adapter suite green (163 tests). Existing poisoned state.db rows scrubbed out-of-band.
…ay 400 recovery Two additive hardening changes on the interleaved-thinking replay path introduced by this PR's anthropic_content_blocks channel. Both are scoped to that channel's blast radius; neither changes correct behavior. 1. Replay-time tool-input re-sourcing (credential safety). The ordered-block channel captures each tool_use `input` from the RAW API response in normalize_response, which is NOT credential-redacted. The parallel tool_calls[].function.arguments IS redacted at storage time (build_assistant_message, #19798). The verbatim-replay fast path in _convert_assistant_message replayed the raw block input, so a secret a model inlined into a tool call (e.g. an Authorization header value passed inside a terminal command) would ride back onto the wire even though it is redacted everywhere else in history. Re-source tool_use input from the redacted tool_calls map by sanitized id; interleave order (the reason this channel exists) is unaffected. Adapted from #36071, which re-sources tool inputs the same way on its replay path. 2. Broaden the thinking-replay 400 classifier (defense-in-depth). error_classifier only matched "signature" + "thinking", so the frozen-block variant — "thinking ... blocks in the latest assistant message cannot be modified. These blocks must remain as they were in the original response." — carried no "signature" token and fell through to a non-retryable abort. The anthropic_content_blocks channel prevents the reorder that triggers this 400 at the source, but if any future mutator reintroduces it, the turn now self-heals via the existing strip-reasoning-and-retry recovery instead of crash-looping. A negative case ensures an unrelated "cannot be modified" 400 (no "thinking") is not swept in. Mirrors the classifier broadening in #36087 and #36071. Tests - tests/agent/test_anthropic_thinking_block_order.py: a replay test asserting an inlined secret is redacted on the wire while interleave order is preserved. - tests/agent/test_error_classifier.py: three cases — frozen-block 400 native and via OpenRouter route to thinking_signature/retryable; an unrelated "cannot be modified" 400 does not. Both grafts verified RED (tests fail with the change reverted) then GREEN. Full adapter, transport, classifier and output-field-leak suites pass. Co-authored-by: AlexanderBFoley <92330381+AlexanderBFoley@users.noreply.github.com>
…olumn) Drop the hermes_state.py column + persistence plumbing from the salvaged interleaved-thinking fix. The ordered-block channel covers the failure window in-memory (turn replayed within the live conversation loop). A session reloaded from disk after a crash falls back to reconstruction; if that replay 400s, the thinking-signature recovery (#43667) strips reasoning_details and retries — one degraded call in a rare resume path instead of a schema column. Replaces the DB-roundtrip test with a fallback-shape test.
🔎 Lint report:
|
| Rule | Count |
|---|---|
unresolved-import |
2 |
not-subscriptable |
1 |
First entries
tests/agent/test_anthropic_thinking_block_order.py:36: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/agent/test_anthropic_output_field_leak.py:16: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/agent/test_anthropic_output_field_leak.py:58: [not-subscriptable] not-subscriptable: Cannot subscript object of type `None` with no `__getitem__` method
✅ Fixed issues: none
Unchanged: 5566 pre-existing issues carried over.
Diagnostics are surfaced as warnings — this check never fails the build.
|
Verification review — reviewed the interleaved-thinking replay mechanism and credential-redaction re-sourcing. This is a well-structured defense-in-depth approach to a real Anthropic SDK bug (HTTP 400 on replay when output-only fields like Architecture:
Whitelist approach in Credential re-sourcing: The Activation guard: The ordered-blocks channel only activates when both signed thinking AND tool_use are present — the exact condition where parallel list reconstruction fails. Pure-text or thinking-then-tools turns use the existing code path untouched. CI: all 20 checks green. |
1 similar comment
|
Verification review — reviewed the interleaved-thinking replay mechanism and credential-redaction re-sourcing. This is a well-structured defense-in-depth approach to a real Anthropic SDK bug (HTTP 400 on replay when output-only fields like Architecture:
Whitelist approach in Credential re-sourcing: The Activation guard: The ordered-blocks channel only activates when both signed thinking AND tool_use are present — the exact condition where parallel list reconstruction fails. Pure-text or thinking-then-tools turns use the existing code path untouched. CI: all 20 checks green. |
3 participants
Summary
Interleaved-thinking turns (signed thinking blocks mixed with tool_use, Claude 4.6+/Opus agentic work) now replay to Anthropic in their original block order, eliminating the HTTP 400 "
thinking... blocks in the latest assistant message cannot be modified" at its source (salvages #35586, the structural fix for #17861).Root cause:
normalize_responsesplit the turn into parallelreasoning_details+tool_callslists, discarding cross-type order;_convert_assistant_messagerebuilt it as[all thinking][text][all tool_use]. Anthropic signs each thinking block against the content preceding it at its position, so the reorder invalidated signatures and 400'd.Changes
agent/transports/anthropic.py+types.py: capture a verbatim, order-preservinganthropic_content_blockschannel — gated to turns that actually interleave signed thinking with tool_use (@Spaceman-Spiffy)agent/anthropic_adapter.py: replay the ordered blocks unchanged for the latest assistant turn; new_sanitize_replay_blockwhitelist strips SDK output-only fields (parsed_output,citations=None,caller) that the Messages input schema rejects with "Extra inputs are not permitted" (@Spaceman-Spiffy)agent/anthropic_adapter.py: tool_useinputre-sourced from the redactedtool_callsmap so raw un-redacted secrets never replay onto the wire (read_file returns line-numbered content that pollutes files when written back via write_file #19798 contract) (@Spaceman-Spiffy)agent/chat_completion_helpers.py: carry the channel onto the stored message dict (@Spaceman-Spiffy)state.dbcolumn + persistence plumbing is dropped — the channel is in-memory only. A session reloaded from disk after a crash falls back to reconstruction; if that replay 400s, the thinking-signature recovery (fix(agent): classify and actually recover the 'thinking blocks cannot be modified' 400 (#17861) #43667) strips and retries. One degraded call in a rare resume path instead of a schema column.scripts/release.py: AUTHOR_MAP entry (commits authored under the contributor's renamed account)error_classifier.pybroadening was dropped during cherry-pick — byte-equivalent to fix(agent): route 'thinking blocks cannot be modified' 400 to recovery #36087, already on main via fix(agent): classify and actually recover the 'thinking blocks cannot be modified' 400 (#17861) #43667.Validation
[thinking, text, tool_use, tool_use]), multi-turn replay clean, completed=TrueSupersedes #20997 (same idea, but replays unsanitized SDK dicts → "Extra inputs are not permitted" 400, and keeps thinking on all turns) and #26959 (verbatim
passwould revert the orphan-strip demotion guard from 64628ea and break unsigned-block handling on Kimi-synthesized paths).Infographic