fix(agent): strip api_messages in thinking-signature recovery so the retry actually omits thinking blocks

[0xyg3n]

Summary

The thinking-signature recovery in agent/conversation_loop.py popped reasoning_details from messages and continued to retry. That had two defects, only one of which was the visible one.

The visible defect: the pop mutated the canonical conversation list. messages is the same list _persist_session writes to state.db and the session transcript, so a single recovery permanently wiped every signed thinking block from the stored conversation. Subsequent turns reloaded the stripped state, hit the same HTTP 400 ("invalid signature" or "cannot be modified", see #24107), and the agent stopped responding on that chat. Cascading compaction-ended sessions chained off the corrupted parent.

The hidden defect: the strip never reached the wire payload either. api_messages is built once at the start of the turn by shallow-copying every entry in messages (line 919 area). Each entry in api_messages has its own reference to the same reasoning_details list. When build_api_kwargs runs on every retry inside the inner while-loop, it consumes api_messages, not messages. Popping reasoning_details from messages left api_messages untouched, so the retry's request still carried the same thinking blocks Anthropic had just rejected. The classifier had already latched thinking_sig_retry_attempted = True after the first attempt, so the recovery branch did not fire a second time, and the loop terminated with max_retries_exhausted on the same 400.

This PR moves the strip onto api_messages. messages is no longer touched, so disk I/O stays clean, and the strip actually reaches the wire because api_messages is what build_api_kwargs consumes on every retry.

Root cause

agent/conversation_loop.py, current behavior:

for _m in messages:
    if isinstance(_m, dict):
        _m.pop("reasoning_details", None)

agent/conversation_loop.py, message-list construction earlier in run_conversation:

api_messages = []
for idx, msg in enumerate(messages):
    api_msg = msg.copy()
    ...
    api_messages.append(api_msg)

.copy() is a shallow dict copy. Top-level keys are independent across the two dicts, so a pop on one does not affect the other. The values, including the reasoning_details list, are shared by reference. After the loop, messages[i] and api_messages[i] are two dicts each carrying their own key "reasoning_details" pointing at the same list object.

agent/conversation_loop.py, retry loop:

while retry_count < max_retries:
    ...
    api_kwargs = agent._build_api_kwargs(api_messages)
    ...
    response = agent._interruptible_api_call(api_kwargs)

api_kwargs is rebuilt from api_messages on every iteration. The strip needs to reach api_messages to affect the next call.

Fix

agent/conversation_loop.py:

thinking_sig_retry_attempted = True
_api_stripped = 0
for _m in api_messages:
    if isinstance(_m, dict) and "reasoning_details" in _m:
        _m.pop("reasoning_details", None)
        _api_stripped += 1

messages is not touched. _persist_session writes intact reasoning_details on every persist boundary. The strip propagates into api_kwargs on the next iteration through _build_api_kwargs(api_messages), so the retry's request goes out without thinking blocks.

Versions observed

Reproduced against the native Anthropic Messages API on claude-opus-4-7 and claude-opus-4-8 with the interleaved-thinking-2025-05-14 beta enabled (Hermes sends this by default for Claude 4.x).

• hermes-agent 0.12.0 (editable install)
• hermes-agent 0.14.0 (editable install, including a clean vanilla checkout with no other patches)

The trigger surface is independent of the local hermes version: any deployment that sends signed thinking blocks against the interleaved-thinking beta and replays them on multi-turn tool-use conversations is exposed once FailoverReason.thinking_signature fires for any reason.

Reproduction signature in state.db

Affected sessions show a clean before/after split in the messages table: assistant rows up to the recovery point carry intact reasoning_details with signatures; every assistant row from that point on has reasoning_details = NULL even though the model returned a thinking block. Counting signed vs tool_calls without signed reasoning_details per session is a reliable detector for the corruption signature this PR prevents.

Tests

tests/run_agent/test_thinking_sig_recovery_persistence.py covers the mutation surface in isolation:

• test_pop_on_shallow_copy_does_not_affect_source - the invariant the recovery relies on.
• test_strip_api_messages_leaves_canonical_messages_intact - the recovery loop mirrored against shallow-copied messages.
• test_strip_is_idempotent_when_run_twice - duplicate firing is safe.
• test_strip_skips_messages_without_reasoning_details - non-applicable messages are untouched.

Regression sweep:

pytest tests/run_agent/test_thinking_sig_recovery_persistence.py   4 passed
pytest tests/agent/test_error_classifier.py                      155 passed
pytest tests/run_agent/test_run_agent.py                         353 passed
pytest tests/run_agent/test_empty_response_recovery_persistence.py 3 passed

Compatibility

• _build_api_kwargs signature is unchanged.
• _persist_session signature and behaviour are unchanged.
• The outbound retry payload now matches the recovery's documented intent (no thinking blocks on the wire). The previous behaviour shipped them by accident.
• Sessions whose messages table was already wiped by the previous behaviour are not retroactively repaired. Such sessions continue to fail on replay until they are trimmed or the user starts a fresh chat. PR fix(anthropic): preserve thinking blocks on prior tool_use turns (interleaved-thinking contract) #24107 reduces the rate at which fresh sessions reach this state.

Related

• fix(anthropic): preserve thinking blocks on prior tool_use turns (interleaved-thinking contract) #24107 - preserve thinking blocks on prior tool_use turns (interleaved-thinking contract). Narrows the recovery's trigger surface. Complementary to this PR.
• fix(agent): preserve thinking blocks in latest assistant message verbatim #26959 - preserve thinking blocks in latest assistant message verbatim.
• [Bug] Multi-turn history loses thinking/redacted_thinking blocks -- raw Anthropic content array not preserved as source of truth #17861 - multi-turn history loses thinking/redacted_thinking blocks.

[teknium1]

Merged via PR #43667 — your commit was cherry-picked onto current main with your authorship preserved in git log (9f95f72). One adjustment during salvage: main had moved the retry flag into TurnRetryState, so the resolved version uses _retry.thinking_sig_retry_attempted with your api_messages strip logic intact. Thanks!