[MikeTuev]

Summary

This broke prompt caching for Claude Code (claude CLI) sessions routed through a plain anthropic-compatible-* connection. Every turn re-wrote almost the whole conversation to the cache instead of reading it.

What the client sends

During a session, Claude Code appends system-role turns to messages[], which come after the first user turn. They are not part of the top-level system. Examples from real traffic:

{ "role": "system", "content": "<total_tokens>14967640 tokens left</total_tokens>" }
{ "role": "system", "content": "The TodoWrite tool hasn't been used recently. If you're working on tasks that would benefit from tracking progress, consider using the TodoWrite tool ..." }
{ "role": "system", "content": "… While auto mode is active …" }

(The last two are abbreviated.)

The <total_tokens>…</total_tokens> budget notice is appended as a new system turn after each tool round, and older ones stay in the history. A 2-message request carries 1 such turn, an 89-message request carries 30, and so on.

What OmniRoute did with it

For any provider other than native claude, the claude → claude passthrough ran normalizeClaudeUpstreamMessages() → extractSystemRoleMessages(). That function moves every role: "system" turn out of messages[] and appends its text to the top-level system array:

client sends                          OmniRoute sent upstream
system: [BASE]                        system: [BASE, env, <total_tokens>#1, <total_tokens>#2, … , <total_tokens>#N]
messages: [user, system(env),         messages: [user, assistant, user, assistant, user, …]   (system turns removed)
           assistant, user,
           system(<total_tokens>#1),
           …]

system comes before messages, so each turn inserted one more block ahead of the entire conversation history. Anthropic prompt caching matches prefixes. The cached history therefore never matched again, and only the stable head of the prompt (tools + original system) was reused.

Measured impact (one real claude CLI session, claude-opus-5-5, single connection, no account rotation, gaps < 4 min)

	total input	cache read	cache write
turn N‑1	133,153	27,890	105,261
turn N	133,704	27,911	105,791

• Across 54 consecutive calls, the cache-hit rate was ~29.5%. In 45 of 53 transitions cache read grew by exactly +21 tokens, which is roughly one more <total_tokens> block in the cached prefix.
• For comparison, a call from an OpenCode client on the same connection and model, whose body has no mid-conversation system turns, hit ~99% (105,003 of 105,989 tokens read from cache).

Fix

Plain anthropic-compatible-* passthrough no longer hoists mid-conversation system turns:

• Only the leading run of system turns before the first real user/assistant turn is hoisted (hoistLeadingTextSystemMessages). Anthropic rejects a text system turn at messages[0]. Directive-only turns (content: [] + output_config) are moved off messages[0] (relocateDirectiveOnlyMessages). Every later system turn keeps its position, so system stays byte-stable and history is only appended.
• This applies regardless of the client's anthropic-beta, so cache stability does not depend on the exact beta set a client version negotiates.
• mid-conversation-system-2026-04-07 is forwarded when the client sent it and a mid-conversation system turn survived into the body. Before this change, the generic beta allowlist dropped it. OmniRoute never adds this beta on its own.
• A body carrying an OpenAI developer turn keeps the full legacy hoist. Accepting system turns does not imply accepting developer turns.
• The global system prompt (injectSystemPromptPostTranslation) on claude targets now always goes to body.system. Before, once a system turn was present in messages[], the suffix was written onto the last system turn, which moves every turn and would break the prefix in the same way.

New helper: open-sse/services/claudeMidConversationSystem.ts. Call sites: the claude passthrough branch in chatCore.ts (non-semantic path) and DefaultExecutor.buildHeaders().

Why native claude and the CC bridge are not changed here

• Native claude already keeps these turns for Opus/Fable full-agent requests (shouldUseMidConversationSystem, fix(sse): preserve Claude Code cache breakpoints #8934 / fix(sse): preserve Fable mid-conversation cache prefixes #13173, together with the beta from selectBetaFlags). For other models (e.g. Sonnet), it still hoists them. Anthropic validates the position of these turns per model ([BUG] Opus 5 rejects cache-safe memory system message after assistant text #11290, fix(claude): 3.8.50 Claude Code passthrough — mid-conversation system output_config 400 (claude-opus), web_search interception sends OpenAI tool shape to Claude #15118), so widening that gate needs live verification per model family. That belongs in a separate PR.
• CC bridge (anthropic-compatible-cc-*) builds its own body (prepareClaudeCodeCompatibleSemanticBody / extractCustomSystemBlocks) and hoists there as well. It has the same cache problem in principle. However, it sits behind the CC wire image/signing and relay-specific beta handling, and that path is out of scope for this change.

Both are candidates for follow-ups. I can open issues for them if useful.

Related Issues

• Refs fix(backend): Claude-native path hoists mid-conversation system messages into system[] and duplicates them #9506, fix(providers): Claude system-role hoisting moves cache_control out of messages causing history-cache misses #9436 — the same class of issue, fixed for the native claude provider only (fix(sse): preserve Claude Code cache breakpoints #8934). anthropic-compatible-* kept the unconditional hoist.

Target Branch

• Base is the active release/v3.8.52 branch, not main

Proof Of Fix (required)

• TDD — the new test fails on release/v3.8.52 without the fix and passes with it.
  • Test file: tests/unit/claude-mid-conversation-system-compatible-passthrough.test.ts (11 tests). The test drives handleChatCore() with a synthetic Claude Code-shaped body (env system turn + one <total_tokens> system turn per tool round, thinking blocks, tool_use/tool_result, 1h cache markers) and asserts on the final body and headers handed to fetch, not on the helper alone.
  • Failing output before the fix (7 of 11 fail):

    ✖ default: mid-conversation system turns stay in place in the final body
    ✖ default: successive turns keep system stable and earlier messages a prefix
      AssertionError [ERR_ASSERTION]: system changed between turn 1 and 2
    ✖ a non-Claude Code client gets the same treatment and its beta is forwarded
    ✖ client without the beta: turns still stay in place, and OmniRoute never invents it
    ✖ client without any anthropic-beta header: turns stay in place, no beta header invented
    ✖ successive turns from a client without the beta also keep earlier messages a prefix
    ✖ leading text system turn is hoisted, directive-only one relocated
    ℹ pass 4
    ℹ fail 7
    

  • After the fix: 11/11 pass. Added a case to tests/unit/global-prompt-single-injection.test.ts for the global prompt with a mid-conversation system turn.

Scope

• Only the files the failing tests prove need changing are touched
• No test was weakened or deleted

Validation

• Change type: provider / routing
• Focused tests: Claude / chatCore / executor / system-prompt / combo slice — 3,495 tests, 3,480 pass, 7 skipped, 8 fail. All 8 are in combos-auto-skipped-variants.test.ts (6) and combo-test-auto-route.test.ts (2) and fail the same way on clean release/v3.8.52 (5 pass / 8 fail there).
• npm run typecheck:core clean; ESLint clean on changed files; pre-commit hooks pass

Reviewer Notes

• Behaviour change for third-party relays: a relay that rejects role: "system" inside messages[] will now return 400 for bodies that carry such turns. Before, these were silently flattened into system at the cost of the cache. Native claude already sends these turns to Anthropic for Opus/Fable, but I have not verified every third-party relay.
• Live before/after cache numbers against a real relay are not included yet. I will add them here once this build has run on a self-hosted instance.

[diegosouzapw]

Hi @MikeTuev — closing this pull request to clear the open queue. The branch fix/claude-compatible-mid-conversation-system is kept, so the commits stay. Comment here if it should be reopened.

[diegosouzapw]

Hi @MikeTuev — reopening this pull request. The close was a mistake, and it is back in the merge queue. The branch fix/claude-compatible-mid-conversation-system was kept.