fix(sse): make anthropic-compatible mid-conversation system turns a per-connection opt-in

changelog.d/fixes/18341-anthropic-compatible-keep-mid-conversation-system-per-connection.md

@@ -0,0 +1 @@
+- **fix(sse):** the in-place mid-conversation system turns for plain `anthropic-compatible-*` relays are now a per-connection setting (`providerSpecificData.keepMidConversationSystem`, toggle in the edit-connection modal) instead of the global `ANTHROPIC_COMPATIBLE_MID_CONVERSATION_SYSTEM` feature flag, and no longer require the client to send the `mid-conversation-system-2026-04-07` beta — Claude Code builds that keep such turns without negotiating the beta were hoisted again and re-wrote ~110K tokens of cache on every turn. The beta is still forwarded when the client sends it, never invented

docs/reference/FEATURE_FLAGS.md

@@ -46,7 +46,7 @@ A boolean flag is considered **enabled** when its effective value is `"true"`,
 
 ## Flag Catalog
 
-116 flags across 6 categories. **Default** is the definition default — the value
+115 flags across 6 categories. **Default** is the definition default — the value
 used when neither a DB override nor an environment variable is present.
 
 ### Security (11)
@@ -120,7 +120,7 @@ Invalid enum values and unreadable feature-flag storage fail closed. Neither opt
 priced spend or a failed pricing lookup. Enforcement and the administrator report resolve the
 same effective policy.
 
-### Runtime (56)
+### Runtime (55)
 
 | Key                                            | Type    | Default | Restart | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
 | ---------------------------------------------- | ------- | ------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -167,7 +167,6 @@ same effective policy.
 | `KIRO_TOOL_CALL_REPAIR_ENABLED`                | boolean | `true`  |         | When a streaming Kiro response emits a malformed `tool_call` wrapper (missing `input.name` or `input.arguments`) before any visible output, suppress it and re-send the request once with a corrective instruction instead of ending the turn with `invalid_kiro_tool_call`. The retry costs one extra Kiro request; a second failure ends the turn with `kiro_tool_call_repair_retry_failed`. On by default. Set to `false` to fail the turn on the first malformed wrapper.                                                                                                                                                                   |
 | `CATALOG_MIRROR_REFINEMENTS_ENABLED`           | boolean | `false` |         | Refine the `claude/` discovery and functional-gateway mirrors on `/v1/models`. Off by default: both mirror passes behave exactly as before. On: `claude/` discovery mirrors also cover listed reasoning-effort variants (`-low`/`-medium`/`-high`/`-xhigh`), skip built-in `auto/*` ids the request path cannot materialize and skip an alias already listed; functional-gateway mirrors skip combos and OmniRoute-synthesized ids (`claude/`, `no-think/`), take the owner from `owned_by`, never use the owner as its own gateway, skip gateways disabled in `blockedProviders` and models outside a gateway's authoritative live catalog.    |
 | ---------------------------------------------- | ------- | ------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `ANTHROPIC_COMPATIBLE_MID_CONVERSATION_SYSTEM` | boolean | `false` |         | On plain `anthropic-compatible-*` relays (claude → claude passthrough), keep mid-conversation `role: "system"` turns inside `messages[]` instead of hoisting them into the top-level `system` parameter, so the cached prompt prefix stays stable across turns. Applies only when the client itself sent the `mid-conversation-system-2026-04-07` `anthropic-beta` (forwarded upstream as-is). Off by default: every system turn is hoisted exactly as before, because relays without the beta reject `role: "system"` messages with 400.                                                                                                       |
 | `CLAUDE_CACHE_TAIL_TTL_1H`                     | boolean | `false` |         | Mark the conversation tail that OmniRoute anchors for Claude prompt caching (the second-to-last user turn and the last assistant turn) with the same `1h` TTL already used for `system` and `tools`, so a pause longer than five minutes does not rewrite the cached prefix. Vertex keeps the 5m TTL. Off by default: the tail keeps the 5m TTL exactly as before. A 1h cache write costs 2x the input price instead of 1.25x, so fast agent loops that never idle past five minutes pay more with this on. Client-managed (passthrough) markers are never changed.                                                                             |
 | `MODELS_DEV_EFFORT_TIERS_ENABLED`              | boolean | `false` |         | Use the reasoning effort tiers models.dev publishes (narrowed to the known effort vocabulary) when neither an operator override nor the provider registry declares any, and treat a declared empty tier list as "no tiers" in `/v1/models` `effort_tiers` and combo tier intersections. Off by default: `effort_tiers` and resolved `supportedThinkingEfforts` stay exactly as before.                                                                                                                                                                                                                                                          |
 | `AUTO_REQUEST_RESOLVED_CAPABILITIES`           | boolean | `false` |         | auto/* requests build the candidate pool with resolved model capabilities (vision, reasoning, thinking) and observed quality scores in one bulk read (#16419). Off by default: request-time snapshot weights stay exactly as before. On: those signals feed the snapshot weights, so candidate weights and selection can change.                                                                                                                                                                                                                                                                                                                |
@@ -267,10 +266,10 @@ Returns every flag with its effective value, source, and a summary.
       "requiresRestart": false,
       "warningLevel": "caution",
     },
-    // ... all 116 flags
+    // ... all 115 flags
   ],
   "summary": {
-    "total": 116,
+    "total": 115,
     "active": 0,
     "inactive": 0,
     "overriddenByDb": 0,

open-sse/config/anthropicHeaders.ts

@@ -394,12 +394,12 @@ export function mergeClientAnthropicBeta(
  * third-party upstream did not advertise. A merge that survives to nothing leaves
  * the header unset instead of emitting an empty one.
  *
- * Same seeded path: with ANTHROPIC_COMPATIBLE_MID_CONVERSATION_SYSTEM on and the beta
- * negotiated by the client, chatCore keeps mid-conversation system turns inside messages[]
- * for these relays (open-sse/services/claudeMidConversationSystem.ts), so the client's own
- * `mid-conversation-system-2026-04-07` token is forwarded too — never invented, and only
- * while such a turn is still in the body (with the flag off every turn is hoisted first,
- * so the header stays exactly as before).
+ * Same seeded path: when a connection opts into keepMidConversationSystem, chatCore keeps
+ * mid-conversation system turns inside messages[] for these relays
+ * (open-sse/services/claudeMidConversationSystem.ts), so the client's own
+ * `mid-conversation-system-2026-04-07` token is forwarded too when the client sent it —
+ * never invented, and only while such a turn is still in the body (with the opt-in off every
+ * turn is hoisted first, so the header stays exactly as before).
  */
 export function applyClientAnthropicBeta(
   headers: Record<string, string>,

open-sse/handlers/chatCore.ts

@@ -2167,13 +2167,8 @@ async function handleChatCoreInner({
         ...options,
         provider,
         sourceFormat,
-        // #16236: in-place mid-conversation system turns are opt-in AND need the client's beta.
-        clientAnthropicBeta: getHeaderValueCaseInsensitive(
-          clientRawRequest?.headers ?? null,
-          "anthropic-beta"
-        ),
-        isMidConversationSystemEnabled: () =>
-          isFeatureFlagEnabled("ANTHROPIC_COMPATIBLE_MID_CONVERSATION_SYSTEM"),
+        // #16236: in-place mid-conversation system turns are a per-connection opt-in.
+        providerSpecificData: credentials?.providerSpecificData,
       },
       log
     );

open-sse/handlers/chatCore/claudeSystemRole.ts

@@ -231,16 +231,14 @@ export function hoistLeadingTextSystemMessages(payload: Record<string, unknown>)
 export type SystemRoleRoute = {
   provider?: string | null;
   sourceFormat?: string | null;
-  /** The client's own `anthropic-beta` header (the beta must be negotiated by the client). */
-  clientAnthropicBeta?: string | null;
-  /** Injected ANTHROPIC_COMPATIBLE_MID_CONVERSATION_SYSTEM reader; absent/throwing = off. */
-  isMidConversationSystemEnabled?: (() => boolean) | null;
+  /** The connection's providerSpecificData (carries the keepMidConversationSystem opt-in). */
+  providerSpecificData?: unknown;
 };
 
 /**
  * Picks how system/developer turns leave messages[] before a native Claude upstream call.
  * Upstreams that accept mid-conversation system turns (resolveClaudeMidConversationSystemPolicy:
- * opt-in flag + client-negotiated beta) only get the leading run hoisted
+ * per-connection keepMidConversationSystem opt-in) only get the leading run hoisted
  * (and directive-only messages moved off messages[0]), so later turns keep their position
  * and the cache prefix stays stable. Accepting `system` turns says nothing about OpenAI
  * `developer` turns, so a body carrying one keeps the full hoist.
@@ -253,8 +251,7 @@ export function liftSystemRoleMessages(
     provider: route?.provider,
     sourceFormat: route?.sourceFormat,
     targetFormat: FORMATS.CLAUDE,
-    clientAnthropicBeta: route?.clientAnthropicBeta,
-    isPassthroughEnabled: route?.isMidConversationSystemEnabled,
+    providerSpecificData: route?.providerSpecificData,
   });
   const messages = Array.isArray(payload.messages)
     ? (payload.messages as Array<{ role?: unknown }>)

open-sse/services/claudeMidConversationSystem.ts

@@ -9,65 +9,45 @@
  * the cache each turn.
  *
  * Keeping those turns in place is only valid on an upstream that accepts `role: "system"`
- * inside `messages[]`, which Anthropic gates behind the `mid-conversation-system-2026-04-07`
- * beta. Many Anthropic-shaped relays (GLM, Kimi, MiniMax endpoints) do not support it and
- * answer such a body with 400, so the in-place shape is used only when BOTH hold:
- *   1. the operator turned on `ANTHROPIC_COMPATIBLE_MID_CONVERSATION_SYSTEM` (default off),
- *   2. the client itself negotiated the beta in its `anthropic-beta` header — the same signal
- *      native `claude` relies on (shouldUseMidConversationSystem in executors/claudeIdentity.ts
- *      adds the beta it then needs). OmniRoute never invents the beta; the client's token is
- *      forwarded by applyClientAnthropicBeta (config/anthropicHeaders.ts) while such a turn is
- *      still in the body.
- * Anything else — flag off, beta absent, flag read failing — keeps the legacy full hoist.
+ * inside `messages[]`. Many Anthropic-shaped relays (GLM, Kimi, MiniMax endpoints) answer such
+ * a body with 400, and whether a relay accepts it is a property of that relay, not of the
+ * client: a client does not always send the `mid-conversation-system-2026-04-07` beta even
+ * when its relay accepts the shape. So the in-place shape is a per-connection opt-in
+ * (`providerSpecificData.keepMidConversationSystem`, default off). The client's own beta is
+ * still forwarded when present (applyClientAnthropicBeta, config/anthropicHeaders.ts) and is
+ * never invented. Anything else keeps the legacy full hoist.
  */
 
-import { MID_CONVERSATION_SYSTEM_BETA } from "../config/anthropicHeaders.ts";
 import { isClaudeCodeCompatibleProvider } from "./claudeCodeCompatible.ts";
 import { FORMATS } from "../translator/formats.ts";
 
-export const MID_CONVERSATION_SYSTEM_FLAG = "ANTHROPIC_COMPATIBLE_MID_CONVERSATION_SYSTEM";
-
-/** Whether a client `anthropic-beta` comma-list carries the mid-conversation-system token. */
-export function clientNegotiatedMidConversationSystem(clientBeta: unknown): boolean {
-  if (typeof clientBeta !== "string") return false;
-  return clientBeta
-    .split(",")
-    .some((token) => token.trim().toLowerCase() === MID_CONVERSATION_SYSTEM_BETA);
-}
-
-/** Fail closed: a throwing flag reader means "off" (legacy hoist). */
-function readFlag(isEnabled: (() => boolean) | null | undefined): boolean {
-  if (typeof isEnabled !== "function") return false;
-  try {
-    return isEnabled() === true;
-  } catch {
-    return false;
-  }
+/** Per-connection opt-in on providerSpecificData (strict boolean). */
+export function keepsMidConversationSystem(providerSpecificData: unknown): boolean {
+  return (
+    !!providerSpecificData &&
+    typeof providerSpecificData === "object" &&
+    (providerSpecificData as Record<string, unknown>).keepMidConversationSystem === true
+  );
 }
 
 /**
  * True when this request should keep its mid-conversation system turns in `messages[]`.
  * Only plain `anthropic-compatible-*` passthrough is covered: native `claude` keeps its own
  * model/shape policy (shouldUseMidConversationSystem) and the CC bridge builds its own body.
- * The flag reader is injected (chatCore owns the feature-flag import) and only invoked once the
- * cheap scope checks passed.
  */
 export function resolveClaudeMidConversationSystemPolicy({
   provider,
   sourceFormat,
   targetFormat,
-  clientAnthropicBeta,
-  isPassthroughEnabled,
+  providerSpecificData,
 }: {
   provider: string | null | undefined;
   sourceFormat: string | null | undefined;
   targetFormat: string | null | undefined;
-  clientAnthropicBeta?: string | null;
-  isPassthroughEnabled?: (() => boolean) | null;
+  providerSpecificData?: unknown;
 }): boolean {
   if (sourceFormat !== FORMATS.CLAUDE || targetFormat !== FORMATS.CLAUDE) return false;
   if (typeof provider !== "string" || !provider.startsWith("anthropic-compatible-")) return false;
   if (isClaudeCodeCompatibleProvider(provider)) return false;
-  if (!clientNegotiatedMidConversationSystem(clientAnthropicBeta)) return false;
-  return readFlag(isPassthroughEnabled);
+  return keepsMidConversationSystem(providerSpecificData);
 }

open-sse/services/systemPrompt.ts

@@ -232,7 +232,7 @@ export function injectSystemPromptPostTranslation<T>(
   // early-return and silently got zero injection.
   // An anthropic-compatible relay can still carry system-role turns inside
   // messages[] when the opt-in mid-conversation-system passthrough kept them in
-  // place (#16236 — ANTHROPIC_COMPATIBLE_MID_CONVERSATION_SYSTEM + client beta;
+  // place (#16236 — per-connection keepMidConversationSystem opt-in;
   // with it off every turn is hoisted before this point, so nothing changes);
   // those are conversation turns, not the prompt carrier — writing the suffix
   // onto the LAST one would move it every turn and break the cached prefix.

src/app/(dashboard)/dashboard/providers/[id]/components/modals/EditConnectionModal.tsx

@@ -180,6 +180,7 @@ export default function EditConnectionModal({
     passthroughModels: connectionProviderSpecificData?.passthroughModels === true,
     disableCooling: connectionProviderSpecificData?.disableCooling === true,
     requireUserLastMessage: connectionProviderSpecificData?.requireUserLastMessage === true,
+    keepMidConversationSystem: connectionProviderSpecificData?.keepMidConversationSystem === true,
     importFreeModelsOnly: connectionProviderSpecificData?.importFreeModelsOnly === true,
     commandCodeZdrEnabled: connectionProviderSpecificData?.zdrEnabled === true,
     tunnelId: stringField(connectionProviderSpecificData?.tunnelId),
@@ -411,6 +412,8 @@ export default function EditConnectionModal({
         passthroughModels: connection?.providerSpecificData?.passthroughModels === true,
         disableCooling: connection?.providerSpecificData?.disableCooling === true,
         requireUserLastMessage: connection?.providerSpecificData?.requireUserLastMessage === true,
+        keepMidConversationSystem:
+          connection?.providerSpecificData?.keepMidConversationSystem === true,
         importFreeModelsOnly: connection?.providerSpecificData?.importFreeModelsOnly === true,
         commandCodeZdrEnabled: connection.providerSpecificData?.zdrEnabled === true,
         tunnelId: stringField(connection.providerSpecificData?.tunnelId),
@@ -714,6 +717,10 @@ export default function EditConnectionModal({
             ? true
             : undefined;
         }
+        if (isPlainAnthropicCompatible) {
+          updates.providerSpecificData.keepMidConversationSystem =
+            formData.keepMidConversationSystem ? true : undefined;
+        }
         updates.providerSpecificData.peakHourProtection = normalizePeakHourProtectionForSave(
           formData.peakHourProtection
         );
@@ -793,6 +800,22 @@ export default function EditConnectionModal({
       )}
     />
   ) : null;
+  const keepMidConversationSystemToggle = isPlainAnthropicCompatible ? (
+    <Toggle
+      checked={formData.keepMidConversationSystem}
+      onChange={(checked) => setFormData({ ...formData, keepMidConversationSystem: checked })}
+      label={providerText(
+        t,
+        "keepMidConversationSystemLabel",
+        "Keep mid-conversation system messages in place"
+      )}
+      description={providerText(
+        t,
+        "keepMidConversationSystemDescription",
+        'Leave system messages sent mid-conversation inside the message list instead of moving them into the top-level system prompt, so the prompt cache keeps hitting across turns. Enable only if this upstream accepts role "system" inside messages (otherwise it answers 400).'
+      )}
+    />
+  ) : null;
   const openaiResponsesStoreToggle = isResponsesConnection ? (
     <Toggle
       checked={formData.openaiResponsesStoreEnabled}
@@ -878,6 +901,7 @@ export default function EditConnectionModal({
           {preserveEncryptedReasoningToggle}
           {openaiResponsesStoreToggle}
           {requireUserLastMessageToggle}
+          {keepMidConversationSystemToggle}
           <CommandCodeZdrToggle
             provider={provider}
             checked={formData.commandCodeZdrEnabled}