fix(vertex,azure): model-aware mid-conversation system for Claude /v1/messages by mateo-berri · Pull Request #33807 · BerriAI/litellm

mateo-berri · 2026-07-18T03:14:01Z

Relevant issues

Closes the Vertex/Azure gap called out in the customer RCA (High Spend on Claude Code via Bedrock Invoke), item 3 under "Gaps still there": test Vertex and Azure and hoist mid-conversation system messages where needed

Linear ticket

Resolves LIT-4563

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
My PR passes all CI/CD checks (e.g., lint, format, unit tests)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

All calls hit real Claude on Azure AI Foundry (litellm-e2e-suite-resource.services.ai.azure.com), no mocks, real spend. azure-opus-4-8 is flagged supports_mid_conversation_system; azure-opus-4-7 is not. The request is a mid-session Claude Code shape: a top-level system, a user turn, a role: "system" reminder, an assistant turn, a fresh user turn

BEFORE (base 40e914cfa7, no fix) — proxy on the base commit, role: "system" forwarded in place:

$ curl -sS http://localhost:PORT/v1/messages -H "Authorization: Bearer sk-****" \
    -H "Content-Type: application/json" -d @midsys.json   # model=azure-opus-4-7
HTTP 400 :: {"type":"error","error":{"type":"invalid_request_error",
             "message":"role 'system' is not supported on this model"}}

# model=azure-opus-4-8 (already worked, nothing was hoisted)
HTTP 200 :: "Bye."

Every Claude Code session on an older Azure/Vertex Claude model 400s the moment it appends a reminder. The supported model works only because nothing is hoisted, which is also why a naive fix that hoisted everything would collapse its prompt cache (the regression from the customer RCA)

AFTER (this PR, 23b5b7d199) — same proxy, same requests:

# model=azure-opus-4-7 (unflagged): reminder hoisted into top-level system
HTTP 200 :: "Bye."

# model=azure-opus-4-8 (flagged): reminder kept in place
HTTP 200 :: "Bye."

Wire-level (detailed_debug), the request LiteLLM sends upstream:

azure-opus-4-8 (flagged) : messages roles = [user, system, assistant, user]   # reminder kept in place
                           top-level system = ["You are terse. Answer in one word."]
azure-opus-4-7 (unflagged): messages roles = [user, assistant, user]          # reminder removed from messages
                           top-level system = ["You are terse...", "<system-reminder>...</system-reminder>"]

Cache preserved on the flagged model — two turns against azure-opus-4-8, second turn carries the mid-conversation reminder:

turn 1 (prime)   : cache_creation=6013  cache_read=0       # wrote the prefix
turn 1 (re-send) : cache_creation=0     cache_read=6013    # cache warm
turn 2 (+reminder): cache_creation=41   cache_read=6013    # full prefix read back, only the new turn is written

The reminder does not mutate the system prefix, so cache_read stays at the full 6013 instead of collapsing to a fresh write

Vertex AI (live, real spend) — same proof over vertex_ai/claude-opus-4-8 (flagged) and vertex_ai/claude-opus-4-7 (unflagged) on the Vertex global endpoint (project vertex-check-481318):

BEFORE (base 40e914cfa7, Vertex transform reverted) : model=vertex-opus-4-7 mid-conversation
HTTP 400 :: "messages: Unexpected role \"system\". The Messages API accepts a
             top-level `system` parameter, not \"system\" as an input message role."

AFTER (this PR):
  vertex-opus-4-7 (unflagged): reminder hoisted into top-level system -> HTTP 200 :: "Bye."
  vertex-opus-4-8 (flagged)  : reminder kept in messages             -> HTTP 200 :: "Bye."

Wire-level (detailed_debug), the request LiteLLM sends upstream to Vertex:

vertex-opus-4-8 (flagged) : messages roles = [user, system, assistant, user]   # reminder kept in place
                            top-level system = ["You are terse. Answer in one word."]
vertex-opus-4-7 (unflagged): messages roles = [user, assistant, user]          # reminder removed from messages
                            top-level system = ["You are terse...", "<system-reminder>...</system-reminder>"]

Cache preserved on the flagged Vertex model — two turns against vertex-opus-4-8, second turn carries the reminder:

turn 1 (prime)    : cache_creation=15615  cache_read=0       # wrote the prefix
turn 1 (re-send)  : cache_creation=0      cache_read=15615   # cache warm
turn 2 (+reminder): cache_creation=0      cache_read=15615   # full prefix read back, reminder didn't touch system

Vertex's rejection wording differs from Azure's ("Unexpected role system / use the top-level system parameter" vs "role 'system' is not supported on this model"), but the behavior class is identical: the older/unflagged Claude rejects a mid-conversation role: "system" while the flagged 4.8+/5 model accepts it in place, so the same model-aware hoist closes both. Both vertex.mid_conversation_system registry rows are now fail_before_fix: proven

Type

🐛 Bug Fix

✅ Test

Changes

Azure AI Foundry and Vertex AI serve Claude on the first-party Anthropic /v1/messages contract. Probing api.anthropic.com and Azure Foundry live returned byte-identical validation: a leading role: "system" entry inside messages is rejected on every model ("messages.0: use the top-level 'system' parameter for the initial system prompt"), a mid-conversation role: "system" reminder is accepted in place on Claude 4.8+/5 but rejected on Claude 4.7 and older ("role 'system' is not supported on this model"). Bedrock Invoke already handles this model-aware (#32578/#32831/#32882); Vertex and Azure did no hoisting at all

Extracted Bedrock's _normalize_system_role_messages into the shared AnthropicMessagesConfig base and call it from the Vertex and Azure messages configs. Flagged models (supports_mid_conversation_system: Claude 4.8+/5) hoist only the leading run of system entries and keep mid-conversation reminders in place so the top-level system prefix stays byte-identical and the prompt cache survives; unflagged models hoist every system entry so the request returns a completion instead of a 400. Bedrock keeps its exact behavior through the shared method
Added supports_mid_conversation_system to the azure_ai and vertex_ai Claude 4.8+/5 cost-map entries in both model_prices_and_context_window.json and the bundled backup. Exact cost-map hits win over the claude-mid-conversation-system fallback rule, so without the explicit flag these models would be treated as unsupported and hoist every reminder, collapsing the cache. A per-provider test guards this so future 4.8+/5 entries cannot silently miss the flag
The first-party anthropic/ path is intentionally left untouched: it forwards messages as before and keeps billing-header attribution (should_strip_billing_metadata stays False there)

Unit tests cover leading-run hoisting, mid-conversation preservation on flagged models, hoist-all on unflagged models, and the flag-coverage guard, for both Azure and Vertex. Mutation-checked: disabling the hoist kills the leading-run and unflagged tests; removing the flag kills the keep-in-place, leading-run, and flag-coverage tests (the exact cache-collapse mutant)

QA runbook

Prerequisites: a proxy with an Azure AI Foundry Anthropic deployment for a flagged model (azure_ai/claude-opus-4-8) and an unflagged one (azure_ai/claude-opus-4-7) via api_base: os.environ/AZURE_AI_API_BASE, api_key: os.environ/AZURE_AI_API_KEY; and Vertex deployments (vertex_ai/claude-opus-4-8, vertex_ai/claude-sonnet-4-6) via vertex_project: os.environ/VERTEXAI_PROJECT, vertex_location: global. The Vertex legs need working Vertex credentials in the runner

tests/e2e/llm_translation/test_messages_mid_conversation_system_native_providers_e2e.py::TestAzureFoundryMidConversationSystem::test_flagged_model_keeps_prompt_cache_across_system_reminder - a flagged Azure model keeps a mid-conversation system reminder in messages so the prompt cache written on turn one is read back in full on turn two
- Register azure_ai/claude-opus-4-8; prime a >1024-token cached system + a cache-marked user turn until cache_read>0 and cache_creation>0
- Send turn two: same system, the primed user turn (cached), a role: "system" reminder, an assistant turn, a fresh cached user turn
- Expect a non-empty completion and cache_read_input_tokens >= (turn-one system prefix + first user turn)
- Sanity check: this test makes sense to add and is not hand-wavey (it asserts the exact primed token count is read back, not just cache_read > 0) or potentially flaky
tests/e2e/llm_translation/test_messages_mid_conversation_system_native_providers_e2e.py::TestAzureFoundryMidConversationSystem::test_unflagged_model_hoists_system_reminder_and_succeeds - an unflagged Azure model 200s with a mid-conversation reminder because it is hoisted into top-level system
- Register azure_ai/claude-opus-4-7; POST /v1/messages with a top-level system, a user turn, a role: "system" reminder, an assistant turn, a user turn
- Expect HTTP 200 with role: "assistant" and non-empty text (base code returns 400 "role 'system' is not supported on this model")
- Sanity check: this test makes sense to add and is not hand-wavey or potentially flaky
tests/e2e/llm_translation/test_messages_mid_conversation_system_native_providers_e2e.py::TestVertexMidConversationSystem::test_flagged_model_keeps_prompt_cache_across_system_reminder - same as the Azure flagged case, over vertex_ai/claude-opus-4-8
- Register vertex_ai/claude-opus-4-8; prime and assert cache_read is preserved across the reminder turn as above
- Sanity check: this test makes sense to add and is not hand-wavey or potentially flaky
tests/e2e/llm_translation/test_messages_mid_conversation_system_native_providers_e2e.py::TestVertexMidConversationSystem::test_unflagged_model_hoists_system_reminder_and_succeeds - an unflagged Vertex model 200s with the reminder hoisted, over vertex_ai/claude-sonnet-4-6
- Register vertex_ai/claude-sonnet-4-6; POST the reminder conversation and expect HTTP 200 with non-empty assistant text
- Sanity check: this test makes sense to add and is not hand-wavey or potentially flaky

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

greptile-apps · 2026-07-18T03:20:01Z

Greptile Summary

This PR closes the Vertex AI and Azure AI Foundry gap from the Kraken Tech RCA: both providers serve Claude on the native Anthropic /v1/messages contract but previously forwarded role: "system" messages in place, causing 400s on older Claude models and unnecessary prompt-cache invalidation on 4.8+/5. The fix extracts Bedrock's model-aware _normalize_system_role_messages into the shared AnthropicMessagesConfig base and calls it from the Vertex and Azure transform methods, using the polymorphic self.custom_llm_provider so each provider's cost-map lookup resolves correctly.

_normalize_system_role_messages is now in the shared base; Bedrock replaces its local copy with a call to the shared method (behavior unchanged since its custom_llm_provider still returns "bedrock"). Azure and Vertex call it for the first time, hoisting only the leading system run on flagged models (4.8+/5) and all system entries on unflagged models.
supports_mid_conversation_system: true is added to azure_ai/claude-opus-4-8, azure_ai/claude-sonnet-5, azure_ai/claude-fable-5, and the matching vertex_ai/* and vertex_ai/*@default entries; a cost-map flag-coverage test guards against future 4.8+/5 entries missing the flag.
Unit tests pin all three hoist scenarios for both providers; e2e tests exercise cache preservation on flagged models and successful completion on unflagged models against live Azure Foundry and Vertex endpoints.

Confidence Score: 5/5

Safe to merge — the change fixes a live 400 error on older Claude models on Azure/Vertex and prevents prompt-cache collapse on 4.8+/5, with no behavioral change to the first-party Anthropic path or Bedrock.

The refactoring correctly relies on the polymorphic self.custom_llm_provider (returns "azure_ai" / "vertex_ai" / "bedrock" in each subclass) rather than a hardcoded string, so the cost-map lookup resolves to the right provider for every caller. AnthropicMessagesRequest is a TypedDict (no runtime validation), so role: "system" entries survive the assembly step and reach _normalize_system_role_messages intact. The three scenarios — leading-run hoist on flagged models, keep-in-place for mid-conversation entries on flagged models, hoist-all on unflagged models — are each covered by unit tests and, for Azure and Vertex, by live e2e tests against real endpoints. The cost-map flag-coverage guard closes the future-regression surface.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/llms/anthropic/experimental_pass_through/messages/transformation.py	Extracted `_normalize_system_role_messages`, `_as_system_content_blocks`, and `_is_system_role_message` from Bedrock into the shared `AnthropicMessagesConfig` base; base class never calls the method itself (first-party path unchanged), subclasses opt in.
litellm/llms/azure_ai/anthropic/messages_transformation.py	Added `_normalize_system_role_messages` call in `transform_anthropic_messages_request` before `_remove_scope_from_cache_control`; `custom_llm_provider` returns `"azure_ai"` so the cost-map lookup resolves correctly.
litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/experimental_pass_through/transformation.py	Added `_normalize_system_role_messages` call in `transform_anthropic_messages_request`; `custom_llm_provider` returns `"vertex_ai"` so the cost-map lookup resolves correctly.
litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py	Removed the now-redundant Bedrock-local `_normalize_system_role_messages_for_bedrock` and replaced the call site with the shared `_normalize_system_role_messages`; Bedrock behavior unchanged because its `custom_llm_provider` still returns `"bedrock"`.
model_prices_and_context_window.json	Added `supports_mid_conversation_system: true` to `azure_ai/claude-opus-4-8`, `azure_ai/claude-sonnet-5`, `azure_ai/claude-fable-5`, and the matching `vertex_ai/` and `vertex_ai/@default` entries; no new models added without the flag.
tests/test_litellm/llms/azure_ai/claude/test_azure_anthropic_messages_transformation.py	Added unit tests covering leading-run hoist, mid-conversation preservation on flagged models, hoist-all on unflagged models for Azure; the cost-map flag-coverage test correctly uses `next(..., None)` with an explicit assertion, addressing the previous `StopIteration` comment.
tests/test_litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/test_vertex_ai_partner_models_anthropic_messages_config.py	Parallel unit tests for Vertex: same three hoist scenarios plus the cost-map flag-coverage guard with `next(..., None)` + assertion; uses `str(info.get("litellm_provider", "")).startswith("vertex_ai")` to cover `vertex_ai`, `vertex_ai_beta`, etc.
tests/e2e/llm_translation/test_messages_mid_conversation_system_native_providers_e2e.py	New e2e tests for Azure Foundry and Vertex: flagged-model cache-preservation test (asserts `cache_read >= primed prefix`) and unflagged-model hoist-and-succeed test; cache priming loop avoids warming a mutated prefix by generating a fresh user-turn text on each retry.

_{Reviews (3): Last reviewed commit: "test: give cost-map guard next() a defau..." | Re-trigger Greptile}

codecov · 2026-07-18T03:20:24Z

Codecov Report

❌ Patch coverage is 96.96970% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...perimental_pass_through/messages/transformation.py	96.66%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

codspeed-hq · 2026-07-18T03:45:43Z

Merging this PR will improve performance by 54.77%

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 30 untouched benchmarks

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	`test_completion_streaming`	58.8 ms	38 ms	+54.77%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing litellm_vertex_azure_midsys (8b1a19f) with litellm_internal_staging (214945a)¹}

No successful run was found on litellm_internal_staging (b086cd3) during the generation of this report, so 214945a was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

…/merge-skew flake The model_info / get_model_info_with_id endpoint tests drove refactored endpoints with bare, unspec'd MagicMock routers and models. Because the mocks were unspec'd, any attribute or method the (refactored) endpoints newly read auto-materialized a child MagicMock, and whether that child was reached depended on process-global state (premium_user, and the real get_available_models_for_user chain reading litellm globals) that sibling tests in the same xdist worker mutate. When reached, the MagicMock either unpacked to empty (a, b = mock.method() -> 'not enough values to unpack (expected 2, got 0)') or leaked into RouterModelInfo(**model_info) and failed Pydantic str validation. Pass in isolation, fail under xdist. The original TestModelInfoEndpoint failure (BerriAI#33807 CI) was the same class surfaced by merge skew: BerriAI#33721 added a get_configured_token_limits unpack to create_model_info_response, and CI's merge commit ran that against the un-updated bare-mock test before the BerriAI#33742 band-aid landed. Fix (test-only, no product change): - TestModelInfoEndpoint: mock the real seam (get_available_models_for_user), configure the router methods the endpoint actually calls, return a real Deployment, and drop the dead proxy_server.get_key_models/get_team_models/ get_complete_model_list patches the refactor had stranded. - TestGetModelInfoWithIdBlocked: spec the model mock so unset enterprise columns read as None instead of child MagicMocks. - test_ProxyConfig_get_model_info_with_id_missing_model_id_raises: pin premium_user so the asserted AttributeError no longer flips with the ambient license global.

…/messages Azure AI Foundry and Vertex AI serve Claude on the first-party Anthropic Messages contract, which was verified live to be byte-identical to api.anthropic.com: a leading role:"system" entry in messages is rejected on every model ("messages.0: use the top-level 'system' parameter"), and a mid-conversation role:"system" reminder is accepted in place on Claude 4.8+/5 but 400s on Claude 4.7 and older ("role 'system' is not supported on this model"). This is the same contract Bedrock Invoke already handles model-aware (PRs #32578/#32831/#32882); Vertex and Azure did no hoisting at all, so a Claude Code session on an older Vertex/Azure Claude model hard-400s on its reminder turns, and the only thing sparing 4.8+/5 was that nothing was hoisted Extract Bedrock's model-gated normalization into the shared AnthropicMessagesConfig base as _normalize_system_role_messages and call it from the Vertex and Azure messages configs. Flagged models (4.8+/5) hoist only the leading run of system entries and keep mid-conversation reminders in place so the top-level system prefix stays byte-identical and the prompt cache is preserved; unflagged models hoist every system entry so the request returns a completion instead of a 400 Add supports_mid_conversation_system to the azure_ai and vertex_ai Claude 4.8+/5 cost-map entries. Exact cost-map hits win over the claude-mid-conversation-system fallback rule, so without the explicit flag those models would be treated as unsupported and hoist every reminder, collapsing the prompt cache (the exact customer regression). A per-provider test guards this so future 4.8+/5 entries cannot silently miss the flag Closes the Vertex/Azure gap from the customer RCA

Ran the before/after proof live against Vertex Claude (global endpoint, project vertex-check-481318): base transform 400s an unflagged model (claude-opus-4-7) on a mid-conversation role:system reminder, the fix hoists it to a 200, and a flagged model (claude-opus-4-8) keeps the reminder in messages with cache_read held at 15615 across the reminder turn. Flip both vertex.mid_conversation_system rows to fail_before_fix: proven.

mateo-berri · 2026-07-20T23:48:55Z

@greptileai

…th a clear assertion

mateo-berri · 2026-07-20T23:50:27Z

@greptileai

mateo-berri mentioned this pull request Jul 20, 2026

test(proxy): make model_info endpoint tests hermetic to kill an order/merge-skew flake #34037

Merged

5 tasks

mateo-berri added 2 commits July 20, 2026 16:48

mateo-berri force-pushed the litellm_vertex_azure_midsys branch from cc49913 to 335b79d Compare July 20, 2026 23:48

test: give cost-map guard next() a default so a renamed rule fails wi…

8b1a19f

…th a clear assertion

tin-berri approved these changes Jul 21, 2026

View reviewed changes

mateo-berri merged commit 1e74109 into litellm_internal_staging Jul 21, 2026
87 of 88 checks passed

mateo-berri deleted the litellm_vertex_azure_midsys branch July 21, 2026 00:48

tin-berri mentioned this pull request Jul 21, 2026

test(e2e): fix EndpointsClient gateway attribute left behind by the ProxyClient rename #34090

Open

5 tasks

cursor Bot mentioned this pull request Jul 22, 2026

docs(blog): add July 21 progress updates to the Bedrock Invoke caching incident report BerriAI/litellm-docs#628

Merged

deepanshululla mentioned this pull request Aug 11, 2026

Mid-conversation system-role hoist invalidates the entire prompt-cache prefix (AnthropicMessagesConfig) #36559

Closed

Navigation Menu

Uh oh!

Uh oh!

fix(vertex,azure): model-aware mid-conversation system for Claude /v1/messages - #33807

fix(vertex,azure): model-aware mid-conversation system for Claude /v1/messages#33807
mateo-berri merged 3 commits into
litellm_internal_stagingfrom
litellm_vertex_azure_midsys

mateo-berri commented Jul 18, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 18, 2026 •

edited

Loading

Important Files Changed

Uh oh!

codecov Bot commented Jul 18, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Jul 18, 2026 •

edited

Loading

Uh oh!

mateo-berri commented Jul 20, 2026

Uh oh!

mateo-berri commented Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

mateo-berri commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant issues

Linear ticket

Pre-Submission checklist

Screenshots / Proof of Fix

Type

Changes

QA runbook

Final Attestation

Uh oh!

greptile-apps Bot commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

codecov Bot commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed-hq Bot commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by 54.77%

Performance Changes

Footnotes

Uh oh!

mateo-berri commented Jul 20, 2026

Uh oh!

mateo-berri commented Jul 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mateo-berri commented Jul 18, 2026 •

edited

Loading

greptile-apps Bot commented Jul 18, 2026 •

edited

Loading

codecov Bot commented Jul 18, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 18, 2026 •

edited

Loading