test(e2e): bump claude code compat matrix tiers to sonnet-5 and opus-4-8

[mateo-berri]

Relevant issues

Resolves caveat 4 of #32548 (the model-tier bump follow-up for the Claude Code compatibility matrix)

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

All proofs captured at commit 3befa3a against a live proxy booted from this branch with the suite's own config, hitting the real provider APIs (no mocks):

python litellm/proxy/proxy_cli.py --config tests/e2e/claude_code/test_config.yaml --port 28255

One trivial /v1/messages call per configured alias:

for m in claude-haiku-4-5 claude-sonnet-5 claude-opus-4-8 \
         claude-haiku-4-5-bedrock-invoke claude-sonnet-5-bedrock-invoke claude-opus-4-8-bedrock-invoke \
         claude-haiku-4-5-bedrock-converse claude-sonnet-5-bedrock-converse claude-opus-4-8-bedrock-converse \
         claude-haiku-4-5-vertex claude-sonnet-4-6-vertex claude-opus-4-7-vertex \
         claude-haiku-4-5-azure claude-sonnet-4-6-azure claude-opus-4-8-azure; do
  curl -s http://localhost:28255/v1/messages \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
    -d "{\"model\": \"$m\", \"max_tokens\": 32, \"messages\": [{\"role\": \"user\", \"content\": \"Reply with the single word pong\"}]}"
done

claude-haiku-4-5                       HTTP 200  claude-haiku-4-5 | pong
claude-sonnet-5                        HTTP 200  claude-sonnet-5 | pong
claude-opus-4-8                        HTTP 200  claude-opus-4-8 | pong
claude-haiku-4-5-bedrock-invoke        HTTP 200  claude-haiku-4-5-bedrock-invoke | pong
claude-sonnet-5-bedrock-invoke         HTTP 200  claude-sonnet-5-bedrock-invoke | pong
claude-opus-4-8-bedrock-invoke         HTTP 200  claude-opus-4-8-bedrock-invoke | pong
claude-haiku-4-5-bedrock-converse      HTTP 200  claude-haiku-4-5-bedrock-converse | pong
claude-sonnet-5-bedrock-converse       HTTP 200  claude-sonnet-5-bedrock-converse | pong
claude-opus-4-8-bedrock-converse       HTTP 200  claude-opus-4-8-bedrock-converse | pong
claude-haiku-4-5-vertex                HTTP 200  claude-haiku-4-5-vertex | pong
claude-sonnet-4-6-vertex               HTTP 200  claude-sonnet-4-6-vertex | pong
claude-opus-4-7-vertex                 HTTP 429  (pre-existing, see note below)
claude-haiku-4-5-azure                 HTTP 200  claude-haiku-4-5-azure | pong
claude-sonnet-4-6-azure                HTTP 200  claude-sonnet-4-6-azure | pong
claude-opus-4-8-azure                  HTTP 200  claude-opus-4-8-azure | pong

Why Vertex AI keeps the old tiers: the same curls with the new-tier aliases (run before pinning the column back) 429 deterministically

claude-sonnet-5-vertex   HTTP 429  "Quota exceeded for aiplatform.googleapis.com/online_prediction_input_tokens_per_minute_per_base_model with base model: anthropic-claude-sonnet-5. Please submit a quota increase request."
claude-opus-4-8-vertex   HTTP 429  "Quota exceeded for ... base model: anthropic-claude-opus-4-8. Please submit a quota increase request."

The Service Usage API confirms the suite's Vertex project has no TPM granted for either new base model in its region (quota buckets exist with no effective limit), so these 429s are a hard 0-quota condition rather than burst exhaustion

Why Azure keeps sonnet-4-6 in the sonnet slot (opus does bump to 4-8, which is already deployed on the suite's Foundry resource): the subscription's quota limit for Claude Sonnet 5 is 0, so a deployment cannot even be created

$ az cognitiveservices account deployment create ... --deployment-name claude-sonnet-5 \
    --model-name claude-sonnet-5 --model-version 2 --model-format Anthropic \
    --sku-name GlobalStandard --sku-capacity 40
ERROR: (InsufficientQuota) This operation require 40 new capacity in quota Tokens Per Minute (thousands) - Claude Sonnet 5, which is bigger than the current available capacity 0. The current quota usage is 0 and the quota limit is 0 for quota Tokens Per Minute (thousands) - Claude Sonnet 5.

Note on claude-opus-4-7-vertex: that alias is unchanged by this PR (it is the current large tier in the merged suite, and this PR keeps vertex on it). It 429'd on the same per-base-model TPM metric throughout the validation window even though the project has an 8M TPM grant for the opus-4-7 base model and the haiku/sonnet vertex aliases pass, consistent with concurrent live testing consuming that model's quota at capture time. Pre-existing environment condition, orthogonal to this PR

Type

✅ Test

Changes

Bumps the Claude Code compatibility matrix's mid and large model tiers from Sonnet 4.6 / Opus 4.7 to Sonnet 5 / Opus 4.8 wherever the suite's live provider environments can actually serve them, per caveat 4 of #32548. The haiku-4-5 small tier is unchanged everywhere, and run_daily.sh is untouched (owned by another in-flight PR)

Per provider column: anthropic, bedrock_invoke, and bedrock_converse move fully to claude-sonnet-5 / claude-opus-4-8 (aliases in test_config.yaml, the per-feature test files, and the _builder/_driver unit-test fixtures that encode alias names). azure bumps opus to claude-opus-4-8 but keeps claude-sonnet-4-6 in its sonnet slot because the suite's Foundry subscription has a hard 0 TPM quota limit for Claude Sonnet 5 (deployment creation rejected with InsufficientQuota; evidence above). vertex_ai stays fully on claude-sonnet-4-6 / claude-opus-4-7 because both new base models have 0 TPM in the suite's Vertex project and region and deterministically 429

test_v0_layout.py::test_per_provider_test_file_imports_and_parametrizes_three_models now encodes the per-provider tier expectations (DEFAULT_TIERS plus PROVIDER_TIER_OVERRIDES for the two quota-pinned columns) with a docstring explaining why each pin exists and what to flip once quota lands, so the pins are enforced by a failing test instead of being folklore. Tier-naming comments in manifest.yaml and the test docstrings were updated to match; the long-context manifest comment went tier-generic since the exercised models now differ per column. sample_compatibility-matrix.json needed no change (it carries no model names)

Follow-ups for whoever owns the provider accounts: request Claude Sonnet 5 TPM quota in the Foundry subscription (self-serve form), request sonnet-5 and opus-4-8 TPM quota in the Vertex project, then flip the pinned aliases plus the layout-test overrides

QA runbook

Environment prerequisites: a proxy booted from this branch with --config tests/e2e/claude_code/test_config.yaml and the suite's provider credentials in the environment (ANTHROPIC_API_KEY, AWS credentials for us-east-1 Bedrock, VERTEXAI_PROJECT/VERTEXAI_LOCATION plus GCP credentials, AZURE_FOUNDRY_API_BASE/AZURE_FOUNDRY_API_KEY pointing at the Foundry resource that hosts the Claude deployments). Every changed e2e file is the same mechanical alias swap, so the runbook covers one representative cell per provider column plus the structural pin

• tests/e2e/claude_code/basic_messaging_non_streaming/test_anthropic.py::test_basic_messaging_non_streaming_anthropic - the anthropic column now exercises haiku-4-5, sonnet-5, and opus-4-8
  • POST /v1/messages with {"model": "claude-sonnet-5", "max_tokens": 32, "messages": [{"role": "user", "content": "Reply with the single word pong"}]} and the proxy key
  • Expect 200 with an assistant text reply; repeat for claude-opus-4-8 and claude-haiku-4-5
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/claude_code/basic_messaging_non_streaming/test_bedrock_invoke.py::test_basic_messaging_non_streaming_bedrock_invoke - the invoke column routes to bedrock/us.anthropic.claude-sonnet-5 and bedrock/us.anthropic.claude-opus-4-8
  • Same POST as above with claude-sonnet-5-bedrock-invoke then claude-opus-4-8-bedrock-invoke
  • Expect 200 with a text reply from each
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/claude_code/basic_messaging_non_streaming/test_bedrock_converse.py::test_basic_messaging_non_streaming_bedrock_converse - the converse column routes to bedrock/converse/us.anthropic.claude-sonnet-5 and bedrock/converse/us.anthropic.claude-opus-4-8
  • Same POST with claude-sonnet-5-bedrock-converse then claude-opus-4-8-bedrock-converse
  • Expect 200 with a text reply from each
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/claude_code/basic_messaging_non_streaming/test_vertex_ai.py::test_basic_messaging_non_streaming_vertex_ai - the vertex column intentionally stays on sonnet-4-6 and opus-4-7
  • Same POST with claude-sonnet-4-6-vertex; expect 200
  • POST with model claude-sonnet-5-vertex; expect a 400 invalid-model error from the proxy (alias deliberately not registered), confirming the pin is real
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/claude_code/basic_messaging_non_streaming/test_azure.py::test_basic_messaging_non_streaming_azure - the azure column bumps opus to 4-8 and keeps sonnet on 4-6
  • Same POST with claude-opus-4-8-azure then claude-sonnet-4-6-azure; expect 200 from each
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/claude_code/_builder_unit_tests/test_v0_layout.py::test_per_provider_test_file_imports_and_parametrizes_three_models - every provider file must reference its column's pinned tier set
  • Run the test and expect all parametrizations to pass; then edit e.g. tool_use/test_azure.py to say sonnet-5-azure and expect the azure parametrizations to fail naming the missing sonnet-4-6 tier
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR bumps the Claude Code compatibility matrix's mid and large model tiers from Sonnet 4.6 / Opus 4.7 to Sonnet 5 / Opus 4.8 across the anthropic, bedrock-invoke, and bedrock-converse provider columns, with partial updates for azure (opus only) and vertex AI intentionally left on older tiers due to hard 0-TPM quota limits on the newer base models.

• test_config.yaml: registers new claude-sonnet-5 / claude-opus-4-8 aliases for anthropic, bedrock-invoke, and bedrock-converse; azure opus moves to claude-opus-4-8-azure; azure sonnet and all vertex entries are unchanged.
• 69 e2e and unit test files: mechanical alias swap throughout; all _driver_unit_tests fixtures and the _builder_unit_tests matrix builder updated to match.
• test_v0_layout.py: introduces DEFAULT_TIERS / PROVIDER_TIER_OVERRIDES constants with a docstring explaining the quota-pinned columns, so deviations from the expected tier set now cause a failing test rather than silent drift.

Confidence Score: 4/5

Safe to merge; changes are confined to test configuration and e2e test files, with well-documented rationale for the two quota-pinned provider columns.

All 69 changed files are test-only (e2e suite config and unit test fixtures). The alias swaps are internally consistent across test_config.yaml, the per-feature test files, and the builder/driver unit-test fixtures for the three fully-bumped columns. The two quota-pinned columns (vertex, azure-sonnet) are correctly left on their older tiers and the new layout guard in test_v0_layout.py enforces this via PROVIDER_TIER_OVERRIDES. One minor inconsistency exists in test_rate_limiter.py where two provider-inference example aliases reference model names that are not registered in the proxy config due to quota pins — the unit test logic is unaffected since infer_provider is purely suffix-based, but the examples no longer reflect the actual deployed aliases.

tests/e2e/claude_code/_driver_unit_tests/test_rate_limiter.py — the two updated provider-inference examples reference aliases that don't exist in the proxy config due to intentional quota pins.

Important Files Changed

Filename	Overview
tests/e2e/claude_code/test_config.yaml	Bumps anthropic, bedrock-invoke, and bedrock-converse to sonnet-5/opus-4-8; Azure opus bumped to 4-8; Azure sonnet and Vertex AI intentionally left on older tiers. Consistent with the PR description.
tests/e2e/claude_code/_builder_unit_tests/test_v0_layout.py	Adds DEFAULT_TIERS and PROVIDER_TIER_OVERRIDES constants to encode per-provider pinned tiers, with detailed docstring explaining quota rationale; the layout enforcement test now uses provider-specific tier sets instead of a single global set.
tests/e2e/claude_code/_driver_unit_tests/test_rate_limiter.py	Updated provider-inference parametrize examples to reflect bumped model tiers; two examples now reference aliases (claude-sonnet-5-azure, claude-opus-4-8-vertex) that don't exist in test_config.yaml due to quota pins on those columns.
tests/e2e/claude_code/_builder_unit_tests/test_matrix_builder.py	Mechanical alias swap from sonnet-4-6/opus-4-7 to sonnet-5/opus-4-8 in fixture data and assertions; test logic unchanged.
tests/e2e/claude_code/manifest.yaml	Comments updated to reflect new tier names; long_context comment made tier-generic since different providers now exercise different model versions.
tests/e2e/claude_code/thinking/test_azure.py	Opus bumped to 4-8-azure; removed stale note about opus-4-7-preview thinking limitations on Foundry, which is no longer relevant with the 4-8 upgrade.
tests/e2e/claude_code/long_context_1m/test_azure.py	Opus bumped to 4-8-azure; comment updated to reflect sonnet stays at 4-6 but opus moves to 4-8; sonnet model unchanged.

_{Reviews (1): Last reviewed commit: "test(e2e): bump claude code compat matri..." | Re-trigger Greptile}

[greptile-apps]

Rate-limiter examples no longer match deployed aliases

Two of the updated example aliases don't correspond to any entry in test_config.yaml: claude-sonnet-5-azure (azure sonnet intentionally stays on claude-sonnet-4-6-azure due to Foundry quota) and claude-opus-4-8-vertex (vertex intentionally stays on claude-opus-4-7-vertex due to GCP quota). infer_provider is suffix-based so the logic under test still passes, but anyone cross-referencing these examples against the proxy config will find no matching alias, which could be confusing during future maintenance or quota-flip work.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_compat_matrix_tier_bump (3befa3a) with litellm_internal_staging (923c325)}