test(e2e/claude_code): add GPT-5.6 Sol/Terra/Luna columns for OpenAI, Azure OpenAI, and Bedrock Mantle

[mateo-berri]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

All proof below was captured at commit 710a88e against a live proxy booted from the updated suite config, hitting real provider APIs with no mocks

Proxy boot, deliberately without the new Azure OpenAI env vars, to prove the external PR gate cannot be crashed by missing credentials:

$ uv run python litellm/proxy/proxy_cli.py --config tests/e2e/claude_code/test_config.yaml --port 41983
$ curl -s http://localhost:41983/v1/models -H "Authorization: Bearer sk-compat-qa-1234" | jq -r '.data[].id' | sort
claude-haiku-4-5
...
gpt-5-6-luna-azure-openai
gpt-5-6-luna-bedrock-mantle
gpt-5-6-luna-openai
gpt-5-6-sol-azure-openai
gpt-5-6-sol-bedrock-mantle
gpt-5-6-sol-openai
gpt-5-6-terra-azure-openai
gpt-5-6-terra-bedrock-mantle
gpt-5-6-terra-openai

All 24 aliases registered and the proxy served traffic normally with AZURE_API_BASE/AZURE_API_KEY unset

Flagless (the default): the openai and azure_openai cells run live like any other column, the bedrock_mantle cells skip and publish as not_tested, and the Vertex column records its static not_applicable. The real claude CLI (2.1.211) fanned all three GPT-5.6 tiers out through the proxy for all four feature rows and every tier answered

$ LITELLM_PROXY_URL=http://localhost:57391 LITELLM_MASTER_KEY=sk-... \
  uv run pytest tests/e2e/claude_code/basic_messaging_non_streaming/test_{openai,azure_openai,bedrock_mantle}.py \
  tests/e2e/claude_code/basic_messaging_streaming/test_{openai,azure_openai,bedrock_mantle}.py \
  tests/e2e/claude_code/tool_use/test_{openai,azure_openai,bedrock_mantle}.py \
  tests/e2e/claude_code/tool_use_streaming/test_{openai,azure_openai,bedrock_mantle}.py -q
8 passed, 4 skipped in 35.20s
[compat] per-provider breakdown:
  azure_openai         pass=12
  openai               pass=12

COMPAT_MANTLE_CELLS=1: the mantle cells run and fail at the account boundary. The request reaches Bedrock end to end; SigV4 auth succeeds and the model id resolves before the allowlist denies it, so the translation wiring is proven up to the credential boundary

$ curl -s http://localhost:41983/v1/chat/completions -H "Authorization: Bearer sk-compat-qa-1234" \
  -d '{"model":"gpt-5-6-luna-bedrock-mantle","messages":[{"role":"user","content":"say pong"}],"max_tokens":512}'
{"error":{"message":"litellm.APIConnectionError: Bedrock_mantleException - {\"error\":{\"code\":\"access_denied\",
\"message\":\"openai.gpt-5.6-luna is not available for this account. ...\"}}"}}

This credential gap is exactly why the mantle cells ship behind the COMPAT_MANTLE_CELLS=1 opt-in, and the runbook below lists the green-path commands for once the account is allowlisted

Regression check on the existing columns through the same modified config:

$ curl -s http://localhost:41983/v1/messages -H "Authorization: Bearer sk-compat-qa-1234" \
  -d '{"model":"claude-haiku-4-5","max_tokens":32,"messages":[{"role":"user","content":"Reply with the single word pong"}]}'
{"model":"claude-haiku-4-5","id":"msg_011Cd4hr...","content":[{"type":"text","text":"pong"}],"stop_reason":"end_turn",...}

Suite unit tests (layout pins, Bash scanner, rate limiter, matrix builder) passed at capture time with 384 passed in 0.66s. Since then #33578 removed every _*_unit_tests/ tree from tests/e2e/ (the tree is live proxy suites only now), so merging the base branch dropped this PR's unit-pin additions along with them; the live cells above are the tests that remain

Type

✅ Test

Changes

Adds GPT-5.6 (Sol, Terra, Luna; GA 2026-07-09) coverage to the Claude Code compatibility matrix across OpenAI plus the big three clouds. Four new provider columns follow the five v0 Claude columns in manifest.yaml: openai (openai/gpt-5.6-), azure_openai (azure/gpt-5.6-), bedrock_mantle (bedrock_mantle/openai.gpt-5.6-*, the Responses API endpoint Bedrock serves these models on), and vertex_ai_gpt. GCP is the one big-three cloud that does not offer the closed-weight GPT-5.6 family (Model Garden only carries the open-weight gpt-oss MaaS models), so the vertex_ai_gpt cells report a static not_applicable rather than leaving a silent hole in the matrix

The new columns back the four core rows that define "Claude Code works": basic_messaging_non_streaming, basic_messaging_streaming, tool_use, and tool_use_streaming. Each live cell fans out all three tiers via the existing driver and only goes green when Sol, Terra, and Luna all pass, mirroring the three-Claude-tier rule. Remaining rows render not_tested for the GPT columns and can be filled in later

The openai and azure_openai cells run unconditionally: every environment that runs the suite now carries OPENAI_API_KEY with quota and AZURE_API_BASE/AZURE_API_KEY pointing at a resource with the three gpt-5.6 deployments. Only the bedrock_mantle cells are opt-in, via COMPAT_MANTLE_CELLS=1, and skip otherwise, publishing as not_tested; that keeps the published matrix free of credential-shaped red cells until the AWS account clears the Bedrock Mantle allowlist for the openai.gpt-5.6-* models, at which point flipping the flag is the only rollout step. The proxy itself boots fine with all of these env vars missing (proven above). The six tool_use cells resolve the proxy through claude_code._env like the basic_messaging cells, so the whole suite reads the same LITELLM_PROXY_URL/LITELLM_MASTER_KEY pair

Supporting changes: nine proxy aliases in test_config.yaml; three new rate limiter buckets with suffix inference (-azure-openai is matched before -openai and -azure so Azure OpenAI traffic cannot drain the wrong bucket); run_compat.sh env documentation. Earlier revisions added a claude_code entry to the tests/e2e/CLAUDE.md suite list, since superseded by the base branch's own entry, and also extended the suite's layout pins and the Bash allow-rule scanner with unit tests, but those trees were removed from tests/e2e/ by #33578, so the merge with the base branch drops those additions

QA runbook

Environment prerequisites: a proxy booted from tests/e2e/claude_code/test_config.yaml; OPENAI_API_KEY with available chat quota; AZURE_API_BASE and AZURE_API_KEY pointing at a resource whose deployments are named exactly gpt-5.6-sol, gpt-5.6-terra, and gpt-5.6-luna (LiteLLM sends the segment after azure/ in test_config.yaml as the deployment name, so a resource using different deployment aliases 404s); for the mantle cells only, COMPAT_MANTLE_CELLS=1 plus AWS credentials for an account with Bedrock Mantle access to the three openai.gpt-5.6-* models in us-east-1. Provisioning status as of 2026-07-16: the OpenAI org has quota again (rotated key) and the openai column passes 4/4 cells live (Sol, Terra, and Luna on every feature row); the azure_openai column passes 4/4 cells (12/12 tier passes) against a resource with the three deployments, using the same env names and values the suite runners already carry, so neither ungated column needs any new provisioning; bedrock_mantle remains blocked on the AWS sales allowlist for the openai.gpt-5.6-* models. The <alias> placeholders below run once per tier: gpt-5-6-{sol,terra,luna}-openai, -azure-openai, or -bedrock-mantle depending on the column under test

• tests/e2e/claude_code/basic_messaging_non_streaming/test_openai.py::test_basic_messaging_non_streaming_openai (same shape for ::test_basic_messaging_non_streaming_azure_openai and ::test_basic_messaging_non_streaming_bedrock_mantle) - Claude Code gets a plain text reply from each GPT-5.6 tier through the proxy
  • ANTHROPIC_BASE_URL=http://localhost:4000 ANTHROPIC_AUTH_TOKEN=sk-1234 claude --model <alias> -p "Reply with the single word 'pong' and nothing else." --output-format stream-json --verbose
  • Expect exit code 0 and a non-empty assistant text record for each of the three tiers
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/claude_code/basic_messaging_streaming/test_openai.py::test_basic_messaging_streaming_openai (same shape for ::test_basic_messaging_streaming_azure_openai and ::test_basic_messaging_streaming_bedrock_mantle) - the proxy re-emits the upstream stream incrementally instead of buffering it
  • ANTHROPIC_BASE_URL=http://localhost:4000 ANTHROPIC_AUTH_TOKEN=sk-1234 claude --model <alias> -p "Count from 1 to 5, one number per line." --output-format stream-json --include-partial-messages --verbose
  • Expect at least 2 stream_event records carrying SSE payloads (message_start, content_block_delta, message_stop and friends); a buffering proxy collapses these to 0 or 1
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/claude_code/tool_use/test_openai.py::test_tool_use_openai (same shape for ::test_tool_use_azure_openai and ::test_tool_use_bedrock_mantle) - GPT-5.6 tool calls round-trip to Anthropic tool_use blocks
  • ANTHROPIC_BASE_URL=http://localhost:4000 ANTHROPIC_AUTH_TOKEN=sk-1234 claude --model <alias> -p "Use the Bash tool to run the command `echo pong` and report what it printed." --allowed-tools "Bash(echo pong)" --permission-mode dontAsk --output-format stream-json --verbose
  • Expect an assistant event whose content contains a tool_use block for each tier
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/claude_code/tool_use_streaming/test_openai.py::test_tool_use_streaming_openai (same shape for ::test_tool_use_streaming_azure_openai and ::test_tool_use_streaming_bedrock_mantle) - streamed tool arguments arrive as incremental input_json_delta events
  • Run the tool_use command above with --include-partial-messages added
  • Expect a tool_use block plus at least one content_block_delta stream event whose delta type is input_json_delta; zero deltas means the proxy collapsed the streamed tool input
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/claude_code/basic_messaging_non_streaming/test_vertex_ai_gpt.py::test_basic_messaging_non_streaming_vertex_ai_gpt (and the matching stub in the other three feature directories) - the Vertex GPT column documents that GCP does not offer GPT-5.6
  • Run the pytest node with no GPT credentials and no flags; it needs no network
  • Expect 1 passed and a compat-results.json row with status not_applicable and the reason naming Vertex AI's missing GPT-5.6 offering
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

This PR adds GPT-5.6 (Sol, Terra, Luna) coverage to the Claude Code compatibility matrix for four provider columns: openai, azure_openai, bedrock_mantle, and a static vertex_ai_gpt stub. Bedrock Mantle cells are opt-in via COMPAT_MANTLE_CELLS=1 pending AWS allowlist approval, and Vertex AI is documented as not_applicable since GCP does not offer the closed-weight GPT-5.6 family.

• Adds nine proxy aliases to test_config.yaml and four columns to manifest.yaml, backed by twelve new test files across four feature directories.
• Extends rate_limiter.py with three new provider buckets; the -azure-openai suffix is correctly checked before -openai so Azure traffic cannot drain the wrong bucket.
• Introduces _gpt_cells.py as a shared home for the Mantle skip gate and the Vertex not-applicable reason string.

Confidence Score: 5/5

Safe to merge — all changes are confined to the e2e test suite and touch no production code paths.

The rate limiter suffix ordering fix is correct, the Bedrock Mantle opt-in gate prevents credential-shaped red cells in CI, and the proxy config additions are consistent with the existing alias pattern. No production code is touched.

No files require special attention; the only note is duplicated private helpers across the new tool_use and tool_use_streaming test files that could live in _gpt_cells.py.

Important Files Changed

Filename	Overview
tests/e2e/claude_code/_gpt_cells.py	New shared module with skip helper for Bedrock Mantle opt-in gate and the Vertex AI not-applicable reason string; clean and minimal.
tests/e2e/claude_code/rate_limiter.py	Adds three new provider buckets (openai, azure_openai, bedrock_mantle) with correct suffix ordering in infer_provider — azure-openai is checked before openai to prevent bucket misrouting.
tests/e2e/claude_code/test_config.yaml	Adds nine proxy model aliases for GPT-5.6 Sol/Terra/Luna across OpenAI, Azure OpenAI, and Bedrock Mantle; all env vars follow the convention already used by the suite.
tests/e2e/claude_code/manifest.yaml	Appends four new provider columns to the matrix; order is preserved and each new column is explained in comments.
tests/e2e/claude_code/tool_use/test_openai.py	New tool_use cell for OpenAI GPT-5.6; _has_tool_use_event helper is copy-pasted identically in five other new files (test_azure_openai, test_bedrock_mantle in tool_use and tool_use_streaming).
tests/e2e/claude_code/tool_use_streaming/test_openai.py	New tool_use_streaming cell for OpenAI; both _has_tool_use_event and _count_input_json_deltas are duplicated across three tool_use_streaming files and _has_tool_use_event across three tool_use files as well.
tests/e2e/claude_code/basic_messaging_non_streaming/test_bedrock_mantle.py	Bedrock Mantle basic messaging cell; skip gate is correctly placed before any network call.
tests/e2e/claude_code/run_compat.sh	Adds three new providers to the rates display loop and documents COMPAT_MANTLE_CELLS in the header comment.

_{Reviews (5): Last reviewed commit: "test(e2e/claude_code): update run_compat..." | Re-trigger Greptile}

[greptile-apps]

Azure OpenAI deployment names are implicit — LiteLLM infers the deployment name from the segment after azure/ (e.g., gpt-5.6-sol), so the Azure resource must have deployments named exactly gpt-5.6-sol, gpt-5.6-terra, and gpt-5.6-luna. This is not documented in the config block or the runbook. If the provisioned resource uses different deployment names (e.g., a custom alias like gpt56sol), the proxy will silently send a request to the wrong deployment or return a 404. Consider adding a comment here or in the runbook specifying the expected deployment names to reduce operator error when AZURE_OPENAI_API_BASE is first configured.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_claude_code_e2e_gpt_big3 (a017b95) with litellm_internal_staging (68f0fb0)}

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[coderabbitai]

Caution

Review failed

An error occurred during the review process. Please try again later.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch litellm_claude_code_e2e_gpt_big3

---

_{Comment @coderabbitai help to get the list of available commands.}

[mateo-berri]

@greptileai

[mateo-berri]

every legacy tool_use cell already carries its own _has_tool_use_event. The new cells just match that convention, so deduping belongs in a follow-up, not this PR. I will create a follow-up with this change

[mateo-berri]

@greptileai

[yucheng-berri]

• Stale docstring refs: the vertex stubs in tool_use/ and tool_use_streaming/ reference _pr_gate_unit_tests/test_bash_tool_restrictions.py, which the PR body itself says feat(e2e): emit structured E2E_RESULT lines for package status history #33578 deleted from tests/e2e/. Worth a cleanup commit.

[yucheng-berri]

tests/e2e/claude_code/basic_messaging_non_streaming/test_vertex_ai_gpt.py::test_basic_messaging_non_streaming_vertex_ai_gpt (and the matching stub in the other three feature directories) - the Vertex GPT column documents that GCP does not offer GPT-5.6
Run the pytest node with no GPT credentials and no flags; it needs no network
Expect 1 passed and a compat-results.json row with status not_applicable and the reason naming Vertex AI's missing GPT-5.6 offering
Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Why do we need an e2e for this?