test(claude_code): move the Claude Code compatibility matrix under tests/e2e

[devin-ai-integration]

Relevant issues

Supersedes #28027 (and its predecessor #26491) by relocating the whole Claude Code Compatibility Matrix v0 from tests/claude_code/ to tests/e2e/claude_code/, where e2e suites now live. Parent PRD: #26476

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live before/after demonstration, no mocks: the before commands ran at the merge-base commit 528fa38 of litellm_internal_staging (where neither tests/claude_code/ nor tests/e2e/claude_code/ exists) and the after commands ran on this branch's head 5787a28, with a local proxy booted from this branch's code hitting the real Anthropic API with real spend

Before (merge-base 528fa38)

The suite exists at neither the old nor the new path, so the resolver module and the pytest cells both fail:

$ ls tests/e2e/claude_code tests/claude_code
ls: tests/claude_code: No such file or directory
ls: tests/e2e/claude_code: No such file or directory

$ env PYTHONPATH="$PWD/tests/e2e" uv run --no-sync python -m claude_code.pr_gate_version_resolver
.venv/bin/python3: Error while finding module specification for 'claude_code.pr_gate_version_resolver' (ModuleNotFoundError: No module named 'claude_code')

$ uv run --no-sync pytest tests/e2e/claude_code/basic_messaging_non_streaming/test_anthropic.py \
    tests/e2e/claude_code/basic_messaging_streaming/test_anthropic.py \
    tests/e2e/claude_code/count_tokens/test_anthropic.py -q
ERROR: file or directory not found: tests/e2e/claude_code/basic_messaging_non_streaming/test_anthropic.py

no tests ran in 0.00s

After (head 5787a28)

The resolver selects a version end-to-end against the real npm registry from its new module path:

$ env PYTHONPATH="$PWD/tests/e2e" uv run --no-sync python -m claude_code.pr_gate_version_resolver
pr_gate_version_resolver: selected @anthropic-ai/claude-code@2.1.207
2.1.207

That exact version was installed into an isolated npm prefix and put first on PATH, and the proxy was booted from this branch with the suite's own routing config:

$ npm install -g --prefix "$SCRATCH/claude-cli" @anthropic-ai/claude-code@2.1.207
$ claude --version
2.1.207 (Claude Code)

$ uv run python litellm/proxy/proxy_cli.py --config tests/e2e/claude_code/test_config.yaml --port 4000 --detailed_debug

$ curl -s http://localhost:4000/v1/models -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -r '.data[].id' | sort
claude-haiku-4-5
claude-haiku-4-5-azure
claude-haiku-4-5-bedrock-converse
claude-haiku-4-5-bedrock-invoke
claude-haiku-4-5-vertex
... (15 aliases total, same for sonnet-4-6 and opus-4-7)

The Anthropic column then runs live: the two messaging cells drive the real claude binary as a subprocess across all three tiers in parallel, and count_tokens probes the proxy over HTTP:

$ export LITELLM_PROXY_BASE_URL=http://localhost:4000 LITELLM_PROXY_API_KEY=$LITELLM_MASTER_KEY
$ uv run --no-sync pytest tests/e2e/claude_code/basic_messaging_non_streaming/test_anthropic.py \
    tests/e2e/claude_code/basic_messaging_streaming/test_anthropic.py \
    tests/e2e/claude_code/count_tokens/test_anthropic.py -q
...                                                                      [100%]
[compat] session totals:
  pass             9
[compat] per-provider breakdown:
  anthropic            pass=9

3 passed in 11.95s

And the exact end-user path, the pinned CLI pointed at the live proxy from a fresh HOME:

$ env -i PATH="$PATH" HOME="$(mktemp -d)" ANTHROPIC_BASE_URL=http://localhost:4000 \
    ANTHROPIC_AUTH_TOKEN=$LITELLM_MASTER_KEY claude -p "Reply with exactly: compat-ok" --model claude-sonnet-4-6
compat-ok

The proxy's debug log confirms every request went upstream to the real Anthropic API, carrying the CLI's own wire shape:

POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/messages \
-H 'x-claude-code-session-id: 6e7739fc-...' -H 'anthropic-beta: context-management-2025-06-27,effort-2025-11-24,interleaved-thinking-2025-05-14,prompt-caching-scope-2026-01-05,structured-outputs-2025-11-13' -H 'x-app: cli' -H 'anthropic-version: 2023-06-01' ...

The azure/bedrock/vertex columns are deliberately not part of this proof; their pre-existing compat regressions are documented under Changes and are out of scope for a relocation. The tiers stay pinned at haiku-4-5 / sonnet-4-6 / opus-4-7 per caveat 4 under Changes. For full-matrix evidence, the whole suite also ran live on CircleCI build 2022757 at commit ef31a91 before the gate was dropped: 37 of 65 cells passed with real provider calls, and the 28 failures match the pre-existing signatures from the original branch (build 1700813), which failed 37

Type

🚄 Infrastructure
✅ Test

Changes

This moves the compat matrix from #28027 (which now has merge conflicts against litellm_internal_staging) into tests/e2e/claude_code/, rebased onto the current staging head. Everything from the original PR is preserved except the CircleCI PR gate; the changes below are the mechanical relocation plus fixes it forced

Per Mateo's direction, the claude_code_compat_pr_gate CircleCI job from the original PR is NOT carried over: the matrix should run only in the scheduled e2e suite, not per-PR in CircleCI. The job definition, its workflow wiring, and _pr_gate_unit_tests/test_circleci_pr_gate_wiring.py (which pinned that wiring) are dropped. The version resolver module and its unit tests are kept since the scheduled runner needs the same 3-day publish-age selection

Relocation mechanics

• tests/claude_code/ -> tests/e2e/claude_code/; all literal tests/claude_code path references in run_daily.sh, run_compat.sh, the systemd unit, docstrings, and golden fixtures updated
• imports change from from tests.claude_code.X to from claude_code.X. tests/e2e has no __init__.py (adding one would break the harness's import e2e_config rootdir imports), so pytest inserts tests/e2e on sys.path via the claude_code package root and the imports resolve the same way in the cron worktree and local runs
• non-pytest entrypoints that relied on the old tests. package path got explicit wiring: python -m claude_code.pr_gate_version_resolver needs PYTHONPATH=$PWD/tests/e2e, and cron_vm/build_matrix.py bootstraps sys.path before importing the builder
• REPO_ROOT = Path(__file__).resolve().parents[3] becomes parents[4] in the _pr_gate_unit_tests/ and _publisher_unit_tests/ structural tests (the tree is one level deeper now); the "tests" / "claude_code" path joins there gain the "e2e" segment
• run_daily.sh's shim step copies the dev tree into ${WORKTREE}/tests/e2e/ (and mkdir -ps it, since a resolved stable tag may predate tests/e2e)
• tests/e2e/CLAUDE.md gains the required suite-folder line describing claude_code/

Verification: all structural/unit tests under _driver_unit_tests/, _builder_unit_tests/, _publisher_unit_tests/, and _pr_gate_unit_tests/ pass after the move, pytest tests/e2e/ --collect-only collects everything with zero errors under the harness's --strict-markers --strict-config ini, and python -m coverage_registry.collector --strict still passes

Caveats I could not fix in this move

1. the live run at build 2022757 shows real pre-existing compat regressions the scheduled suite will keep reporting red until fixed: Bedrock Converse rejecting requests with "Content block is not a text block" (a proxy translation regression on tool/thinking/vision/caching cells), every Vertex cell timing out with http_probe surfacing the root cause ("Unable to load vertex credentials from environment... Parse error: JSONDecodeError", i.e. the VERTEXAI_* credential env var in the CircleCI project is malformed JSON), tool_use_streaming cells seeing too few stream-json events on several providers (response buffering), and Bedrock Invoke not accepting the tool_search_tool_regex_20251119 tool tag. Fixing those is proxy behavior work and credential repair, out of scope for a relocation

2. the suite intentionally does not use the shared e2e transport harness (transport.py / Gateway / Result unions): it drives the real claude CLI as a subprocess and http_probe.py uses httpx directly. Rewriting 30 cells onto the harness is out of scope for a move; the new CLAUDE.md suite line documents the exception

3. related: the harness's "skip on environment, fail on behavior" contract does not hold for this suite. A bare pytest tests/e2e/ without LITELLM_PROXY_BASE_URL / LITELLM_PROXY_API_KEY hard-fails every live cell by design (the cron wants misconfiguration to surface as red cells, not silent skips). Nothing in CI runs tests/e2e/ wholesale today, but a local dev running the whole tree will hit ~75 failures unless they scope their run or export the env

4. the matrix still pins Haiku 4.5 / Sonnet 4.6 / Opus 4.7 as the three tiers. Opus 4.8 exists in model_prices_and_context_window.json now; bumping the tier touches the manifest, test_config.yaml aliases for all five providers, and the docs schema, and needs live validation of each provider's 4.8 availability, so it is left for a follow-up

5. the feature tests also carry no @pytest.mark.covers(...) markers yet, so the compat cells do not feed the coverage registry despite the registry grammar being seeded from this matrix

6. with the per-PR CircleCI compat gate dropped, the 72-hour publish-age selection in pr_gate_version_resolver.py no longer runs on any automated path: run_daily.sh probes whatever claude is already on the cron VM's PATH and never resolves or installs a version. The resolver and its unit tests stay in-tree so a follow-up can wire it into run_daily.sh behind a checksum-pinned install; until then the VM's claude is operator-managed, and both the version probe and the pytest run already execute under env -i scrubbing as a compensating control

7. the relocated tree is excluded from the zero-error tests/e2e basedpyright gate (tests/e2e/claude_code in pyrightconfig.json exclude): the suite predates that gate and carries 1819 strict-mode errors, mostly untyped fixtures and stream-json event dicts. Typing the suite so the exclusion can be dropped is a follow-up

Link to Devin session: https://app.devin.ai/sessions/725a052e55db4383920ecc5744a544ef
Requested by: @mateo-berri

---

Note

Low Risk
Test and CI/cron infrastructure only; no production proxy or library runtime changes. Main operational risk is cron/scheduled runs depending on correct paths and env scrubbing, which the PR pins with structural tests.

Overview
Relocates the Claude Code compatibility matrix from tests/claude_code/ to tests/e2e/claude_code/, with imports switched to from claude_code.* via tests/e2e on PYTHONPATH, updated cron/systemd/script paths, and tests/e2e/CLAUDE.md documenting the suite. The CircleCI claude_code_compat_pr_gate job and its wiring pin are removed; the PR-gate version resolver and unit tests remain for potential cron use.

The matrix itself is unchanged in behavior: feature×provider pytest cells, compat_result artifacts, matrix_builder, CLI driver with isolated env/HOME, rate limiting, HTTP count_tokens probes, and daily run_daily.sh publishing—plus extensive unit/structural tests and .gitignore entries for compat JSON artifacts. pyrightconfig.json excludes tests/e2e/claude_code from strict checking until the suite is typed.

^{Reviewed by Cursor Bugbot for commit a61aa82. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_claude_code_compat_e2e (a61aa82) with litellm_internal_staging (65ca095)}

[mateo-berri]

@greptile-apps

[mateo-berri]

bugbot run

[greptile-apps]

Greptile Summary

This PR relocates the Claude Code compatibility matrix suite from tests/claude_code/ to tests/e2e/claude_code/, rebasing it onto the current litellm_internal_staging head. No proxy logic is changed; the diff is purely structural relocation plus the mechanical fixes required by the new import root (from claude_code.* instead of from tests.claude_code.*), updated parents[N] depths, a tests/e2e/ shim step in run_daily.sh, and a dropped per-PR CircleCI gate (intentional per the PR description).

• Moves the full v0 matrix stack — cli_driver, conftest, matrix_builder, 13 feature×5-provider pytest cells, http_probe, cron_vm publisher, and all _*_unit_tests/ trees — preserving the security hardening intact (env -i scrubbing for pytest and claude --version, per-invocation isolated HOME in cli_driver.py, InaccessiblePaths= in the systemd unit).
• Import and path wiring is updated throughout: sys.path bootstrap in build_matrix.py uses parents[2] (→ tests/e2e/), publisher/pr-gate structural tests use parents[4] (→ repo root), and the layout-pin tests use parents[1] (→ tests/e2e/claude_code/).
• CLAUDE.md is updated to document the suite's deliberate deviation from the shared e2e transport harness, and pyrightconfig.json excludes the relocated tree from the strict-mode gate (1819 pre-existing type errors, caveat 7).

Confidence Score: 5/5

Safe to merge — this is a pure relocation with no production proxy code changes; all security hardening from the original suite is preserved intact.

The change touches only test infrastructure and cron tooling, not any production proxy code. All import paths, parents[N] depths, shell script path literals, and structural test pins have been updated consistently. The security hardening is preserved and independently pinned by the unit test trees. The only finding is a misleading REPO_ROOT variable name in one structural test file — functionally correct, no runtime impact.

No files require special attention; the misleading REPO_ROOT name in _builder_unit_tests/test_v0_layout.py (line 19) is a cosmetic issue only.

Important Files Changed

Filename	Overview
tests/e2e/claude_code/cron_vm/run_daily.sh	Large cron orchestration script; security hardening is thorough (env -i for pytest and claude --version, isolated CLAUDE_PROBE_HOME, loopback-only proxy, setsid pgroup cleanup, token URL cleanup via rm -rf on exit). Release pagination correctly accumulates all pages before the final sort_by.
tests/e2e/claude_code/cli_driver.py	Thin CLI driver wrapper with per-invocation isolated HOME via _make_isolated_home() and a strict env allowlist; security isolation is well designed.
tests/e2e/claude_code/conftest.py	Pytest plumbing for compat_result fixture, (feature, provider) path inference, and xdist shard merging; logic looks correct.
tests/e2e/claude_code/_builder_unit_tests/test_v0_layout.py	Structural layout tests for the v0 matrix; REPO_ROOT at parents[1] resolves to tests/e2e/claude_code/ rather than the actual repo root — correct for all current uses but inconsistent naming vs the other structural test files in this PR.
tests/e2e/claude_code/_publisher_unit_tests/test_run_daily_pytest_scrubs_env.py	Pins that the cron pytest invocation is wrapped in env -i and excludes provider secrets; parents[4] is correct for the repo root.
tests/e2e/claude_code/_publisher_unit_tests/test_systemd_unit_credential_isolation.py	Pins InaccessiblePaths for credential dotdirs, ReadWritePaths exclusion for .config/gh, and ProtectHome=read-only in the systemd unit; structurally sound.
tests/e2e/claude_code/cron_vm/build_matrix.py	Thin CLI wrapper that bootstraps sys.path with parents[2] (resolves to tests/e2e/) before importing claude_code.matrix_builder; path depth is correct.
pyrightconfig.json	Adds tests/e2e/claude_code to the exclude list; redundant since the include only covers litellm/ but harmless and explicitly intended per caveat 7 in the PR description.
tests/e2e/claude_code/_pr_gate_unit_tests/test_bash_tool_restrictions.py	Pins that every Bash-using cell uses exact-match Bash(echo pong) allow rule and --permission-mode dontAsk; logic in _has_bare_bash_token correctly avoids the short-circuit false-pass bug it documents.

_{Reviews (4): Last reviewed commit: "test(claude_code): assert fine-grained t..." | Re-trigger Greptile}

[cursor]

Cron skips 3-day version gate

Medium Severity

The PR keeps pr_gate_version_resolver so the scheduled runner applies the same 3-day publish-age selection, but run_daily.sh only probes whatever claude is already on PATH and never resolves or installs that version. With the CircleCI PR gate dropped, the security buffer no longer runs on any automated path.

Additional Locations (1)

• tests/e2e/claude_code/pr_gate_version_resolver.py#L1-L20

 

^{Reviewed by Cursor Bugbot for commit 5787a28. Configure here.}

[mateo-berri]

@greptileai

[mateo-berri]

Approving to merge with the documented caveats; review-bot concerns 1-3 fixed in 2a4189f, concern 4 recorded as caveat 6 for a follow-up

[mateo-berri]

bugbot run

[cursor]

Incomplete e2e shim for old tags

Medium Severity

The cron shim only copies claude_code/ into the worktree and mkdir -ps tests/e2e for tags that predate that tree, but it does not provide pytest.ini (or PYTHONPATH=tests/e2e). Suite imports are from claude_code... and rely on tests/e2e being on sys.path via that ini/rootdir. On such tags, pytest collection fails before any cells run, so the daily matrix never builds.

Additional Locations (1)

• tests/e2e/claude_code/cron_vm/run_daily.sh#L394-L408

 

^{Reviewed by Cursor Bugbot for commit 61b5d93. Configure here.}

[mateo-berri]

This is not a concern cuz the suite's from claude_code... imports resolve through pytest's default prepend import mode (the __init__.py chain puts tests/e2e on sys.path automatically), not through pytest.ini or PYTHONPATH, and I tested that a simulated old-tag worktree containing only the shimmed claude_code/ tree collects all 75 cells cleanly with no ini, no e2e conftest, and no strict-marker issue from the root pyproject

[mateo-berri]

bugbot run

[mateo-berri]

@greptileai

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

2 issues from previous reviews remain unresolved.

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a61aa82. Configure here.}