fix(e2e): clear the three standing errors in the scheduled Buildkite suite

[yuneng-berri]

TLDR

Problem this solves:

• Every scheduled Buildkite e2e build since Allow vertex AI to authenticate using GOOGLE_APPLICATION_CREDENTIALS environment variable #230 ends with the same 5 errors
• test_outbound_http2_e2e.py cannot import hypercorn in the runner image
• The team access-group fixture polls a 403 message fix(proxy): hide model allowlist from client-facing model access denied errors #41310 replaced, so 2 tests error at setup
• The vertex batch lifecycle cleanup deletes raw gs:// ids with a virtual key and gets 403

How it solves it:

• Moves the HTTP/2 check into a new sdk shard under tests/integration, the CircleCI tier for SDK clients against local protocol peers
• Reads the team allow-list back from /team/info instead of parsing the denial text
• Deletes raw cloud-storage file ids with the master key, which is the only key allowed to

User Flow

Before: an engineer opens the scheduled e2e build and finds the same 5 errors every 6 hours, none of them about the code under test

1. They open https://buildkite.com/berriai-1/litellm-e2e/builds/241 and the pytest summary ends with 1263 passed, 60 skipped, 5 errors
2. llm_translation/test_outbound_http2_e2e.py shows ModuleNotFoundError: No module named 'hypercorn'
3. Both TestTeamScopedToAccessGroup tests show the team's allow-list never settled to models=[...], with a 403 whose message no longer lists any models
4. Both test_batch_lifecycle[vertex_ai-*] cases pass and then error at teardown with Delete file gs://... failed: HTTP 403

After: the same build reports the suite's real outcome

1. They open the build at this branch's tip and the pytest summary carries none of the 5 errors
2. llm_translation/test_outbound_http2_e2e.py is gone from the e2e suite; its two checks run in the integration-sdk CircleCI job on main
3. Both TestTeamScopedToAccessGroup tests run once /team/info shows the allow-list holding only the access group
4. Both test_batch_lifecycle[vertex_ai-*] cases clean up their gs:// input file with the master key and pass end to end

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both runs are the real litellm-e2e Buildkite pipeline: a fresh ephemeral stack deployed from the litellm SHA, the full suite against live providers

Before (4b368bf)

1. Scheduled build https://buildkite.com/berriai-1/litellm-e2e/builds/241 at the merge base
2. Pytest summary: 1263 passed, 60 skipped, 30 warnings, 5 errors in 1951.59s
3. The 5 errors, verbatim from the short test summary:

ERROR llm_translation/test_outbound_http2_e2e.py - ImportError while importing test module '/app/e2e/llm_translation/test_outbound_http2_e2e.py'.
E   ModuleNotFoundError: No module named 'hypercorn'
ERROR access_control/test_model_access_group_e2e.py::TestTeamScopedToAccessGroup::test_group_grants_the_teams_own_wildcard - Failed: the team's allow-list never settled to models=['e2e-agt-c0813804c6cb']; last denial read {"error":{"message":"The requested model 'gpt-5.2' is not available for this API key, or the model name is invalid. Check the models available to you and try again.","type":"team_model_access_denied","param":"model","code":"403"}}
ERROR access_control/test_model_access_group_e2e.py::TestTeamScopedToAccessGroup::test_group_grants_the_team_nothing_outside_it - Failed: the team's allow-list never settled to models=['e2e-agt-c0813804c6cb']; last denial read {"error":{"message":"The requested model 'gpt-5.2' is not available for this API key, or the model name is invalid. Check the models available to you and try again.","type":"team_model_access_denied","param":"model","code":"403"}}
ERROR batches/test_batches_e2e.py::test_batch_lifecycle[vertex_ai-model_param] - AssertionError('Delete file gs://litellm_bucket-16/litellm-vertex-files/publishers/google/models/gemini-2.5-flash/771f5b7d-7ac7-4c60-a30a-4bdeafa581c1 failed: HTTP 403') [single exception in ExceptionGroup]
ERROR batches/test_batches_e2e.py::test_batch_lifecycle[vertex_ai-provider_fallback] - AssertionError('Delete file gs://litellm_bucket-16/litellm-vertex-files/publishers/google/models/gemini-2.5-flash/a9a25589-46cf-4610-89c6-93fa08a35d3f failed: HTTP 403') [single exception in ExceptionGroup]

After (fa01e2d)

1. No Buildkite build at this branch's tip by request; the e2e-side changes are byte-identical to the ones build https://buildkite.com/berriai-1/litellm-e2e/builds/242 is running, and that build's summary lands here when it finishes
2. The HTTP/2 shard through the tier's own runner locally, which also enforces the contracts manifest:

$ .venv/bin/python tests/integration/run.py sdk --results /tmp/integration-sdk
tests/integration/sdk/test_http2_wire.py::test_async_handler_negotiates_http2_only_when_enabled PASSED
tests/integration/sdk/test_http2_wire.py::test_sync_handler_negotiates_http2_only_when_enabled PASSED
2 passed in 1.42s
$ jq .complete /tmp/integration-sdk/execution.json
true

3. Mutation check: with http2=http2_enabled() replaced by http2=False (and separately http2=True) in http_handler.py, both nodes fail on observed.post_version; restored afterwards
4. The runner image cannot import litellm at all: replicating the image's exact uv sync --frozen --no-install-workspace --extra proxy --extra cli --group e2e-dev into a scratch venv and running python -c 'import litellm' from outside the checkout raises ModuleNotFoundError, which is why the fix is a move and not a dependency pin
5. circleci config validate .circleci/config.yml reports the config valid with the new matrix entry

Type

✅ Test

Caveats (if any)

Medium

• The allow-list settle is now a control-plane read-back; nothing on the data plane exposes the list any more
  • The data plane first fetches the team at the test's own chat call, after the reset, so a stale cached copy is not expected; if that assumption is wrong the first team test can pass through the directly listed wildcard instead of the group

Low

• The integration CircleCI workflow only runs on main, so the new integration-sdk job first executes after merge; the local run above through run.py is the same command the job runs
• The shard reuses run_integration.sh unchanged, so its job still boots Postgres, Redis and a proxy it never talks to
• A non-admin key that uploads through provider=vertex_ai gets a raw gs:// id it can never delete; that is the guard's intended behavior, the suite now just cleans up as an admin

QA runbook

• tests/e2e/access_control/test_model_access_group_e2e.py::TestTeamScopedToAccessGroup::test_group_grants_the_teams_own_wildcard - a team whose allow-list is one access group can call a bare model name that the group's team-scoped wildcard deployment covers
  • POST /team/new with the master key and {"team_alias": "ag-team", "models": ["ag-group"]}; note the team_id
  • POST /key/generate with {"models": [], "team_id": "<team_id>"}; note the key
  • POST /model/new with {"model_name": "openai/gpt-5.6*", "litellm_params": {"model": "openai/gpt-5.6*", "api_key": "<OPENAI_API_KEY>"}, "model_info": {"team_id": "<team_id>", "access_groups": ["ag-group"]}}
  • GET /team/info?team_id=<team_id> and expect models to read ["ag-group", "openai/gpt-5.6*"]
  • POST /team/update with {"team_id": "<team_id>", "team_alias": "ag-team", "models": ["ag-group"]}, then GET /team/info until models reads exactly ["ag-group"]
  • POST /v1/chat/completions with the team key and "model": "gpt-5.6-luna"; expect 200 with a real completion
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/access_control/test_model_access_group_e2e.py::TestTeamScopedToAccessGroup::test_group_grants_the_team_nothing_outside_it - the same team is denied a model outside the group
  • Same setup as above, through the settled /team/info read
  • POST /v1/chat/completions with the team key and "model": "e2e-ag-unknown-1"; expect 403 with "type": "team_model_access_denied"
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/batches/test_batches_e2e.py::test_batch_lifecycle[vertex_ai-model_param] and [vertex_ai-provider_fallback] - a vertex batch created from a raw gs:// input file runs, and the cleanup deletes that file as an admin (needs VERTEXAI_PROJECT, VERTEXAI_CREDENTIALS, GCS_BUCKET_NAME)
  • POST /vertex_ai/v1/files with a virtual key, purpose=batch and a one-line JSONL; expect a gs://<bucket>/litellm-vertex-files/... id
  • POST /v1/batches with that id and "model": "<vertex batch deployment>"; expect a batch object
  • DELETE /vertex_ai/v1/files/ with the virtual key; expect 403 Raw cloud storage file ids can only be deleted by a proxy admin key
  • DELETE /vertex_ai/v1/files/ with the master key; expect 200 with "deleted": true
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/integration/sdk/test_http2_wire.py - the SDK's sync and async httpx handlers negotiate HTTP/2 only when LITELLM_HTTP2 is set, verified by both the client and a local hypercorn TLS peer
  • .venv/bin/python tests/integration/run.py sdk --results /tmp/integration-sdk; expect 2 passed and complete: true in /tmp/integration-sdk/execution.json
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge with no outstanding correctness or repository-rule failures

Summary

This PR repairs three recurring scheduled e2e failures and moves the local HTTP/2 wire test into a dedicated integration shard

• Reads team allow-list state from /team/info
• Uses admin credentials to clean up raw cloud-storage file IDs
• Adds and registers a CircleCI SDK integration shard for HTTP/2 negotiation
• Removes the newly added mock cleanup cases from the e2e test tree

_{Reviews (2) · Last reviewed commit: "fix(e2e): delete raw cloud-storage batch..."}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_/buildkite-241-triage-4a48cf (52308a8) with main (4b368bf)}