test(e2e): drop vertex from pipecat tool smoke and remove key models dropdown suite - #32838
Conversation
Point batch deployments at the credential field names and os.environ refs the gateway actually resolves from process env (compose .env or EKS secret mounts). Missing secrets skip instead of failing red so a red run means a product bug. Mirror S3 bucket env aliases in docker-compose for provider_fallback
The batches suite is live e2e only; no monkeypatch or unit-level tests
Keep object-storage fields through CredentialLiteLLMParams and resolve
os.environ/ refs when reading deployment credentials so Vertex/Bedrock
batch file uploads see bucket and AWS keys from K8s/docker env
Skip managed batch list when the request is provider-scoped so
/{provider}/v1/batches list works instead of 500
Force DB on check_db_only team lookups and stop masking non-404 errors
as "team doesn't exist"
Drop e2e runner-side skip helpers; hard-fail on missing gateway secrets
Reseed spend:tag counters from LiteLLM_TagTable so cold redis still enforces after the spend writer flushes When applying post-call cost to team multi-window counters, load the team from the DB if it is missing from the management cache so window spend is not dropped on cache misses Harden cold-counter reseed e2e (namespace-aware keys, burst success, poll). Give tag budget more headroom. Retry /key/update on redis DNS blips. Ensure NLTK punkt_tab is present for pipecat realtime audio
Reverts all litellm/ and unit-test product edits. This branch is limited to tests/e2e per contributor instruction
provider_fallback list falls back when managed batches reject provider filtering. Team create waits for /team/info and member_add retries on transient team-not-found so split control-plane lag does not red the suite
Leave local .env and docker-compose env wiring as the secret source
OpenAI/Azure batch file uploads need files_settings; budget reset e2e needs a short rescheduler window. Drop unsupported bedrock-encoded create_batch cells, tolerate bedrock file.bytes=0, and surface team-info wait failures instead of hanging silently
When provider-scoped list is rejected, still fetch the unfiltered list and check the envelope. Only skip membership when the id is a raw provider_fallback batch that managed list cannot index
Stage and local proxy runs have no Admin UI driver; these tests only produced setup timeouts and were out of scope for the API e2e gate
Raw-websocket tool_call_round_trip already covers tool calling through the proxy for every provider, including vertex. The pipecat tool smoke only asserts coarse callback/text signals and is known flaky upstream (pipecat-ai/pipecat#2544); stage runs fail pipecat for vertex while raw-ws passes, so the smoke adds noise without a LiteLLM signal
Restore the pipecat tool smoke suite for openai/azure/gemini. Vertex native-audio live remains covered by raw-ws tool_call_round_trip; pipecat tool calling for that provider is flaky upstream and reds the stage suite
|
Superseded by a clean branch off litellm_internal_staging; this PR re-listed already-merged commits from #32744 |
Greptile SummaryThis PR hardens the e2e test suite in three main ways: it drops
Confidence Score: 4/5Safe to merge for the pipecat and dropdown removals; the batch capabilities changes in capabilities.py warrant a second look before merging. The Bedrock encoded routing scenario is dropped from the batch test matrix without any explanation of why it was removed or whether it was previously passing. This creates a gap in regression coverage for the provider-fallback Bedrock path. Everything else is well-motivated and correctly implemented. tests/e2e/batches/capabilities.py (BEDROCK_SCENARIOS narrowing) and tests/e2e/batches/test_batches_e2e.py (assert_file_object bytes weakening for Bedrock)
|
| Filename | Overview |
|---|---|
| tests/e2e/llm_translation/realtime/test_realtime_pipecat_e2e.py | Drops vertex_ai from PROVIDER_PARAMS with clear justification referencing upstream pipecat-ai/pipecat#2544; raw-ws coverage for vertex is preserved. |
| tests/e2e/batches/capabilities.py | Adds AWS/GCS credentials to provider params and introduces _env_ref helper; silently narrows BEDROCK_SCENARIOS from ("encoded", "unified") to ("unified",) without justification in PR description. |
| tests/e2e/batches/test_batches_e2e.py | Adds provider-aware file.bytes assertion and managed-filter fallback for list_batches; weakens the bytes>0 guard for Bedrock without explaining the upstream limitation. |
| tests/e2e/budgets/budget_client.py | Adds _wait_for_team polling and retry loop on add_team_member to harden team creation race conditions. |
| tests/e2e/budgets/test_budget_reset_advances_e2e.py | New regression-guard suite for #25109; well-structured progressive rungs covering scheduling, enforcement, timestamp advancement, multi-window independence, team-member windows, and the non-5xx error path. |
| tests/e2e/management/management_client.py | Adds _wait_for_team, retry on Redis-connectivity errors in update_key_models, and retry on "doesn't exist" in add_team_member for reliability. |
| tests/e2e/management/test_key_models_dropdown_e2e.py | Entire Playwright UI test suite deleted to unblock stage; removes 4 tests covering teamless/team key create and edit dropdown scoping. |
| tests/e2e/docker-compose.yml | Adds proxy_budget_rescheduler min/max time settings, files_settings for OpenAI/Azure, and passes through all required cloud provider env vars to the container. |
Comments Outside Diff (2)
-
tests/e2e/batches/test_batches_e2e.py, line 130-134 (link)file.bytes > 0assertion weakened for Bedrock without upstream citationThe check is now skipped for
provider == "bedrock", allowingfile.bytes == 0to pass silently. If the Bedrock Files API genuinely returns0forbyteson a non-empty upload, a comment citing the upstream behaviour would make it clear this is adapting to a real limitation rather than masking a regression.Rule Used: What: Flag any modifications to existing tests and... (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
-
tests/e2e/budgets/budget_client.py, line 240-255 (link)_wait_for_team/_TEAM_READY_*constants duplicated inmanagement_client.pyIdentical constants and an identical
_wait_for_teammethod appear verbatim intests/e2e/management/management_client.py. Both modules already share helpers frome2e_httpande2e_gateway, so extracting this polling logic into a shared utility would prevent the two copies drifting.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Reviews (1): Last reviewed commit: "test(e2e): drop only vertex from pipecat..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| @@ -84,14 +91,6 @@ def id(self) -> str: | |||
|
|
|||
| @property | |||
| def jsonl_model(self) -> str: | |||
There was a problem hiding this comment.
BEDROCK_SCENARIOS previously ran both "encoded" and "unified" paths for Bedrock. Dropping "encoded" removes coverage of the provider-fallback routing scenario for Bedrock — the path that returns a raw (non-re-encoded) batch ID and exercises a different ID-shape branch in raw_id_matches_provider. The PR description and COVERAGE.md note the change ("yes (unified only)") but give no reason why the encoded path was removed. If it was failing, the failure mode should be documented; if it was passing, removing it weakens the regression guarantee for that routing path.
Rule Used: What: Flag any modifications to existing tests and... (source)
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
1 participant
Relevant issues
Linear ticket
Pre-Submission checklist
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Stage e2e log
Logs-2026-07-10 15_39_37.txt(pre this PR image):test_tool_call_round_trip[vertex_ai]PASSED (raw GA websocket path through LiteLLM)test_pipecat_tool_smoke[openai|azure|gemini]PASSEDtest_pipecat_tool_smoke[vertex_ai]FAILED withpipecat did not invoke the get_weather callbackThat is the same pattern as pipecat-ai/pipecat#2544: LiteLLM Vertex -> OpenAI GA transform is fine under raw-ws; pipecat's tool path is flaky for Vertex native-audio live. After this PR, stage should not collect the vertex pipecat case. Rebuild the e2e image on this SHA and re-run the stage suite for green
Type
✅ Test
Changes
Follow-up to #32744. Stage was red solely on
test_pipecat_tool_smoke[vertex_ai]while the raw-websocket Vertex tool round-trip and the other pipecat providers stayed greenDrop
vertex_aifromPROVIDER_PARAMSintest_realtime_pipecat_e2e.pyso pipecat tool smoke only runs openai, azure, and gemini. Vertex realtime coverage stays on unit transform tests plus raw-wstest_text_conversation/test_tool_call_round_trip. Document that split inREALTIME_COVERAGE_MATRIX.mdAlso remove the Playwright
test_key_models_dropdown_e2e.pysuite so the stage job is not blocked by that UI harness