Skip to content

test(e2e): clear the two rc/1.103.0 e2e reds owned by upstream providers - #43388

Merged
yuneng-berri merged 1 commit into
rc/1.103.0from
litellm_rc_1_103_0_fix_e2e_reds
Sep 27, 2026
Merged

yuneng-berri merged 1 commit into
rc/1.103.0from
litellm_rc_1_103_0_fix_e2e_reds

Conversation

@yuneng-berri

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • litellm-e2e build 308 on rc/1.103.0 went red on two things outside LiteLLM's control
  • Azure took over 660s to cancel a batch, so test_batch_lifecycle[azure-unified] errored in teardown
  • DeepWiki renamed its ask_question tool to ask_wiki_question, so the MCP Tools UI test failed

How it solves it:

Relevant issues

Partial backport of #43321

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

tests/e2e/batches/test_batch_cleanup.py passes on this branch (50 tests). DeepWiki's live tools/list returns ask_wiki_question, read_wiki_contents and read_wiki_structure

Adaptations from #43321: rc/1.103.0 never had the blank S3 env e2e test, so its removal, the integration test that replaces it, and the coverage registry edits are left out. batch_cleanup.py and test_batch_cleanup.py are taken from main as of #43321, which also carries a one-line message assertion from #43294

Type

✅ Test

Caveats (if any)

Low

  • The DeepWiki fix still names upstream tools; main checks against DeepWiki's live tool list instead (test(e2e-ui): check the MCP Tools tab against the upstream's own tools/list #42397), which needs a new e2e UI dependency
  • The Mistral OCR 429 in the same build is a provider rate limit and is not addressed here
  • The two secret manager lanes are fixed separately in project-releaser, since the lane script lives there

Backports the batch cleanup half of #43321 so an Azure cancel still in progress after 120s leaves a warning instead of a teardown error, and points the MCP Tools UI test at DeepWiki's renamed ask_wiki_question tool.
@yuneng-berri
yuneng-berri merged commit 8aca86a into rc/1.103.0 Sep 27, 2026
5 of 6 checks passed
@yuneng-berri
yuneng-berri deleted the litellm_rc_1_103_0_fix_e2e_reds branch September 27, 2026 00:18
@greptile-apps

greptile-apps Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Low risk] Adjusts end-to-end test timeouts and error handling.

The PR should not merge while the Azure lifecycle test newly asserts calls previously excluded for provider-side flakiness.

Findings

  1. P1 Azure cancellation flakes return ▶

Summary

The PR shortens batch cancellation cleanup waits, reports still-cancelling batches and blocked file deletions as warnings, and updates the DeepWiki UI tool expectation. It also begins asserting Azure cancellation in the lifecycle test, which re-exposes the test to previously documented provider-call flakes.

Reviews (1) · Last reviewed commit: "test(e2e): clear the two rc/1.103.0 e2e ..."

# cancel is still deferred for cleanup, just not asserted for these two.
_CANCEL_ASSERTED_PROVIDERS = frozenset({"openai", "bedrock"})
# cancel is still deferred for cleanup, just not asserted for Vertex.
_CANCEL_ASSERTED_PROVIDERS = frozenset({"openai", "azure", "bedrock"})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Azure cancellation flakes return Adding Azure to this set makes every Azure lifecycle case retrieve and cancel the batch during the test. Azure was previously excluded because these provider calls can fail intermittently. If the connection is refused, the helpers do not retry it, so the test fails even when batch creation and retrieval worked and cleanup could proceed.

yuneng-berri added a commit that referenced this pull request Sep 27, 2026
…43382 and #43388 (#43391)

The batch cleanup leftover warning used a class defined in a test-directory module. The xdist controller cannot import it, so an uncaught leftover warning crashed the whole e2e run. It is now a plain UserWarning.

The integration harness backport missed the workers option and Gateway.request headers that the OTel conversation id tests use.

This branch is waiting to be deployed

1 waiting deployment
e2e-changed — da568ab1 Waiting Sep 27, 2026 by yuneng-berri via Run changed e2e tests against the stage-mirror stack #13169
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant