test(e2e): report batch cleanup leftovers as a plain UserWarning on rc/1.104.0

[yuneng-berri]

TLDR

Problem this solves:

• A batch cleanup leftover warning crashes the whole Buildkite e2e run
• xdist cannot import the custom warning class on the controller

How it solves it:

• Warn with a plain UserWarning instead of BatchCleanupLeftover
• Tests and COVERAGE.md updated to match

User Flow

Before: a release qualifier running the litellm-e2e Buildkite pipeline gets a crashed run instead of test results

1. They trigger litellm-e2e with a litellm SHA and unblock the release candidate step
2. A batch test leaves a file in place because LiteLLM refuses to delete it while a batch is still cancelling
3. The full suite step dies with INTERNALERROR and ModuleNotFoundError: No module named 'batch_cleanup', and the retry step then fails with "no artifacts found"

After: the same leftover shows up as a warning and the run finishes with real results

1. They trigger litellm-e2e the same way
2. The same leftover happens
3. The run reports it as a UserWarning naming the file id and keeps going to a pass/fail verdict

Relevant issues

Same change as #43391, which already fixed this on rc/1.103.0. The main copy is its own PR

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

The existing tests/e2e/batches/test_batch_cleanup.py covers both leftover paths and passes 50/50 on this branch

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Before (e0cbb0d)

1. Buildkite litellm-e2e build 313 ran the full suite at rc/1.104.0 6d672483c9, which has the same batch_cleanup.py as this base: https://buildkite.com/berriai-1/litellm-e2e/builds/313
2. The full suite step crashed on the warning:

INTERNALERROR>   File "/app/.venv/lib/python3.13/site-packages/xdist/workermanage.py", line 474, in unserialize_warning_message
INTERNALERROR>     mod = importlib.import_module(data["message_module"])
INTERNALERROR> ModuleNotFoundError: No module named 'batch_cleanup'
[gw4] node down:

After (2263772)

1. Run pytest tests/e2e/batches/test_batch_cleanup.py -q
2. Output: 50 passed, including the two leftover cases that now assert a UserWarning whose message names the left file and batch
3. The warning category now lives in builtins, which the xdist controller can always import

Type

✅ Test

Caveats (if any)

Low

• Not re-run on Buildkite yet
  • The CI OpenAI account is currently out of credits, so a full e2e run fails for an unrelated reason
• The crash did not reproduce in a local xdist run
  • My local run had the batches directory importable on the controller; Buildkite does not

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Confidence Score: 4/5

[Low risk] End-to-end test cleanup code replaces a custom warning class.

The cleanup change appears sound, but the explicit test-coverage requirement should be satisfied before merging.

Findings

1.  Custom warnings still pass ▶

Summary

The PR replaces the batch cleanup helper’s custom leftover warning with the built-in UserWarning and updates its tests and coverage notes.

• Both leftover paths now emit a warning category the xdist controller can import.
• The updated tests check leftover messages, but do not require the warning to be the built-in type.

_{Reviews (1) · Last reviewed commit: "test(e2e): report batch cleanup leftover..."}

[greptile-apps]

Custom warnings still pass

pytest.warns(UserWarning) also accepts custom subclasses, so this test and the batch-leftover test would still pass if cleanup brought back a warning type that the xdist controller cannot import. Check that each captured warning has the exact type UserWarning. The repository requires changes to existing tests not to weaken regression coverage; that requirement should be met before merging.

Rule Used: What: Flag any modifications to existing tests and verify they don't weaken test coverage or mask regressions. Why: Developers may alter tests to make failing code pass rather than fix the actual bug, hiding regressions. Good: ``` // Test updated t... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!