test(e2e): assert only litellm-owned batch behavior and move the blank S3 env pin to an integration test

[yuneng-berri]

TLDR

Problem this solves:

• Buildkite e2e fails on Azure's cancel speed, not LiteLLM behavior
• Blank S3 env e2e fails on runner IAM, not LiteLLM behavior
• Both reds block every PR without saying anything about our code

How it solves it:

• Azure cancel becomes an asserted step: LiteLLM must return cancelling or cancelled
• Teardown leaves a still-cancelling batch, and the file it holds, as a warning instead of a failure
• Any other cleanup status or error still fails the test
• Blank S3 env regression moves to an integration test against local S3 and Bedrock fakes
• The e2e version and its self-booted gateway are deleted

User Flow

Before: an engineer opens a PR and the Buildkite e2e check goes red for reasons outside LiteLLM

1. They push a PR and the buildkite/e2e-tests check runs the batch suite
2. test_batch_lifecycle[azure-unified] errors in teardown with "cancellation did not finish within 660.0s, last status cancelling" and a file delete HTTP 400
3. test_unified_batch_create_ignores_blank_s3_env_vars fails with AccessDenied on sts:TagSession
4. The check is red on every PR until someone reruns it or it happens to pass

After: the same run only fails when LiteLLM itself misbehaves

1. They push a PR and the buildkite/e2e-tests check runs the batch suite
2. The Azure test cancels through LiteLLM and checks the response, and a slow provider cancel shows up as a warning naming the batch and file
3. The blank S3 env case runs in integration-providers on CircleCI with no AWS access, and fails if a blank value reaches Bedrock
4. The check goes red only for a LiteLLM status or error

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

This PR only changes tests, so the proof is the real Buildkite e2e run against live Azure and Bedrock, plus a mutation check on the new integration test

Before (02b2c6d, litellm-e2e-pr build 1109)

Azure batch teardown

1. test_batch_lifecycle[azure-unified] errored: "cancellation did not finish within 660.0s, last status cancelling"
2. The following input file delete returned HTTP 400 from LiteLLM, which refuses to delete a file a non-terminal batch still references

Blank S3 env

1. test_unified_batch_create_ignores_blank_s3_env_vars failed: "not authorized to perform: sts:TagSession on resource: arn:aws:iam::...:role/berrie-litellm-e2e"

After (6601309)

Azure batch teardown

1. Pending the Buildkite e2e run on this PR

Blank S3 env

1. With _resolve_s3_setting reverted to treat "" as a value, the new integration test fails with {'s3BucketOwner': ''} == {}
2. With the fix in place, both the blank and the set case pass

Type

✅ Test

Caveats (if any)

Medium

• Leftover Azure batches and files are not cleaned up by anything yet
  • A scheduled cleanup job is deferred until leftovers become a problem
• The integration test reaches the fake Bedrock through a local HTTPS tunnel
  • The batch create URL has no override, so the test proxy runs with SSL verification off

Low

• The S3 upload already skipped blank values, so only the batch create request catches this regression
• I have not confirmed Buildkite's pytest flags show the warnings summary

QA runbook

• tests/e2e/batches/test_batches_e2e.py::test_batch_lifecycle[azure-unified] - a user can cancel an Azure batch through LiteLLM and gets a cancel status back
  • Upload a batch file: curl -X POST http://localhost:4000/v1/files -H "Authorization: Bearer sk-1234" -F purpose=batch -F target_model_names= -F file=@batch.jsonl
  • Create the batch: curl -X POST http://localhost:4000/v1/batches -H "Authorization: Bearer sk-1234" -d '{"input_file_id": "", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'
  • Cancel it: curl -X POST http://localhost:4000/v1/batches//cancel -H "Authorization: Bearer sk-1234"
  • Expect the same batch id with status cancelling or cancelled
  • Delete the input file while the batch is still cancelling and expect HTTP 400 naming batches in a non-terminal state
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Confidence Score: 4/5

[Medium risk] Moves a Bedrock blank S3 environment test from e2e to integration.

The PR appears safe to merge, with a non-blocking need to clean up batches and files left after slow cancellations.

Findings

1.  Slow cancellations leave resources behind ▶

Summary

The PR asserts Azure’s cancellation response, treats slow managed-batch teardown as a warning, and replaces a live blank-S3-environment e2e test with a local integration test.

• The replacement test checks both blank and populated S3 settings on outbound requests.
• Slow-cancellation leftovers currently have no follow-up cleanup path.

_{Reviews (1) · Last reviewed commit: "test(e2e): assert only litellm-owned bat..."}

[greptile-apps]

Slow cancellations leave resources behind

When a managed batch is still cancelling after two minutes, teardown warns and stops trying to clean it up. Deleting its input file can also return with only a warning. Repeated live Azure runs can therefore leave batches and files behind with no later cleanup attempt. Please add a way to clean up these leftovers after cancellation finishes.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!