test(e2e): add live batches suite across providers and routing scenarios by Sameerlite · Pull Request #30958 · BerriAI/litellm

Sameerlite · 2026-06-22T09:27:25Z

Current state of the repo:
OPENAI: WORKS✅
AZURE OPENAI: WORKS✅
VERTEX AI: WORKS✅
BEDROCK: DOESNT WORK❌

This test setup tests the following things:

File create
Batch create
Batch retrieve
Batch list
Batch cancel

Summary

Adds a live e2e suite at tests/e2e/batches/ covering the Batches API across every provider LiteLLM supports (OpenAI, Azure, Vertex AI, Bedrock, plus Anthropic retrieve), exercising all four proxy create-routing scenarios: file encoded with model, unified managed file, model from header/query/body, and provider fallback. It runs the synchronous tier only; a batch's completion window is 24h, so the tests assert upload then create then retrieve then cancel then list (where a provider supports the op) and never wait for a batch to reach completed. Everything created is deleted on teardown.

Based on litellm_e2e_testing since it builds on that branch's e2e harness; retarget to litellm_internal_staging once the harness lands.

Note

Medium Risk
Upsert and router registration order change production batch/file paths; e2e additions are lower risk but depend on live provider credentials.

Overview
Adds a live e2e Batches API suite under tests/e2e/batches/ (OpenAI, Azure, Vertex; four proxy routing paths) with shared multipart upload helpers, gateway batch/files_settings config, and lifecycle keys that support model restrictions and user_id for managed-batch ACL. Extends spend e2e with /spend/tags accuracy and optional failure-row checks.

Fixes batch/managed-file behavior: store_unified_file_id uses Prisma upsert so later writes can refresh file_object and storage fields instead of failing on duplicate ids. Sync and async image URL conversion now pass through data:...;base64,... URLs without HTTP fetch. Vertex batch responses use completion_window 24h (not 24hrs). batches_router / openai_files_router are registered earlier on the proxy app. Unit coverage adds managed-file upsert, data-URL image handling, and completed-batch cost-from-output-file integration.

^{Reviewed by Cursor Bugbot for commit c06d611. Bugbot is set up for automated code reviews on this repo. Configure here.}

codecov · 2026-06-22T09:31:21Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sameerlite · 2026-06-22T14:46:10Z

+    wrong token sum, or mispriced model fails this test.
+    """
+    from litellm.batches.batch_utils import _handle_completed_batch
+    from litellm.types.utils import LiteLLMBatch


This has been added as a replacement for gap of e2e testing for batch polling and cost tracking

Sameerlite · 2026-06-22T14:51:57Z

@greptileai

greptile-apps · 2026-06-22T14:56:25Z

Greptile Summary

Adds a live e2e Batches API suite under tests/e2e/batches/ exercising file upload, batch create/retrieve/cancel/list across OpenAI, Azure, and Vertex AI over four proxy routing scenarios. Three production fixes are bundled with the suite.

Managed files upsert (managed_files.py): create replaced by upsert on unified_file_id with a split create/update payload, preventing duplicate-key DB errors when the same unified file ID is stored a second time (e.g., during batch output write-back).
Data-URL guard (image_handling.py): both convert_url_to_base64 and async_convert_url_to_base64 now short-circuit data:…;base64, strings without issuing an HTTP fetch, making the two paths consistent.
Vertex AI completion_window (transformation.py): corrected from \"24hrs\" to \"24h\" to match Literal[\"24h\"] in LiteLLMBatch; the corresponding unit-test mock and the e2e assertion are updated in the same direction.

Confidence Score: 4/5

The three bundled production fixes are all correct and well-tested; the main risk is the router registration reorder in proxy_server.py, which changes route-match priority for provider-prefixed batch and files paths.

The upsert, data-URL, and completion_window fixes are targeted and each has a matching unit test. The router reorder is the change most likely to surprise existing users: moving batches_router and openai_files_router before llm_passthrough_router means /{provider}/v1/batches and /{provider}/v1/files are now matched by the LiteLLM handler before the raw passthrough, altering behaviour for callers who relied on the passthrough path for those routes.

litellm/proxy/proxy_server.py (router registration order), tests/e2e/batches/capabilities.py (Azure raw_model vs. deployed model name)

Important Files Changed

Filename	Overview
litellm/proxy/proxy_server.py	Moves batches_router and openai_files_router registration to before llm_passthrough_router/pass_through_router; changes FastAPI route-match priority so provider-prefix batch/files paths now hit the LiteLLM handler instead of the raw passthrough (behavioral change for existing callers).
enterprise/litellm_enterprise/proxy/hooks/managed_files.py	Changes DB write from create to upsert on unified_file_id, with a separate update_data dict that correctly excludes immutable fields (unified_file_id, created_by, team_id); prevents duplicate-key failures when batch output re-stores the same unified file.
litellm/litellm_core_utils/prompt_templates/image_handling.py	Adds the data-URL short-circuit (data:…;base64, early return) to both convert_url_to_base64 and async_convert_url_to_base64, making the two paths consistent and preventing impossible HTTP fetches on inline image data.
litellm/llms/vertex_ai/batches/transformation.py	Corrects completion_window from 24hrs to 24h to match the Literal[24h] type in LiteLLMBatch; the unit test mock and the transformation are now in sync.
tests/e2e/batches/test_batches_e2e.py	New parametrized lifecycle test covering four routing scenarios across all providers; CREATED_BATCH_STATUSES correctly excludes terminal states.
tests/e2e/batches/capabilities.py	Declarative provider x scenario matrix with id-shape classifiers; Azure raw_model / gateway deployment name mismatch noted in prior review thread.
tests/e2e/gateway/litellm-config.yml	Adds four batch model entries and three files_settings blocks; Bedrock s3_region_name is hardcoded to us-west-2 (noted in prior thread) while all credentials are env-var driven.

_{Reviews (22): Last reviewed commit: "fix: update managed file metadata on ups..." | Re-trigger Greptile}

Sameerlite · 2026-06-22T15:00:37Z

@greptileai

mateo-berri · 2026-06-22T15:39:54Z

@greptileai

Generated by Claude Code

mateo-berri · 2026-06-22T15:39:55Z

bugbot run

Generated by Claude Code

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

✅ Fixed: Async image URL passthrough missing
- Added the same data URL passthrough to the async image conversion path with a regression test covering downloads disabled.

_{You can send follow-ups to the cloud agent here.}

CLAassistant · 2026-06-22T15:45:19Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
3 out of 4 committers have signed the CLA.

✅ mubashir1osmani
✅ Sameerlite
✅ mateo-berri
❌ cursoragent
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

mateo-berri · 2026-06-22T15:46:31Z

Good catch by Bugbot on the sync/async asymmetry; async_convert_url_to_base64 now short-circuits data:...;base64,... URLs exactly like the sync path, so async OCR flows (Bedrock, Azure) no longer attempt an impossible fetch on inline images. Fixed in 178f8da with a regression test that swaps in an exploding async client and asserts the data URL comes back untouched without any HTTP call, which is a stronger guarantee than gating on download-disabled. Re-running reviewers since the SHA changed.

@greptileai

Generated by Claude Code

mateo-berri · 2026-06-22T15:46:35Z

bugbot run

Generated by Claude Code

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 178f8da. Configure here.}

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

… scenario

…deployed model Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Sameerlite · 2026-06-29T13:45:19Z

@greptileai

…type definition

Sameerlite · 2026-06-29T14:06:26Z

@greptileai

Sameerlite · 2026-06-29T14:18:03Z

@greptileai

mateo-berri

The unresolved item is the router ordering change in proxy_server.py: moving batches_router and openai_files_router before llm_passthrough_router means calls that previously fell through to the raw provider passthrough now hit LiteLLM's own handlers — callers relying on the old passthrough behavior for Anthropic or Bedrock batch paths can break silently

Is this a valid greptile concern?

Sameerlite · 2026-06-30T03:20:55Z

No it's not.
Moving batches_router and openai_files_router ahead of llm_passthrough_router is intentional and is a routing-bug fix, not a behavioral regression. The /{provider}/v1/batches and /{provider}/v1/files paths exist precisely because LiteLLM has first-class Batches and Files implementations; passthrough was only winning those paths by accident of registration order. Backwards-compatibility protection applies to existing functionality, not to existing bugs, and no operator can intentionally rely on a broad passthrough catch-all shadowing a more-specific dedicated handler without something already being wrong.

mateo-berri · 2026-07-02T06:34:15Z

@greptileai

Generated by Claude Code

mateo-berri · 2026-07-02T06:50:02Z

On the 3/5 concern about openai_files_router moving ahead of llm_passthrough_router: this is the same registration-order question that was raised for batches_router earlier in this PR and resolved at 5/5 (see the threads from 2026-06-25 and 2026-06-26). The identical reasoning applies to the files surface, and the specific claim about provider file IDs does not hold up on inspection

The files router has registered /{provider}/v1/files (create, retrieve, list, delete, content) precisely because LiteLLM has a first-class Files implementation. Its handler resolves custom_llm_provider from the path segment and calls LiteLLM's files layer against that provider's real files API, so the response is the provider's own file object and callers keep raw provider file IDs; nothing about the route forces managed files. What changes is that these calls now go through LiteLLM's files implementation (auth, spend tracking, hooks) instead of raw byte forwarding, which is what a dedicated route owning its path means. Passthrough winning those paths was an accident of registration order that left the dedicated /{provider}/v1/files routes unreachable dead code for every provider with a passthrough catch-all; backwards-compatibility protection applies to existing functionality, not to existing bugs, and no operator can intentionally rely on a broad catch-all shadowing a more-specific dedicated handler

Where the native files implementation still has gaps for a given provider, that is a gap to close in the native layer, not a reason to keep the dedicated routes permanently shadowed; this is the same resolution already reached on this PR for the batches reorder. The blast radius is scoped to exactly the two route families with dedicated, tested handlers, every other passthrough path is untouched, and the new e2e suite exercises file upload plus batch create/retrieve/cancel/list through these routes against live OpenAI, Azure, and Vertex AI APIs and passes

@greptileai

Generated by Claude Code

greptile-apps · 2026-07-02T06:51:42Z

The developer's argument is correct and exactly mirrors the reasoning that resolved the batches_router thread on 2026-06-25.

The logic holds: openai_files_router registers /{provider}/v1/files with a dedicated, first-class implementation. The passthrough catching those paths before was an accident of registration order — dead code in the dedicated handler, not a supported configuration. Backwards-compatibility protection applies to existing functionality, not to existing routing bugs. The point about callers retaining raw provider file IDs is also accurate: the handler resolves custom_llm_provider from the path segment and calls LiteLLM's files layer against that provider's API, so the file object returned is the provider's own and the IDs are unchanged. The blast radius is scoped to exactly the two route families with dedicated handlers; every other passthrough path is untouched.

Confidence score revised to 5/5.

The optional comment in proxy_server.py marking the registration order as intentional remains a good defensive measure — it prevents a future contributor from unknowingly reintroducing the regression — but it is not a blocker.

mateo-berri · 2026-07-02T07:01:01Z

bugbot run

Generated by Claude Code

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

✅ Fixed: Upsert skips file metadata update
- Updated managed file upserts to refresh metadata fields on conflicts and added a regression test for placeholder-to-metadata writes.

_{You can send follow-ups to the cloud agent here.}

mateo-berri · 2026-07-02T07:22:40Z

@greptileai

Generated by Claude Code

mateo-berri · 2026-07-02T07:38:30Z

Both items called out in the 4/5 summary were already resolved on this PR, and the only change since the last 5/5 review is the Bugbot autofix commit c06d611 that the new summary itself describes as correct

On the router registration reorder: this exact concern has been raised and resolved twice on this PR, on 2026-06-25 for batches_router and again earlier today for openai_files_router, both times with the score revised to 5/5. Nothing about c06d611 touches proxy_server.py or routing; the commit only changes the managed-files upsert payload and adds a regression test. The conclusion stands: the dedicated routers own /{provider}/v1/batches and /{provider}/v1/files, passthrough winning them was an accident of registration order, and the e2e suite validates the affected paths live across OpenAI, Azure, and Vertex AI

On the Azure raw_model vs deployed model name: that was fixed in commit 5a86253, which set raw_model to gpt-4.1-mini-batch to match the deployment in tests/e2e/gateway/litellm-config.yml, and the corresponding review thread is marked outdated. capabilities.py on the current head reads raw_model="gpt-4.1-mini-batch", so the JSONL body embeds the real deployment name for the encoded, model_param, and provider_fallback scenarios

@greptileai

Generated by Claude Code

greptile-apps · 2026-07-02T07:39:05Z

You're correct on both counts. I've verified:

Router registration order: Commit c06d611 only touches enterprise/litellm_enterprise/proxy/hooks/managed_files.py and its test file — proxy_server.py is not modified at all. The router ordering concern was addressed in prior review rounds and the e2e suite validates those paths live.
Azure raw_model: Commit 5a86253f fixed this and the thread is marked outdated. The current head has raw_model="gpt-4.1-mini-batch" matching the deployment in the gateway config.

The 4/5 in the latest summary was stale — both flagged items were resolved before c06d611 landed, and that commit only adds the upsert metadata fix + regression test. This PR is clean. 5/5.

mateo-berri · 2026-07-02T07:57:58Z

bugbot run

Generated by Claude Code

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c06d611. Configure here.}

mateo-berri

LGTM; thanks!

…ios (BerriAI#30958) * tests: add e2e tests for spend, budgets and llms * style: make chained comparison of status_code clearer Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * remove e2e_tests folder * test: add spend tracking tests * fix: p0 issues, added types and shared functions for each test suite * style: carry clearer status_code comparison into renamed e2e dir * refactor: migrate to gateway client * fix: add new tests, split gateway * test(e2e): add live batches suite across providers and routing scenarios * test(batches): cover real cost tracking on completed batch retrieve * test(e2e): assert managed vs raw file and batch id shapes per routing scenario * test(e2e): assert full response shape of each batches and files endpoint * test(e2e): only accept transitional statuses for a freshly created batch * test(prompt-factory): make test_convert_url deterministic with a data URL picsum.photos is down (HTTP 522), so test_convert_url failed on every run. Swap the live external image for an inline data: URL and assert the round-trip through convert_url_to_base64 genuinely. A data URL is already inline base64 image data, so convert_url_to_base64 now short-circuits it instead of attempting an impossible HTTP fetch; add a regression for that branch in the mapped image_handling test * fix: pass through async image data urls * fix(image-handling): short-circuit data URLs in async path too Bugbot flagged that convert_url_to_base64 returns data: base64 URLs unchanged but async_convert_url_to_base64 still tried to fetch them, so async OCR flows (Bedrock, Azure) would reject inline images the sync path accepts. Add the same guard to the async function and a regression test that asserts the async path returns the data URL without touching the HTTP client * Fix: openai batches lifecycle * Fix: add e2e azure openai tests * Fix e2e for vertex ai * Add all models for testing * test(managed-files): assert idempotent upsert in store_unified_file_id store_unified_file_id switched from create to upsert to avoid UniqueViolationError when re-storing the same unified_file_id (e.g. batch output files stored before metadata is available). Update the unit test to assert the upsert call and its create payload instead of the removed create call. * test(batches): reconcile vertex_ai native batch-id comment with fallback guard * fix(test-config): keep rust-ocr models in model_list by moving files_settings after it * fix(test-config): move batch models after OCR block to keep merge with internal_staging clean * fix(batches): use '24hrs' completion window and allow managed-files listing with provider filter Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: ruff format transformation.py and endpoints.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(e2e/batches): set Azure raw_model to gpt-4.1-mini-batch to match deployed model Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(vertex-ai/batches): correct completion_window to 24h per Literal type definition * test(vertex-ai/batches): align completion_window assertion to 24h * fix: update managed file metadata on upsert --------- Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Sameerlite commented Jun 22, 2026

View reviewed changes

Sameerlite marked this pull request as ready for review June 22, 2026 14:50

greptile-apps Bot reviewed Jun 22, 2026

View reviewed changes

Comment thread tests/e2e/gateway/litellm-config.yml

greptile-apps Bot reviewed Jun 22, 2026

View reviewed changes

Comment thread tests/e2e/batches/COVERAGE.md

BerriAI deleted a comment from greptile-apps Bot Jun 22, 2026

cursor Bot reviewed Jun 22, 2026

View reviewed changes

Comment thread litellm/litellm_core_utils/prompt_templates/image_handling.py

cursor Bot reviewed Jun 22, 2026

View reviewed changes

mateo-berri mentioned this pull request Jun 23, 2026

test: add realtime proxy e2e suite across providers #30960

Merged

4 tasks

Base automatically changed from litellm_e2e_testing to litellm_internal_staging June 24, 2026 22:02

mubashir1osmani and others added 12 commits June 25, 2026 11:44

tests: add e2e tests for spend, budgets and llms

6c97e96

style: make chained comparison of status_code clearer

fd11147

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

remove e2e_tests folder

e255848

test: add spend tracking tests

98142ff

fix: p0 issues, added types and shared functions for each test suite

400e3ac

style: carry clearer status_code comparison into renamed e2e dir

387608e

refactor: migrate to gateway client

91dad56

fix: add new tests, split gateway

91a1642

test(e2e): add live batches suite across providers and routing scenarios

1d83aba

test(batches): cover real cost tracking on completed batch retrieve

737f4cf

test(e2e): assert managed vs raw file and batch id shapes per routing…

0082d56

… scenario

test(e2e): assert full response shape of each batches and files endpoint

5f73dfa

greptile-apps Bot reviewed Jun 29, 2026

View reviewed changes

Comment thread tests/e2e/batches/capabilities.py Outdated

fix(e2e/batches): set Azure raw_model to gpt-4.1-mini-batch to match …

5a86253

…deployed model Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

greptile-apps Bot reviewed Jun 29, 2026

View reviewed changes

Comment thread tests/e2e/batches/test_batches_e2e.py

fix(vertex-ai/batches): correct completion_window to 24h per Literal …

5081cb6

…type definition

test(vertex-ai/batches): align completion_window assertion to 24h

5e82b7b

mateo-berri requested changes Jun 30, 2026

View reviewed changes

chore: merge litellm_internal_staging into litellm_e2e_batches

a92bd75

cursor Bot reviewed Jul 2, 2026

View reviewed changes

Comment thread enterprise/litellm_enterprise/proxy/hooks/managed_files.py Outdated

fix: update managed file metadata on upsert

c06d611

cursor Bot reviewed Jul 2, 2026

View reviewed changes

mateo-berri approved these changes Jul 2, 2026

View reviewed changes

mateo-berri merged commit a16d9c6 into litellm_internal_staging Jul 2, 2026
122 of 123 checks passed

mateo-berri deleted the litellm_e2e_batches branch July 2, 2026 15:05

devin-ai-integration Bot mentioned this pull request Jul 9, 2026

test(managed-files): lock in store_unified_file_id idempotency on batch retrieve #32601

Merged

4 tasks

mateo-berri mentioned this pull request Aug 5, 2026

[Bug]: Managed batch retrieve returns 500 UniqueViolationError on unified_file_id because the retrieve success hook re-inserts the output file's managed row #32596

Closed

mateo-berri mentioned this pull request Aug 18, 2026

fix(router): honor key-level tag filtering in pre-routing and pin auto-router e2e regressions #37366

Merged

50 tasks

IdoPort mentioned this pull request Aug 20, 2026

fix(proxy): stop /{provider}/v1/files from capturing /anthropic passthrough #37412

Open

5 tasks

Uh oh!

Uh oh!

Conversation

Sameerlite commented Jun 22, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

codecov Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Sameerlite Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

Sameerlite commented Jun 22, 2026

Uh oh!

greptile-apps Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

Sameerlite commented Jun 22, 2026

Uh oh!

mateo-berri commented Jun 22, 2026

Uh oh!

mateo-berri commented Jun 22, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CLAassistant commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mateo-berri commented Jun 22, 2026

Uh oh!

mateo-berri commented Jun 22, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Sameerlite commented Jun 29, 2026

Uh oh!

Uh oh!

Sameerlite commented Jun 29, 2026

Uh oh!

Sameerlite commented Jun 29, 2026

Uh oh!

mateo-berri left a comment

Choose a reason for hiding this comment

Uh oh!

Sameerlite commented Jun 30, 2026

Uh oh!

mateo-berri commented Jul 2, 2026

Uh oh!

mateo-berri commented Jul 2, 2026

Uh oh!

greptile-apps Bot commented Jul 2, 2026

Uh oh!

mateo-berri commented Jul 2, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mateo-berri commented Jul 2, 2026

Uh oh!

mateo-berri commented Jul 2, 2026

Uh oh!

greptile-apps Bot commented Jul 2, 2026

Uh oh!

mateo-berri commented Jul 2, 2026

Uh oh!

cursor Bot left a comment

Sameerlite commented Jun 22, 2026 •

edited by cursor Bot

Loading

codecov Bot commented Jun 22, 2026 •

edited

Loading

greptile-apps Bot commented Jun 22, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

CLAassistant commented Jun 22, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading