fix(router): pin batch, file, and fine-tuning job ids to their owning model group on fallback by mateo-berri · Pull Request #38742 · BerriAI/litellm

mateo-berri · 2026-08-29T07:52:52Z

TLDR

Problem this solves:

Batch cancel carries batch_id, yet fallback still crossed model groups
The foreign provider is asked to cancel an id it never issued
Wasted round trip, ERROR-level fallback noise, id leaked to the wrong provider
Same hole for file_id and fine_tuning_job_id calls
File-content errors even quoted the foreign provider's 404 to the caller

How it solves it:

Adds those ids to the provider-scoped fallback pin next to input_file_id
Retries stay inside the model group that owns the resource
The owning provider's real error surfaces immediately
File-content error bodies now carry the owner's error instead
The pin arms only on real file, batch, and fine-tuning job operations, matched by handler name on both the direct and generic router dispatch paths, so a stray top-level file_id-style field on an ordinary completion request cannot silently disable that request's cross-group fallbacks

An earlier tip also carried repairs for two responses-suite tests that went stale on staging when PR #38593 gated temperature for reasoning models; PR #38748 has since landed its own versions of those repairs on staging, and the merge commit here adopts them, so this PR no longer touches those files

User Flow

Before: an admin canceling a batch during an OpenAI key outage watches the gateway also try the cancel against their unrelated Azure group before the real error returns

They upload a batch input file: POST http://litellm-domain/v1/files with purpose=batch and target_model_names=batch-owner (their OpenAI-backed group), and get back a long scrambled gateway file id
They create the batch: POST http://litellm-domain/v1/batches with that file id, and get back a long scrambled batch id with status validating; the batch runs in their OpenAI account
The OpenAI key on batch-owner breaks, and they cancel: POST http://litellm-domain/v1/batches/{batch_id}/cancel
The response is OpenAI's own 401 invalid_api_key, but only after an extra provider round trip: the gateway log shows Falling back to model_group = batch-neighbor and then an ERROR carrying Azure's 404 "The requested job ... does not exist under the account", because their Azure resource was really asked to cancel a batch id it never issued

After: the same cancel never leaves the group that owns the batch, so the real error comes straight back and Azure never sees the foreign id

They upload a batch input file: POST http://litellm-domain/v1/files with purpose=batch and target_model_names=batch-owner, and get back a long scrambled gateway file id
They create the batch: POST http://litellm-domain/v1/batches with that file id, and get back a long scrambled batch id with status validating
The OpenAI key on batch-owner breaks, and they cancel: POST http://litellm-domain/v1/batches/{batch_id}/cancel
The response is the same OpenAI 401 invalid_api_key, now after a single call to OpenAI: the gateway log shows Skipping fallback to model_group = batch-neighbor: request names a resource owned by model_group = openai-batch-dep, and their Azure resource gets no call at all

Relevant issues

Linear ticket

Resolves LIT-6361

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live before/after run against real OpenAI and a real Azure OpenAI resource, no mocks. Both legs ran the identical flow and topology, differing only in the commit the proxy was booted from: Before at the merge base, After at this PR's tip

Shared setup: config defines batch-owner (openai/gpt-5.4-mini, model_info.id: openai-batch-dep) and batch-neighbor (azure/gpt-4o-mini on a real Azure resource) with router_settings.default_fallbacks: ["batch-neighbor"]. Every proxy boot used --num_workers 2. Phase 1 (good config, one process) served the upload and batch create; phase 2 rebooted with only the batch-owner api_key replaced by sk-invalid-simulated-outage, as two separate processes on two random ports sharing the leg's Postgres DB, with the cancel sent to one process and the file-content read to the other, so state written through the phase-1 process was read back through both phase-2 processes. Both legs' real batches were canceled directly against OpenAI afterward with the real key

Before (`ae7e50f`)

Upload and batch create (owner group healthy)

Upload the batch input through the phase-1 proxy (port 33456)

curl -s http://localhost:33456/v1/files -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -F purpose=batch -F target_model_names=batch-owner -F file=@lit6361-qa-input.jsonl

{"id": "bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9vY3RldC1zdHJlYW07dW5pZmllZF9pZCw1MjZmNjRmZS1lNGRjLTQ1NDQtYjU0MC0yYzJmYTMyOGU3ZGQ7dGFyZ2V0X21vZGVsX25hbWVzLGJhdGNoLW93bmVyO2xsbV9vdXRwdXRfZmlsZV9pZCxmaWxlLU5BYzVlM001NnNYb2ZUc0ZFZWl4bWE7bGxtX291dHB1dF9maWxlX21vZGVsX2lkLG9wZW5haS1iYXRjaC1kZXA",
 "bytes": 195, "object": "file", "purpose": "batch", "status": "uploaded"}

Create the batch with that unified file id

curl -s http://localhost:33456/v1/batches -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"input_file_id": "<unified file id above>", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'

{"id": "bGl0ZWxsbV9wcm94eTttb2RlbF9pZDpvcGVuYWktYmF0Y2gtZGVwO2xsbV9iYXRjaF9pZDpiYXRjaF82YTkyOTE3MmIyZGM4MTkwOGEzYzdmZDNmMTVlY2IyYQ",
 "completion_window": "24h", "endpoint": "/v1/chat/completions", "object": "batch", "status": "validating"}

Batch cancel with the owner key broken

Cancel through phase-2 process B (port 35802); the caller gets the owner's 401

curl -s -w '\nHTTP %{http_code}\n' -X POST "http://localhost:35802/v1/batches/<unified batch id>/cancel" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

{"error":{"message":"Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-inval***************tage. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'code': 'invalid_api_key', 'param': None}, 'status': 401}","type":"internal_server_error","param":null,"code":"401"}}
HTTP 401

Read the gateway log the admin sees: the fallback fired and the Azure resource was really asked to cancel a batch id it never issued

01:00:57 - LiteLLM Router:INFO: fallback_event_handlers.py:393 - Falling back to model_group = batch-neighbor
01:00:58 - LiteLLM Router:ERROR: router.py:6091 - litellm._acancel_batch(model=batch-neighbor, {'batch_id': 'batch_6a929172b2dc81908a3c7fd3f15ecb2a', ...}) Exception Error code: 404 - {'error': {'code': 'UserError', ... 'message': "The requested job 'batch_6a929172b2dc81908a3c7fd3f15ecb2a' does not exist under the account 'mateo-resource'. ...", 'innerError': {'code': 'NotFound', 'innerError': {'code': 'JobNotFound', ...}}}}

Grep counts in this process's log: Falling back to model_group 1, Skipping fallback to model_group 0, JobNotFound 5, mateo-resource (the Azure host) 10

File content with the owner key broken

Fetch content through phase-2 process C (port 41415); the foreign provider's 404 reaches the caller verbatim inside the error body

curl -s -w '\nHTTP %{http_code}\n' "http://localhost:41415/v1/files/<unified file id>/content" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

{"error":{"message":"Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-inval***************tage. ...'}, 'status': 401}. Received Model Group=openai-batch-dep\nAvailable Model Group Fallbacks=['batch-neighbor']\nError doing the fallback: Error code: 404 - {'error': {'code': 'notFound', 'message': 'The specified entity cannot be found.'}}","type":"invalid_request_error","param":null,"code":"401"}}
HTTP 401

This process's log confirms the OpenAI provider file id was sent to the Azure api_base

01:01:04 - LiteLLM Router:INFO: fallback_event_handlers.py:393 - Falling back to model_group = batch-neighbor
01:01:04 - LiteLLM:DEBUG: utils.py:534 - litellm.file_content(file_id='file-NAc5e3M56sXofTsFEeixma', model='azure/gpt-4o-mini', custom_llm_provider='azure', ...)

Grep counts in this process's log: Falling back to model_group 1, Skipping fallback to model_group 0, mateo-resource 7

After (`8cf090b`)

The current tip 2d11870 is a conflict-resolution merge of the base that only adopts PR #38748's versions of two responses-suite test files; git diff 8cf090b368..2d11870793 -- litellm/ tests/test_litellm/ is empty, so this proof is exactly what the tip runs

Upload and batch create (owner group healthy)

Upload the batch input through the phase-1 proxy (port 27446)

curl -s http://localhost:27446/v1/files -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -F purpose=batch -F target_model_names=batch-owner -F file=@lit6361-qa-batch-input.jsonl

{"id": "bGl0ZWxsbV9wcm94eTphcHBsaWNhdGlvbi9vY3RldC1zdHJlYW07dW5pZmllZF9pZCxmNDRkZGI4NC1jZWNkLTRmOTQtYjgyYy1iODVlMDA1YmEzNjA7dGFyZ2V0X21vZGVsX25hbWVzLGJhdGNoLW93bmVyO2xsbV9vdXRwdXRfZmlsZV9pZCxmaWxlLVlSNWZIaExlbzh6MXVoY1h6NXNxeHY7bGxtX291dHB1dF9maWxlX21vZGVsX2lkLG9wZW5haS1iYXRjaC1kZXA",
 "bytes": 195, "object": "file", "purpose": "batch", "status": "uploaded"}

Create the batch with that unified file id

curl -s http://localhost:27446/v1/batches -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"input_file_id": "<unified file id above>", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'

{"id": "bGl0ZWxsbV9wcm94eTttb2RlbF9pZDpvcGVuYWktYmF0Y2gtZGVwO2xsbV9iYXRjaF9pZDpiYXRjaF82YTkyOWQzYTE4OGM4MTkwOTE2ZDRmNDI0M2YwMTUwZQ",
 "completion_window": "24h", "endpoint": "/v1/chat/completions", "object": "batch", "status": "validating"}

Batch cancel with the owner key broken

Cancel through phase-2 process B (port 27447); identical caller-visible 401, now after a single owner-side attempt

curl -s -w '\nHTTP %{http_code}\n' -X POST "http://localhost:27447/v1/batches/<unified batch id>/cancel" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

{"error":{"message":"Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-inval***************tage. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'code': 'invalid_api_key', 'param': None}, 'status': 401}","type":"internal_server_error","param":null,"code":"401"}}
HTTP 401

The gateway log now shows the pin instead of the fallback, and zero foreign-call markers
```
LiteLLM Router:INFO: fallback_event_handlers.py:394 - Skipping fallback to model_group = batch-neighbor: request names a resource owned by model_group = openai-batch-dep
```
Grep counts in this process's log: Falling back to model_group 0, Skipping fallback to model_group 1, JobNotFound 0, mateo-resource 0, does not exist under the account 0

File content with the owner key broken

Fetch content through phase-2 process C (port 27448); the error body now carries the owner's real 401 where the foreign 404 used to be

curl -s -w '\nHTTP %{http_code}\n' "http://localhost:27448/v1/files/<unified file id>/content" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY"

{"error":{"message":"Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-inval***************tage. ...'}, 'status': 401}. Received Model Group=openai-batch-dep\nAvailable Model Group Fallbacks=['batch-neighbor']\nError doing the fallback: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-inval***************tage. ...', 'code': 'invalid_api_key', ...}, 'status': 401}","type":"invalid_request_error","param":null,"code":"401"}}
HTTP 401

This process's log shows the same pin line and no foreign call
```
LiteLLM Router:INFO: fallback_event_handlers.py:394 - Skipping fallback to model_group = batch-neighbor: request names a resource owned by model_group = openai-batch-dep
```
Grep counts in this process's log: Falling back to model_group 0, Skipping fallback to model_group 1, mateo-resource 0

The real batch behind the After leg was canceled directly against OpenAI afterward with the real key (200, status cancelling)

Stray top-level file_id on a chat completion (the handler-name gate)

An ordinary /v1/chat/completions request carrying a stray top-level file_id extra body field must keep its cross-group fallbacks. On the broken-owner config, both legs got the same request

curl -s -w '\nHTTP %{http_code}\n' http://localhost:<port>/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"model": "batch-owner", "messages": [{"role": "user", "content": "say ping"}], "file_id": "app-metadata-123"}'

Before (ae7e50f, port 41400) and After (8cf090b, port 59632) returned byte-identical bodies: the owner's 401, then Error doing the fallback: litellm.NotFoundError: AzureException NotFoundError, proving the fallback really crossed to batch-neighbor and reached the Azure resource on both legs. The After log shows Falling back to model_group = batch-neighbor and zero Skipping fallback lines. At this PR's previous tip (0b14897) the same request instead retried the owner's own broken key (Error doing the fallback: litellm.AuthenticationError ... sk-inval), silently losing the cross-group fallback; the handler-name gate in 8cf090b removes that regression

Observations from the run (things the diff alone would not show):

Skip line names deployment id, not group name (this PR, cosmetic)
Fallback wrapper text persists on pinned file reads (pre-existing, unchanged)
Cancel 401 body labeled internal_server_error (pre-existing, unchanged)
QA batches self-failed upstream, input file missing (pre-existing, unchanged)
Cancel hides fallback text; file route exposes it (pre-existing, unchanged)

Type

🐛 Bug Fix

Caveats (if any)

Vector store file operations dispatch through generically named router aliases (acreate, adelete, aretrieve, aretrieve_content), so they are deliberately left out of the pin's handler allowlist and keep today's fallback behavior. Router.aretrieve_batch fans out to every deployment itself and never enters the fallback wrapper, so it needs no pin

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
2d11870 passes /live-pr-risk (verified live at 8cf090b; the merge tip's litellm/ and tests/test_litellm/ trees are byte-identical to it)

Note

Medium Risk
Changes core router fallback gating for batch/file/fine-tuning APIs; misclassified handlers could block valid fallbacks or allow incorrect cross-group calls, though tests target the main paths.

Overview
Stops router fallbacks from sending batch, file, and fine-tuning job operations to another model group when the request carries provider-owned ids (batch_id, file_id, fine_tuning_job_id, alongside existing input_file_id / training_file). Cross-group fallback is skipped so callers get the owning deployment’s error instead of a wasted foreign 404 and leaked ids.

The pin only arms for known file/batch/fine-tuning handlers, checking both original_function and original_generic_function (generic API dispatch). Chat completions with stray top-level fields like file_id still get cross-group fallbacks. Same-group retries (e.g. order-based failover) are unchanged.

Tests cover unit fallback behavior and an integration case where acancel_batch does not hit the default fallback provider.

^{Reviewed by Cursor Bugbot for commit 2d11870. Bugbot is set up for automated code reviews on this repo. Configure here.}

… model group on fallback

greptile-apps · 2026-08-29T07:54:53Z

Greptile Summary

The PR prevents resource-specific batch, file, and fine-tuning operations from falling back to unrelated provider groups while retaining normal cross-group fallback for ordinary requests.

Adds handler-aware detection of provider-scoped resource identifiers.
Keeps retries for recognized resource operations within the owning model group.
Adds focused fallback tests and repairs two Responses API tests for current reasoning-model behavior.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/router_utils/fallback_event_handlers.py	Adds handler-gated provider-resource detection so resource identifiers pin fallback attempts to their owning model group.
tests/test_litellm/router_utils/test_fallback_event_handlers.py	Covers batch, file, and fine-tuning pinning, same-group retries, generic dispatch, and stray completion fields.
tests/test_litellm/test_router.py	Verifies end-to-end router behavior prevents batch cancellation from reaching a foreign fallback provider.
tests/llm_responses_api_testing/test_openai_responses_api.py	Updates a Responses API test to explicitly disable reasoning so temperature remains applicable.
tests/openai_endpoints_tests/test_e2e_openai_responses_api.py	Uses a non-reasoning model so the invalid-temperature test continues exercising provider rejection.

_{Reviews (4): Last reviewed commit: "fix(router): arm the provider-scoped fal..." | Re-trigger Greptile}

codecov · 2026-08-29T08:02:50Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

mateo-berri · 2026-08-29T08:09:24Z

@greptileai

PR #38593 stopped forwarding temperature to reasoning models, which left test_extra_body_merges_with_request_data raising UnsupportedParamsError and test_bad_request_bad_param_error no longer getting a rejection from OpenAI because drop_params now eats the param. Both repairs are the same hunks PR #38739 carries, so the branches merge clean in either order

mateo-berri · 2026-08-29T08:32:35Z

@greptileai

…erating handlers

mateo-berri · 2026-08-29T08:52:18Z

@greptileai

mateo-berri · 2026-08-29T09:15:09Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8cf090b. Configure here.}

…itellm_batch_id_fallback_pin # Conflicts: # tests/openai_endpoints_tests/test_e2e_openai_responses_api.py

mateo-berri · 2026-08-29T09:37:13Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2d11870. Configure here.}

codspeed · 2026-08-29T10:03:56Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_batch_id_fallback_pin (2d11870) with litellm_internal_staging (ae7e50f)¹}

No successful run was found on litellm_internal_staging (2d11870) during the generation of this report, so ae7e50f was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

fix(router): pin batch, file, and fine-tuning job ids to their owning…

0b14897

… model group on fallback

greptile-apps Bot reviewed Aug 29, 2026

View reviewed changes

Comment thread tests/test_litellm/test_router.py

mateo-berri added the run-ci label Aug 29, 2026

fix(router): arm the provider-scoped fallback pin only on resource-op…

8cf090b

…erating handlers

mateo-berri added run-ci and removed run-ci labels Aug 29, 2026

cursor Bot reviewed Aug 29, 2026

View reviewed changes

Merge remote-tracking branch 'origin/litellm_internal_staging' into l…

2d11870

…itellm_batch_id_fallback_pin # Conflicts: # tests/openai_endpoints_tests/test_e2e_openai_responses_api.py

mateo-berri added run-ci and removed run-ci labels Aug 29, 2026

cursor Bot reviewed Aug 29, 2026

View reviewed changes

mateo-berri enabled auto-merge August 29, 2026 09:41

yucheng-berri approved these changes Aug 29, 2026

View reviewed changes

mateo-berri merged commit efe51da into litellm_internal_staging Aug 29, 2026
125 of 126 checks passed

mateo-berri deleted the litellm_batch_id_fallback_pin branch August 29, 2026 09:57

Navigation Menu

Uh oh!

fix(router): pin batch, file, and fine-tuning job ids to their owning model group on fallback - #38742

fix(router): pin batch, file, and fine-tuning job ids to their owning model group on fallback#38742
mateo-berri merged 4 commits into
litellm_internal_stagingfrom
litellm_batch_id_fallback_pin

mateo-berri commented Aug 29, 2026 •

edited by cursor Bot

Loading

Uh oh!

greptile-apps Bot commented Aug 29, 2026 •

edited

Loading

Important Files Changed

Uh oh!

Uh oh!

codecov Bot commented Aug 29, 2026 •

edited

Loading

Uh oh!

mateo-berri commented Aug 29, 2026

Uh oh!

mateo-berri commented Aug 29, 2026

Uh oh!

mateo-berri commented Aug 29, 2026

Uh oh!

mateo-berri commented Aug 29, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

mateo-berri commented Aug 29, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

codspeed Bot commented Aug 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mateo-berri commented Aug 29, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before (ae7e50f)

Upload and batch create (owner group healthy)

Batch cancel with the owner key broken

File content with the owner key broken

After (8cf090b)

Upload and batch create (owner group healthy)

Batch cancel with the owner key broken

File content with the owner key broken

Stray top-level file_id on a chat completion (the handler-name gate)

Type

Caveats (if any)

Final Attestation

Uh oh!

greptile-apps Bot commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

codecov Bot commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mateo-berri commented Aug 29, 2026

Uh oh!

mateo-berri commented Aug 29, 2026

Uh oh!

mateo-berri commented Aug 29, 2026

Uh oh!

mateo-berri commented Aug 29, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Aug 29, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codspeed Bot commented Aug 29, 2026

Merging this PR will not alter performance

Footnotes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mateo-berri commented Aug 29, 2026 •

edited by cursor Bot

Loading

Before (`ae7e50f`)

After (`8cf090b`)

greptile-apps Bot commented Aug 29, 2026 •

edited

Loading

codecov Bot commented Aug 29, 2026 •

edited

Loading