fix(batches): skip discarded tokenization on allowlist-only batch input file reads

[mynkyu]

TLDR

Problem this solves:

• POST /v1/batches spends ~10s per 652-row file tokenizing rows whose token totals are then thrown away
• It happens whenever a key has a restricted models list, even with disable_batch_input_file_rate_limiting: true
• The opt-out is unreachable for those keys: _should_skip_batch_input_file_processing returns on the allowlist check first

How it solves it:

• Adds a models-only read: collect each row's body.model, skip _count_entry_tokens
• Allowlist enforcement unchanged — file still downloaded, every body.model still validated
• Purely additive; no existing signature or return shape changes

The problem

_should_skip_batch_input_file_processing consults the operator opt-outs after the model-allowlist check:

if self._key_requires_batch_model_access_check(user_api_key_dict):
    return False, None                      # restricted `models` -> full path, always

if general_settings.get("disable_batch_input_file_rate_limiting") is True:
    return True, None                       # never reached for such keys

The download itself is genuinely required — the allowlist can only be enforced by inspecting every row. But count_input_file_usage also runs _count_entry_tokens (a real tokenizer) per row, and async_pre_call_hook then discards the totals because nothing charges them.

That tokenization is the expensive half, and it is synchronous CPU work inside the async pre-call hook.

Production measurements

Observed on a proxy running v1.96.2, during a nightly backfill submitting 652-row batches to vertex_ai/gemini-2.5-flash:

	latency
POST /v1/batches (652 rows)	10~12s
batch status polling, same key	0.55~0.64s
interactive calls, same deployment	0.5~0.7s

This held a proxy-wide average-latency monitor above its threshold for as long as a batch was running. Reproduced with zero 429s, so it is not retry-driven — it is the cost of the read itself.

The fix

• _batch_input_file_models_only() reports when the JSONL is needed solely for the allowlist
• count_input_file_usage(models_only=...) then collects body.model without tokenizing
• async_pre_call_hook returns before counter enforcement in that mode, leaving _batch_*_count unset — matching the existing full-skip path rather than recording zeroes

Deliberately conservative — it returns False whenever anything still needs the totals:

• unrestricted keys (already covered by the existing full-skip paths)
• enqueued-token scopes, whose reservation is priced from the totals
• no opt-out configured

Tests

4 added to tests/test_litellm/proxy/hooks/test_batch_file_validation.py; suite goes 79 → 83 passing.

test	asserts
..._skips_tokenization_for_restricted_key	file is downloaded and allowlist is enforced, but _count_entry_tokens is never called and no counters are charged
..._still_rejects_unauthorized_model	a disallowed model in the JSONL still raises 403 — the opt-out must not become an authz bypass
test_restricted_key_without_opt_out_still_counts_tokens	the pre-existing path is untouched
..._disabled_when_enqueued_scopes_apply	enqueued reservations still get real totals

async_pre_call_hook swallows unexpected exceptions and returns data, so the returned value alone proves nothing — each test also asserts on the calls that separate the intended path from error recovery.

Control: reverting only the source change (tests kept) fails the first test with

AssertionError: Expected '_count_entry_tokens' to not have been called. Called 2 times.

confirming the test is load-bearing rather than vacuous.

Relevant issues

I could not find an existing issue for this. Note that #35408 and #35813 touch the same function (bounding the input-file read with a deadline) — this change is orthogonal (it reduces work within the read) but will likely need a trivial rebase against whichever lands first.

Linear ticket

n/a — external contribution.

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (not yet run)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5

Screenshots / Proof of Fix

Run against main in a container, with the import origin printed so it is unambiguous which tree is under test:

IMPORTED FROM: /oss/litellm
helper present: True
83 passed, 1 warning in 9.13s

Control, same command against an otherwise identical tree with only the source change reverted:

IMPORTED FROM: /ctrl/litellm
helper present: False
FAILED tests/.../test_batch_file_validation.py::test_models_only_read_skips_tokenization_for_restricted_key
1 failed, 3 passed

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
4 out of 5 committers have signed the CLA.

✅ mphilippnv
✅ mynkyu
✅ mateo-berri
✅ yuneng-berri
❌ cursoragent
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR combines the batch models-only optimization with broader changes to managed pass-through IDs, Prometheus caller identity, exception mapping, streaming utilities, and dashboard primitives

• Skips discarded tokenization while retaining batch model allowlist enforcement
• Adds streamed Responses API managed-ID ownership and shared SSE frame splitting
• Adds configurable Prometheus caller identity labels and broader HTTP status normalization
• Updates dashboard primitives for ref-as-prop behavior

Confidence Score: 4/5

The PR appears safe to merge after addressing the non-blocking repository-convention issues in the Prometheus changes

The batch optimization retains model authorization, and no concrete runtime regression remains; accepted feedback is limited to mutable label transformation and unnecessary source comments

Files Needing Attention: litellm/types/integrations/prometheus.py, litellm/integrations/prometheus.py

Important Files Changed

Filename	Overview
litellm/proxy/hooks/batch_rate_limiter.py	Adds a models-only JSONL read that preserves allowlist checks while skipping unused tokenization
litellm/proxy/pass_through_endpoints/managed_id_rewriter.py	Adds buffered SSE response-ID ownership and rewriting for managed Responses API objects
litellm/types/integrations/prometheus.py	Adds caller-identity label modes, with an in-place transformation that violates the repository's immutability convention
litellm/integrations/prometheus.py	Validates caller-identity configuration and propagates user email through deployment failure metrics, but adds prohibited explanatory comments
litellm/litellm_core_utils/exception_mapping_utils.py	Maps additional provider HTTP statuses to normalized LiteLLM exceptions
ui/litellm-dashboard/src/components/shared/Alert.tsx	Wraps the new alert primitive while retaining application-specific status variants

_{Reviews (1): Last reviewed commit: "fix(batches): skip discarded tokenizatio..." | Re-trigger Greptile}

[greptile-apps]

In-place label mutation

This resolver mutates resolved_labels in place, violating the repository’s immutable-data convention and complicating the label transformation

Suggested change

	if caller_identity == "user_email":
	resolved_labels[alias_index] = UserAPIKeyLabelNames.USER_EMAIL.value
	elif caller_identity == "both":
	resolved_labels.insert(alias_index + 1, UserAPIKeyLabelNames.USER_EMAIL.value)
	return resolved_labels
	if caller_identity == "user_email":
	return [
	*resolved_labels[:alias_index],
	UserAPIKeyLabelNames.USER_EMAIL.value,
	*resolved_labels[alias_index + 1 :],
	]
	if caller_identity == "both":
	return [
	*resolved_labels[: alias_index + 1],
	UserAPIKeyLabelNames.USER_EMAIL.value,
	*resolved_labels[alias_index + 1 :],
	]
	return resolved_labels

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Redundant implementation comments

These comments narrate the calls immediately below and duplicate ordinary behavior, adding prose that must remain synchronized with future changes

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codecov]

Codecov Report

❌ Patch coverage is 90.47619% with 20 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...roxy/pass_through_endpoints/managed_id_rewriter.py	90.00%	7 Missing ⚠️
...ellm/litellm_core_utils/exception_mapping_utils.py	87.50%	5 Missing ⚠️
litellm/proxy/hooks/batch_rate_limiter.py	81.81%	4 Missing ⚠️
litellm/types/integrations/prometheus.py	89.74%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing mynkyu:fix/batch-models-only-read (ba3e3e8) with main (6e569ee)}