fix(batches): skip unnecessary batch input file reads

[Sameerlite]

Summary

• skip batch input file pre-processing when no rate limits apply and model-allowlist validation is not required
• add explicit opt-outs for batch input file processing (disable_batch_input_file_rate_limiting, provider/model skip lists, and per-request metadata flag)
• decode model-embedded batch input file IDs before file-content retrieval and route with model credentials to avoid pre-batch 404s

Test plan

• python -m pytest tests/test_litellm/proxy/hooks/test_batch_file_validation.py -q
• validated added regressions for provider/global skip behavior and model-embedded file-id decode path

Fixes LIT-3266

[codecov]

Codecov Report

❌ Patch coverage is 97.54098% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/hooks/batch_rate_limiter.py	97.54%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR optimises the batch pre-call path by skipping unnecessary JSONL downloads when no rate limits apply, a provider is on the operator's skip list, or rate limiting is globally disabled. It also fixes a pre-batch 404 regression by decoding model-embedded file IDs and routing file-content retrieval through the correct deployment credentials before fetching.

• _should_skip_batch_input_file_processing guards every skip path behind _key_requires_batch_model_access_check, so restricted keys always download and validate the JSONL regardless of operator settings.
• _resolve_batch_input_file_fetch_params decodes model-embedded file-<base64> IDs to their original provider file IDs and injects matching API credentials, with HTTPException caught in both the model-embedded and request-model branches so credential failures fall open rather than abort the batch.
• The PR description mentions a per-request metadata flag as an opt-out, but the implementation deliberately omits it; test_should_skip_ignores_client_supplied_metadata_flag confirms client-supplied flags are not honored.

Confidence Score: 5/5

Safe to merge; all skip paths are gated behind the model-access check and operator-only configuration, and the credential-fallback logic fails open without blocking the request.

The three previously raised concerns (client metadata bypass, unguarded HTTPException in the model-embedded credential branch, and provider/model skip lists firing before the access-check guard) are all addressed in the current implementation: the model-access check is the first gate in _should_skip_batch_input_file_processing, the metadata flag is not implemented (and a test confirms it is ignored), and both credential-resolution branches in _resolve_batch_input_file_fetch_params now catch HTTPException. The only remaining gap — unified file IDs with multiple models using only the first model for provider resolution — is an edge case with minimal real-world exposure.

No files require special attention, though the unified-file-ID multi-model path in _get_file_bound_batch_model is worth revisiting if heterogeneous-provider unified files become a supported configuration.

Important Files Changed

Filename	Overview
litellm/proxy/hooks/batch_rate_limiter.py	Adds skip-path helpers (_should_skip_batch_input_file_processing, _get_batch_routing_model, _resolve_batch_provider, _resolve_batch_input_file_fetch_params) to avoid unnecessary JSONL downloads; model-access check gates all skip paths so restricted keys always validate; HTTPException is caught in both credential-resolution branches; minor edge case in unified-file-ID multi-model resolution.
tests/test_litellm/proxy/hooks/test_batch_file_validation.py	Adds 18 new unit tests covering all new skip-path branches; spoofed-provider, forged-file-id, and no-metadata-flag scenarios are explicitly validated; all tests use mocks (no real network calls); existing tests unchanged.

_{Reviews (17): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ mateo-berri
✅ Sameerlite
❌ cursoragent
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Sameerlite]

@greptileai Addressed both comments:

• blocked per-request metadata skip from bypassing model allowlist checks for restricted keys
• made embedded-model credential lookup fail open (fallback) instead of hard-failing batch submission when router/model lookup is unavailable

Re-ran: python -m pytest tests/test_litellm/proxy/hooks/test_batch_file_validation.py -q

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Bugbot Autofix prepared fixes for both issues found in the latest run.

• ✅ Fixed: Early access_group_ids check bypasses wildcard model detection
  • Reordered _key_requires_batch_model_access_check so the "*" and all-proxy-models checks run before the access_group_ids truthiness check, matching _can_object_call_model semantics.
• ✅ Fixed: Embedded-model credential lookup hard-fails instead of failing open
  • Wrapped the embedded-model get_credentials_for_model call in try/except HTTPException and moved the get_original_file_id return outside the try, so router/model lookup failures fall back to defaults with the decoded provider file id.

Preview (f41ec3e2ab)

diff --git a/litellm/proxy/hooks/batch_rate_limiter.py b/litellm/proxy/hooks/batch_rate_limiter.py
--- a/litellm/proxy/hooks/batch_rate_limiter.py
+++ b/litellm/proxy/hooks/batch_rate_limiter.py
@@ -17,7 +17,7 @@
 - async_log_success_event() fires on GET /v1/batches/{id} (batch completion)
 """
 
-from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Union
+from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Tuple, Union
 
 from fastapi import HTTPException
 from pydantic import BaseModel
@@ -25,12 +25,13 @@
 import litellm
 from litellm._logging import verbose_proxy_logger
 from litellm.batches.batch_utils import (
+    _extract_file_access_credentials,
     _get_batch_job_input_file_usage,
     _get_file_content_as_dictionary,
     _get_models_from_batch_input_file_content,
 )
 from litellm.integrations.custom_logger import CustomLogger
-from litellm.proxy._types import UserAPIKeyAuth
+from litellm.proxy._types import SpecialModelNames, UserAPIKeyAuth
 
 if TYPE_CHECKING:
     from opentelemetry.trace import Span as _Span
@@ -98,6 +99,202 @@
         self.internal_usage_cache = internal_usage_cache
         self.parallel_request_limiter = parallel_request_limiter
 
+    def _get_batch_routing_model(self, data: Dict) -> Optional[str]:
+        """Resolve the deployment/model used for this batch from request data."""
+        model = data.get("model")
+        if isinstance(model, str) and model:
+            return model
+
+        input_file_id = data.get("input_file_id")
+        if not isinstance(input_file_id, str) or not input_file_id:
+            return None
+
+        from litellm.proxy.openai_files_endpoints.common_utils import (
+            _is_base64_encoded_unified_file_id,
+            decode_model_from_file_id,
+            get_models_from_unified_file_id,
+        )
+
+        model_from_file_id = decode_model_from_file_id(input_file_id)
+        if model_from_file_id:
+            return model_from_file_id
+
+        unified_file_id = _is_base64_encoded_unified_file_id(input_file_id)
+        if unified_file_id:
+            target_model_names = get_models_from_unified_file_id(unified_file_id)
+            if target_model_names:
+                return target_model_names[0]
+
+        return None
+
+    def _matches_skip_list(self, value: str, skip_list: List[str]) -> bool:
+        if not skip_list:
+            return False
+        for entry in skip_list:
+            if not isinstance(entry, str) or not entry:
+                continue
+            if value == entry or value.startswith(f"{entry}/"):
+                return True
+        return False
+
+    def _should_skip_batch_input_file_processing(
+        self,
+        data: Dict,
+        user_api_key_dict: UserAPIKeyAuth,
+    ) -> bool:
+        """
+        Skip downloading batch input files when configured or when there is
+        nothing to enforce (no applicable rate limits and no model allowlist).
+
+        When the caller's key has a model allowlist to enforce, no skip path
+        is honored: the JSONL must still be downloaded so
+        ``_enforce_batch_file_model_access`` can validate every ``body.model``
+        entry. Otherwise a restricted key could smuggle unauthorized models
+        into the file via an admin-configured skip (global disable,
+        per-model, per-provider) or via the user-controlled
+        ``custom_llm_provider`` field.
+        """
+        if self._key_requires_batch_model_access_check(user_api_key_dict):
+            return False
+
+        from litellm.proxy.proxy_server import general_settings
+
+        if general_settings.get("disable_batch_input_file_rate_limiting") is True:
+            return True
+
+        litellm_metadata = data.get("litellm_metadata") or {}
+        if litellm_metadata.get("skip_batch_input_file_rate_limiting") is True:
+            return True
+
+        batch_model = self._get_batch_routing_model(data)
+        skip_models = (
+            general_settings.get("skip_batch_input_file_rate_limiting_for_models") or []
+        )
+        if batch_model and self._matches_skip_list(batch_model, skip_models):
+            verbose_proxy_logger.debug(
+                f"Skipping batch input file processing for model={batch_model}"
+            )
+            return True
+
+        skip_providers = (
+            general_settings.get("skip_batch_input_file_rate_limiting_for_providers")
+            or []
+        )
+        custom_llm_provider = data.get("custom_llm_provider")
+        if (
+            isinstance(custom_llm_provider, str)
+            and custom_llm_provider in skip_providers
+        ):
+            verbose_proxy_logger.debug(
+                "Skipping batch input file processing for "
+                f"custom_llm_provider={custom_llm_provider}"
+            )
+            return True
+
+        descriptors = self.parallel_request_limiter._create_rate_limit_descriptors(
+            user_api_key_dict=user_api_key_dict,
+            data=data,
+            rpm_limit_type=None,
+            tpm_limit_type=None,
+            model_has_failures=False,
+        )
+        if not self._has_applicable_batch_rate_limits(descriptors):
+            verbose_proxy_logger.debug(
+                "Skipping batch input file processing: no rate limits configured"
+            )
+            return True
+
+        return False
+
+    @staticmethod
+    def _key_requires_batch_model_access_check(
+        user_api_key_dict: UserAPIKeyAuth,
+    ) -> bool:
+        """True when the key may only call a subset of models (JSONL must be checked)."""
+        models = user_api_key_dict.models or []
+        if "*" in models:
+            return False
+        if SpecialModelNames.all_proxy_models.value in models:
+            return False
+        if user_api_key_dict.access_group_ids:
+            return True
+        if not models:
+            return False
+        return True
+
+    @staticmethod
+    def _has_applicable_batch_rate_limits(
+        descriptors: List["RateLimitDescriptor"],
+    ) -> bool:
+        for descriptor in descriptors:
+            rate_limit = descriptor.get("rate_limit") or {}
+            if (
+                rate_limit.get("requests_per_unit") is not None
+                or rate_limit.get("tokens_per_unit") is not None
+                or rate_limit.get("max_parallel_requests") is not None
+            ):
+                return True
+        return False
+
+    def _resolve_batch_input_file_fetch_params(
+        self,
+        file_id: str,
+        custom_llm_provider: str,
+        data: Dict,
+    ) -> Tuple[str, Dict[str, Any]]:
+        """
+        Map proxy-facing file IDs to provider file IDs and credentials.
+
+        Model-embedded IDs (``file-<base64>``) are not unified managed-file IDs;
+        without decoding them, ``afile_content`` is called with the encoded ID
+        and the upstream provider returns 404.
+        """
+        from litellm.proxy.openai_files_endpoints.common_utils import (
+            decode_model_from_file_id,
+            get_credentials_for_model,
+            get_original_file_id,
+        )
+        from litellm.proxy.proxy_server import llm_router
+
+        fetch_kwargs: Dict[str, Any] = {
+            "custom_llm_provider": custom_llm_provider,
+        }
+
+        model_from_file_id = decode_model_from_file_id(file_id)
+        if model_from_file_id:
+            try:
+                credentials = get_credentials_for_model(
+                    llm_router=llm_router,
+                    model_id=model_from_file_id,
+                    operation_context="batch input file read (rate limiting)",
+                )
+                fetch_kwargs.update(_extract_file_access_credentials(credentials))
+                fetch_kwargs["model"] = model_from_file_id
+                provider = credentials.get("custom_llm_provider")
+                if provider:
+                    fetch_kwargs["custom_llm_provider"] = provider
+            except HTTPException:
+                pass
+            return get_original_file_id(file_id), fetch_kwargs
+
+        request_model = data.get("model")
+        if isinstance(request_model, str) and request_model and llm_router is not None:
+            try:
+                credentials = get_credentials_for_model(
+                    llm_router=llm_router,
+                    model_id=request_model,
+                    operation_context="batch input file read (rate limiting)",
+                )
+                fetch_kwargs.update(_extract_file_access_credentials(credentials))
+                fetch_kwargs["model"] = request_model
+                provider = credentials.get("custom_llm_provider")
+                if provider:
+                    fetch_kwargs["custom_llm_provider"] = provider
+            except HTTPException:
+                pass
+
+        return file_id, fetch_kwargs
+
     def _raise_rate_limit_error(
         self,
         status: "RateLimitStatus",
@@ -211,6 +408,7 @@
         file_id: str,
         custom_llm_provider: Literal["openai", "azure", "vertex_ai"] = "openai",
         user_api_key_dict: Optional[UserAPIKeyAuth] = None,
+        data: Optional[Dict] = None,
     ) -> BatchFileUsage:
         """
         Count number of requests and tokens in a batch input file.
@@ -238,14 +436,27 @@
                     user_api_key_dict=user_api_key_dict,
                 )
             else:
+                provider_file_id, fetch_kwargs = (
+                    self._resolve_batch_input_file_fetch_params(
+                        file_id=file_id,
+                        custom_llm_provider=custom_llm_provider,
+                        data=data or {},
+                    )
+                )
                 # For non-managed files, use the standard litellm.afile_content
                 file_content = await litellm.afile_content(
-                    file_id=file_id,
-                    custom_llm_provider=custom_llm_provider,
+                    file_id=provider_file_id,
                     user_api_key_dict=user_api_key_dict,
+                    **fetch_kwargs,
                 )
 
-            file_content_as_dict = _get_file_content_as_dictionary(file_content.content)
+            file_content_bytes = getattr(file_content, "content", None)
+            if not isinstance(file_content_bytes, bytes):
+                raise ValueError(
+                    f"Expected bytes content from file retrieval for {file_id}, "
+                    f"got {type(file_content_bytes)}"
+                )
+            file_content_as_dict = _get_file_content_as_dictionary(file_content_bytes)
 
             # Validate every model named in the batch JSONL against the
             # caller's per-key model allowlist. Without this, a caller
@@ -435,6 +646,11 @@
                 )
                 return data
 
+            if self._should_skip_batch_input_file_processing(
+                data=data, user_api_key_dict=user_api_key_dict
+            ):
+                return data
+
             # Get custom_llm_provider for token counting
             custom_llm_provider = data.get("custom_llm_provider", "openai")
 
@@ -446,6 +662,7 @@
                 file_id=input_file_id,
                 custom_llm_provider=custom_llm_provider,
                 user_api_key_dict=user_api_key_dict,
+                data=data,
             )
 
             verbose_proxy_logger.debug(

diff --git a/tests/test_litellm/proxy/hooks/test_batch_file_validation.py b/tests/test_litellm/proxy/hooks/test_batch_file_validation.py
--- a/tests/test_litellm/proxy/hooks/test_batch_file_validation.py
+++ b/tests/test_litellm/proxy/hooks/test_batch_file_validation.py
@@ -14,7 +14,6 @@
 
 from litellm.proxy._types import LitellmUserRoles, UserAPIKeyAuth
 
-
 # ---------------------------------------------------------------------------
 # Token counter — covers all three batch payload shapes
 # ---------------------------------------------------------------------------
@@ -261,6 +260,112 @@
 
 
 @pytest.mark.asyncio
+async def test_pre_call_skips_file_fetch_when_disabled_in_general_settings():
+    from litellm.proxy.hooks.batch_rate_limiter import _PROXY_BatchRateLimiter
+
+    rate_limiter = _PROXY_BatchRateLimiter(
+        internal_usage_cache=MagicMock(),
+        parallel_request_limiter=MagicMock(),
+    )
+    user = UserAPIKeyAuth(api_key="sk-ok", user_id="alice", models=["*"])
+
+    with patch(
+        "litellm.proxy.proxy_server.general_settings",
+        {"disable_batch_input_file_rate_limiting": True},
+    ):
+        result = await rate_limiter.async_pre_call_hook(
+            user_api_key_dict=user,
+            cache=MagicMock(),
+            data={"input_file_id": "file-abc123"},
+            call_type="acreate_batch",
+        )
+
+    assert result == {"input_file_id": "file-abc123"}
+    rate_limiter.parallel_request_limiter._create_rate_limit_descriptors.assert_not_called()
+
+
+@pytest.mark.asyncio
+async def test_pre_call_skips_file_fetch_for_configured_provider():
+    from litellm.proxy.hooks.batch_rate_limiter import _PROXY_BatchRateLimiter
+
+    rate_limiter = _PROXY_BatchRateLimiter(
+        internal_usage_cache=MagicMock(),
+        parallel_request_limiter=MagicMock(),
+    )
+    user = UserAPIKeyAuth(api_key="sk-ok", user_id="alice", models=["*"])
+
+    with patch(
+        "litellm.proxy.proxy_server.general_settings",
+        {"skip_batch_input_file_rate_limiting_for_providers": ["hosted_vllm"]},
+    ):
+        result = await rate_limiter.async_pre_call_hook(
+            user_api_key_dict=user,
+            cache=MagicMock(),
+            data={
+                "input_file_id": "file-abc123",
+                "custom_llm_provider": "hosted_vllm",
+            },
+            call_type="acreate_batch",
+        )
+
+    assert result["custom_llm_provider"] == "hosted_vllm"
+
+
+@pytest.mark.asyncio
+async def test_count_input_file_usage_decodes_model_embedded_file_id():
+    import base64
+
+    from litellm.proxy.hooks.batch_rate_limiter import _PROXY_BatchRateLimiter
+
+    original_file_id = "file-provider-xyz"
+    encoded_payload = (
+        base64.urlsafe_b64encode(
+            f"litellm:{original_file_id};model,my-vllm-batch".encode()
+        )
+        .decode()
+        .rstrip("=")
+    )
+    encoded_file_id = f"file-{encoded_payload}"
+
+    rate_limiter = _PROXY_BatchRateLimiter(
+        internal_usage_cache=MagicMock(),
+        parallel_request_limiter=MagicMock(),
+    )
+
+    mock_content = MagicMock()
+    mock_content.content = b'{"custom_id": "1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "my-vllm-batch", "messages": [{"role": "user", "content": "hi"}]}}\n'
+
+    with (
+        patch(
+            "litellm.afile_content",
+            new=AsyncMock(return_value=mock_content),
+        ) as mock_afile_content,
+        patch(
+            "litellm.proxy.proxy_server.llm_router",
+            MagicMock(),
+        ),
+        patch(
+            "litellm.proxy.openai_files_endpoints.common_utils.get_credentials_for_model",
+            return_value={
+                "api_key": "test-key",
+                "api_base": "http://vllm:8000/v1",
+                "custom_llm_provider": "hosted_vllm",
+            },
+        ),
+    ):
+        await rate_limiter.count_input_file_usage(
+            file_id=encoded_file_id,
+            custom_llm_provider="openai",
+            user_api_key_dict=UserAPIKeyAuth(api_key="sk-ok", user_id="alice"),
+            data={},
+        )
+
+    mock_afile_content.assert_awaited_once()
+    assert mock_afile_content.await_args.kwargs["file_id"] == original_file_id
+    assert mock_afile_content.await_args.kwargs["custom_llm_provider"] == "hosted_vllm"
+
+
+@pytest.mark.asyncio
 async def test_pre_call_skips_check_when_no_models_present():
     """Files without any `body.model` (corrupt or empty) must not 500;
     the rate limiter logs a warning elsewhere and proceeds."""

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 4f9cedd. Configure here.}

[Sameerlite]

@greptileai

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 5 · PR risk: 0/10

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@Sameerlite You marked this as draft -- not ready for review?

[Sameerlite]

Hi yes, let me retest and verify if this doesn’t create any side effects. Plus there is no rush on this feat

…

On Sat, 30 May 2026 at 9:52 PM, Mateo Wang ***@***.***> wrote: *mateo-berri* left a comment (BerriAI/litellm#29114) <#29114 (comment)> @Sameerlite <https://github.com/Sameerlite> You marked this as draft -- not ready for review? — Reply to this email directly, view it on GitHub <#29114?email_source=notifications&email_token=BXGLABASPCTFH2SWWEDPXD345MDFRA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJYGM2DCMBUGAZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4583410403>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BXGLABBD6Z4PXISX4KHITVD45MDFRAVCNFSM6AAAAACZQLIFZKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DKOBTGQYTANBQGM> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/BXGLABGMDIXEOSR2Q3ADG6L45MDFRA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJYGM2DCMBUGAZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/BXGLABDUJUTKZXKOCBEDAKD45MDFRA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJYGM2DCMBUGAZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[mateo-berri]

Got it thanks

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

LGTM; thanks!