perf(proxy): bound event-loop blocking from oversized requests

[yassin-berriai]

Relevant issues

Resolves LIT-3541

Linear ticket

LIT-3541

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Single-worker proxy pods stall their only event loop on oversized requests, which resets co-tenant requests into edge 502s. Two on-loop costs are removed here. Before/after captured against a live proxy and against the exact production functions, with the fix toggled via git stash

Live proxy config (router_settings.enable_pre_call_checks: true, max_request_size_mb unset, model small-model -> openai/gpt-4o-mini)

Token counting short-circuit (Router._pre_call_checks)

Driving the real Router._pre_call_checks with a ~1.5M-token prompt and a deployment that resolves to no max_input_tokens (the affected models), while a co-tenant probe coroutine samples event-loop lag:

# UNFIXED (fix stashed)
_pre_call_checks wall time :    151.5 ms
max co-tenant loop stall   :    151.6 ms
litellm.token_counter calls: 1

# FIXED
_pre_call_checks wall time :      0.0 ms
max co-tenant loop stall   :      0.7 ms
litellm.token_counter calls: 0

The count is now skipped entirely when no deployment in the group declares a context-window limit, so the co-tenant probe stays responsive

End to end, a real completion still routes with enable_pre_call_checks=true

$ curl -s -X POST http://127.0.0.1:4541/v1/chat/completions \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
    -d '{"model":"small-model","messages":[{"role":"user","content":"Reply with exactly: pong"}],"max_tokens":5}'
content: pong
usage: {'completion_tokens': 1, 'prompt_tokens': 12, 'total_tokens': 13, ...}

Surrogate-repair bound (_read_request_body)

A 50MB malformed JSON body posted to /chat/completions. The body is unparseable either way, so the 400 is identical; only the on-loop repair work differs

# UNFIXED (fix stashed) -- two full-body re.sub passes run before the 400
sample 1: http_status=400  time_total=0.603119s
sample 2: http_status=400  time_total=0.605100s
sample 3: http_status=400  time_total=0.591351s

# FIXED -- repair skipped above the cap, 400 raised immediately
sample 1: http_status=400  time_total=0.052693s
sample 2: http_status=400  time_total=0.052415s
sample 3: http_status=400  time_total=0.050723s

{"error":{"message":"Invalid JSON payload: unexpected end of data: line 1 column 52428863 (char 52428862)","type":"invalid_request_error","param":"request_body","code":"400"}}

Type

🚄 Infrastructure

Changes

Two synchronous costs run on the proxy's event loop per request and dominate the stall on oversized payloads. Both are now bounded

litellm.token_counter in Router._pre_call_checks runs tiktoken on the full prompt while routing every completion when enable_pre_call_checks=true. Its result is only ever compared against a deployment's max_input_tokens, so for model groups where no deployment declares that limit (the affected models) the count is pure waste. The count is now computed lazily inside the deployment loop, at most once, and only when a deployment actually declares an integer max_input_tokens; otherwise it is skipped. Token-counting failure still returns the unfiltered deployment list, and context-window filtering is unchanged when a limit is set

The surrogate-repair fallback in _read_request_body decodes the whole body and runs two full-body re.sub passes on an orjson.JSONDecodeError before retrying json.loads. On a multi-MB malformed body that blocks the loop for hundreds of ms. Above MAX_REQUEST_BODY_SIZE_TO_REPAIR_MB (new constant in litellm/constants.py, default 1MB, env-overridable, set to 0 to disable the cap) the repair is skipped and the existing 400 is raised immediately. Bodies at or below the limit are still repaired exactly as before

Regression tests cover both paths: the token count is asserted not to run when no deployment declares max_input_tokens, to run at most once and filter when one does, and the surrogate repair is asserted skipped above the size cap and preserved below it

Docs

Documentation for the new MAX_REQUEST_BODY_SIZE_TO_REPAIR_MB env var: BerriAI/litellm-docs#424

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[yassin-berriai]

@greptileai

[greptile-apps]

Greptile Summary

This PR reduces event-loop blocking from oversized proxy requests. The main changes are:

• Adds an env-configurable cap for JSON surrogate repair work
• Skips surrogate repair for malformed JSON bodies above that cap while keeping the existing 400 response
• Moves router token counting behind a lazy max_input_tokens check and reuses the count across deployments
• Adds tests for both the request-body repair cap and router token-count filtering

Confidence Score: 4/5

The change is narrowly scoped to bounding synchronous request-processing work and includes regression coverage for both modified paths.

The implementation preserves existing behavior for bounded request-body repair and context-window filtering while avoiding unnecessary work for oversized or unconstrained inputs. Tests cover the key behavioral branches, with remaining risk mainly around environment-specific proxy configurations and integrations not exercised here.

litellm/constants.py, litellm/proxy/common_utils/http_parsing_utils.py, litellm/router.py, tests/test_litellm/proxy/common_utils/test_http_parsing_utils.py, tests/test_litellm/test_router.py

T-Rex Logs

What T-Rex did

• Collected the pre-call artifacts to document the base run and the subsequent head run, including the validation script and JSON results.
• Analyzed the head run details: the default cap value is 1, the oversized body yields a 400 Invalid JSON payload, and environment overrides show SET_MODULE_CAP_TO_ZERO and SUBPROCESS_ENV_CAP / IMPORTED_CAP were parsed.

_{Ran code and verified through T-Rex}

_{Reviews (4): Last reviewed commit: "perf(proxy): bound event-loop blocking f..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

This PR bounds two proxy request-path costs for oversized requests. The main changes are:

• Adds an environment-configurable cap for JSON surrogate repair work
• Skips surrogate repair for malformed JSON bodies above the cap while keeping the existing 400 response
• Defers router token counting until a deployment declares an integer max_input_tokens limit
• Reuses the token count across deployments during pre-call filtering
• Adds tests for the repair cap behavior and lazy token-counting path

Confidence Score: 4/5

The changes are narrowly scoped to request parsing and router pre-call checks, with tests covering the intended bounded-work behavior.

The implementation adds focused safeguards around oversized malformed bodies and unnecessary token counting while preserving existing routing and error behavior in covered cases.

No specific files require follow-up from the review.

T-Rex Logs

What T-Rex did

• The token-counting short-circuit tests were run on the base commit to inspect the before-state behavior.
• The token-counting short-circuit tests were run on the head commit to inspect the after-state behavior.
• A validation script containing executable monkeypatched Router._pre_call_checks was prepared and used for both commits.
• The surrogate repair bound tests were examined in the before-state log to document the initial behavior with a 240-byte lone-surrogate body.
• The surrogate repair bound tests were examined in the after-state log to document how the same body is handled after the cap is applied.
• A surrogate repair bound validation script was prepared and used for both commits.

_{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "perf(proxy): bound event-loop blocking f..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 81.25000% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/router.py	70.00%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai re-trigger: pushed a small follow-up routing the new env read through get_env_int (typed helper) instead of a literal os.getenv, so the env-key documentation check passes. Logic is unchanged.

[yassin-berriai]

@greptileai