refactor(rate-limits): move the v3 limiter per-request stash off request metadata onto a ContextVar

[mateo-berri]

TLDR

Problem this solves:

• v3 limiter stashed internal bookkeeping in request body metadata
• On Responses-style routes that metadata is a provider parameter
• Internal keys leaked to OpenAI and polluted caller metadata
• Containment needed denylist stripping plus dual-channel mirror writes

How it solves it:

• Per-request stash moved to an asyncio ContextVar
• Typed RequestRateLimiterStash shared by every callback of a request
• Stash pinned to the owning litellm_call_id, so nested calls (LLM-judge guardrails, silent experiments) cannot release it
• Request body is never created or mutated by the limiter
• Strip helpers, mirror writes, and denylist entries deleted

Relevant issues

Fixes #35197. Supersedes #35207, which contained the leak by rerouting the body writes into the proxy-internal bucket and has since merged into staging; this PR replaces that containment by removing the body writes entirely

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy against real OpenAI (openai/gpt-5.6), key generated with {"tpm_limit": 100000, "rpm_limit": 100, "max_parallel_requests": 5}

Before, at staging tip 6f1625d: the limiter's stash reaches OpenAI and comes back stored in the response's provider-side metadata

$ curl -s http://localhost:41873/v1/responses -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
    -d '{"model": "gpt-5.6", "input": "Reply with exactly: hello"}'
HTTP 200 ... "metadata":{"_litellm_tpm_reserved_model":"gpt-5.6"} ...

$ curl -s http://localhost:41873/v1/responses ... -d '{"model": "gpt-5.6", "input": "Reply with exactly: hello", "metadata": {"env": "qa"}}'
HTTP 200 ... "metadata":{"env":"qa","_litellm_tpm_reserved_model":"gpt-5.6"} ...

After, at this PR's head f507a11: body forwarded untouched

$ curl -s http://localhost:47311/v1/responses ... -d '{"model": "gpt-5.6", "input": "Reply with exactly: hello"}'
HTTP 200 ... metadata: {}

$ curl -s http://localhost:47311/v1/responses ... -d '{"model": "gpt-5.6", "input": "Reply with exactly: hello", "metadata": {"env": "qa"}}'
HTTP 200 ... metadata: {"env": "qa"}

Rate limiting itself is unchanged; the after run reproduces the before run's chat headers byte for byte

$ curl -s -D - http://localhost:47311/v1/chat/completions -H "Authorization: Bearer $KEY" \
    -d '{"model": "gpt-5.6", "messages": [{"role": "user", "content": "Reply with exactly: hi"}], "max_tokens": 20}'
HTTP/1.1 200 OK
x-ratelimit-api_key-limit-max_parallel_requests: 5
x-ratelimit-api_key-limit-requests: 100
x-ratelimit-api_key-limit-tokens: 100000
x-ratelimit-api_key-remaining-max_parallel_requests: 4
x-ratelimit-api_key-remaining-requests: 97
x-ratelimit-api_key-remaining-tokens: 99943

Enforcement still fires, on a key with {"rpm_limit": 2}

call 1 -> HTTP 200
call 2 -> HTTP 200
call 3 -> HTTP 429  "Rate limit exceeded for api_key: ce36e09d... Limit type: requests. Current limit: 2, Remaining: 0"

TPM reservation still reconciles to actual usage through the ContextVar: call 1 holds a 25-token reservation in flight (remaining-tokens: 99975), settles to the actual 15 once logged, and call 2 in flight shows 99960 = 100000 - 15 - 25

call 1 in-flight remaining-tokens: 99975   (actual total_tokens: 15)
call 2 in-flight remaining-tokens: 99960   (actual total_tokens: 15)

Type

🐛 Bug Fix
🧹 Refactoring

Changes

The v3 parallel-request limiter kept its per-request bookkeeping (TPM reservation, reserved scopes, parallel slot acquisition, rate-limit response snapshot, refund-released flag) in the request body's metadata/litellm_metadata channels and on data top level. That state now lives in a single typed RequestRateLimiterStash dataclass on a module-level ContextVar, created by the pre-call hook and read or cleared by the success/failure logging callbacks, the disconnect release, and the post-call hooks. The logging worker captures the request context at enqueue time and every task forked from the request shares the same stash instance, so the refund and slot release stay idempotent across sibling callbacks exactly as before

Because the limiter no longer touches the body, the caller-injection stripping (_strip_stash_keys_from_all_channels), the metadata mirror and lookup helpers, the SLO scrubbing, and the all_litellm_params denylist entries for the stash keys are deleted. dynamic_rate_limiter_v3 writes its rate-limit response snapshot to the same stash instead of data, and async_release_max_parallel_requests_on_disconnect no longer needs request_data. Client-supplied lookalike keys are now inert data the limiter never reads, rather than something to strip

Since the stash is context-inherited, nested LiteLLM calls made inside the request (LLM-judge guardrails, silent experiments) would also see it from their own logging callbacks and could release the owning request's parallel slot or refund its TPM reservation early. The stash therefore records the request's litellm_call_id at pre-call, and the kwargs-driven callbacks (log success/failure and the header mirror) ignore a stash owned by a different call id. Router retries and fallbacks reuse the request's call id (litellm.utils.function_setup only mints one when absent), so the shared refund and slot idempotency across attempts is unchanged; nested calls mint fresh ids and are shut out. The guard only rejects a positive mismatch: an unclaimed stash or a callback without a call id behaves exactly as before

Tests updated to seed the ContextVar stash instead of metadata dicts, plus new regressions: the chat body is deep-equal before and after pre-call, Responses bodies with and without caller metadata are forwarded byte-identical, injected stash lookalikes cannot trigger a refund, a full chat lifecycle proves reservation, refund, slot release, and double-refund idempotency through the stash, nested-call events with a foreign call id leave the owner's slot and reservation untouched while owner events still release them, and one-sided call-id metadata (unclaimed stash or callback without an id) keeps working

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

The PR moves v3 rate-limiter request bookkeeping from provider-facing request metadata into a typed per-request ContextVar stash.

• Adds call-ID ownership checks for logging callbacks and nested LiteLLM calls.
• Updates success, failure, streaming-disconnect, and rate-limit-header paths to consume the ContextVar state.
• Removes metadata mirror, stripping, and denylist behavior previously needed to contain internal stash fields.
• Updates limiter tests to exercise request-body preservation, lifecycle reconciliation, idempotency, and nested-call handling.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains within the scope of the eligible follow-up findings.

Important Files Changed

Filename	Overview
litellm/proxy/hooks/parallel_request_limiter_v3.py	Introduces the typed ContextVar-backed request stash and migrates reservation, parallel-slot, response-header, and cleanup bookkeeping to it.
litellm/proxy/hooks/dynamic_rate_limiter_v3.py	Stores dynamic rate-limit responses in the shared request stash instead of mutating request data.
litellm/proxy/common_request_processing.py	Updates streaming-disconnect cleanup to release limiter state without passing request data.
litellm/proxy/utils.py	Adapts the proxy disconnect-release helper to the ContextVar-based limiter interface.
litellm/types/utils.py	Removes obsolete request-parameter denylist entries for the former metadata-backed stash keys.
tests/test_litellm/proxy/hooks/test_parallel_request_limiter_v3.py	Reworks limiter lifecycle tests around isolated ContextVar state and adds ownership and request-mutation regressions.
tests/test_litellm/proxy/hooks/test_tpm_concurrent.py	Updates concurrent TPM reservation and reconciliation tests to use the ContextVar stash.

_{Reviews (3): Last reviewed commit: "test(rate-limits): drop the removed data..." | Re-trigger Greptile}

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[codecov]

Codecov Report

❌ Patch coverage is 98.88889% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/hooks/parallel_request_limiter_v3.py	98.82%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_v3_limiter_contextvar_stash (3b62b90) with litellm_internal_staging (4eecf7a)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (fb79a4e) during the generation of this report, so 4eecf7a was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai