fix(logging): stop pinning large request payloads past request end

[yassin-berriai]

Relevant issues

Linear ticket

Resolves LIT-4434

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

A production deployment reported stepwise memory growth that never returns to baseline after traffic bursts, ending in periodic restarts to avoid OOM. Their traffic averages ~73K tokens/request (max ~1M) with a PII content-filter guardrail on most requests. The growth reproduces identically on v1.84.0 and v1.90.0, so this is not a version regression; it is a payload-retention problem that large-token traffic surfaces

Rig (mirrors the reporter's config): live proxy + Postgres + Redis cache, litellm_content_filter guardrail (mode: pre_call, MASK patterns, default_on: true), callbacks: [prometheus], store_prompts_in_spend_logs: error-only, plus a local openai-compatible upstream for the burst load so the burst itself is free; the end-to-end proof below uses the real OpenAI API. Load: bursts of 30 chat completions x ~300KB message content, 8 concurrent, then a 10s idle read of the worker's RSS

Before (unfixed, v1.90.0 tag 6e8282d406; the current staging base reproduces the same curve):

$ python burst.py 5 30 300 8
baseline rss_total=733.4MB
burst 1:  after 10s idle: rss_total=766.5MB
burst 2:  after 10s idle: rss_total=787.3MB
burst 3:  after 10s idle: rss_total=809.0MB
burst 4:  after 10s idle: rss_total=831.9MB

~16-29MB retained per burst, never released; over ~20 bursts one worker ratchets past 1.2GB. A gc census 10 seconds after each burst finds ~55 complete request dicts (~20MB of message content) still referenced, and gc.DEBUG_SAVEALL shows them sitting in reference cycles through the request's Logging object

After (this branch; captured at 18494e5993, whose runtime behavior is identical to head 7d53a23d5b which only widens a type annotation), same rig, same load:

$ python burst.py 6 30 300 8
baseline rss_total=876.1MB
burst 1:  after 10s idle: rss_total=967.3MB   (first-burst warmup: arenas, connection pools)
burst 2:  after 10s idle: rss_total=969.2MB
burst 3:  after 10s idle: rss_total=971.0MB
burst 4:  after 10s idle: rss_total=972.6MB
burst 5:  after 10s idle: rss_total=973.3MB
burst 6:  after 10s idle: rss_total=974.6MB
$ python burst.py 8 30 300 8   # continued on the same worker
burst 7..14: 1066.5 -> 1067.0 -> 1067.6 -> 1070.1 -> 1071.0 -> 1071.4 -> 1071.9 -> 1072.1MB

Growth per burst drops from ~16-29MB to under 1MB and the curve flattens instead of ratcheting

End-to-end sanity on the fixed proxy against the real OpenAI API (costs real $): the PII guardrail still masks before the provider sees the payload, and the spend log still records the guardrail info

$ curl -s http://localhost:4434/v1/chat/completions -H "Authorization: Bearer sk-..." \
    -d '{"model":"gpt-5.4-nano","messages":[{"role":"user","content":"my phone is 13812345678 and my email is jane@example.com. Repeat back exactly the phone and email I just gave you."}]}'
content: Sorry—I can’t repeat back those exact phone number or email details you provided.
usage: 60 total tokens

$ psql ... -c "SELECT model, metadata->'guardrail_information'->0->>'masked_entity_count' FROM \"LiteLLM_SpendLogs\" ORDER BY \"startTime\" DESC LIMIT 2;"
 gpt-5.4-nano | {"email": 1, "china_phone": 1}
 gpt-5.4-nano | {"email": 1, "china_phone": 1}

Why this is not a curl-visible bug: the defect is in-process memory retention, so the metric is the worker's RSS across bursts plus a gc census of surviving request dicts; the curls above prove the touched paths (logging, caching handler, router deployment selection, guardrails) still behave end to end

Type

🐛 Bug Fix

Changes

Three independent retention points each kept a full request payload (messages included) alive after the request finished

litellm_core_utils/litellm_logging.py: pre_call/post_call stored their entire locals() (the messages, the Logging object, complete_input_dict) into the module-level litellm.error_logs dict. Nothing in the codebase reads that dict, so the write pinned the most recent request's whole payload per worker, forever. The two writes are removed; the (now always empty) module attribute stays for import compatibility

caching/caching_handler.py: LLMCachingHandler stored the raw request kwargs while itself hanging off logging_obj._llm_caching_handler, and the kwargs contain litellm_logging_obj, closing a reference cycle Logging -> LLMCachingHandler -> kwargs -> Logging. Cyclic garbage is only reclaimed by generational GC, so megabytes of dead payload lingered until a rare gen-2 pass, and the churn of large blocks fragments the allocator into a permanent RSS high-water mark. The handler now drops litellm_logging_obj from its stored kwargs at every assignment site; the caching layer never reads it (cache-key generation ignores litellm-internal params)

router.py + router_utils/pre_call_checks/io_token_rate_limit_check.py: _update_kwargs_with_deployment stored every request's kwargs in the ITPM/OTPM contextvar even when no deployment configures itpm/otpm. Pooled resources created mid-request (a redis connection's captured Context was one confirmed holder) extend that pin far past the request. The slot is now populated only for deployments that actually have io token limits and is overwritten with None otherwise, which also releases a previous request's kwargs when an asyncio context is reused. The reservation-sentinel stripping still runs unconditionally

Tests: one regression test per retention point, in the mapped test files. Each fails on the unfixed tree (verified by reverting the four production files and re-running: 4 failed) and passes with the fix

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR fixes three independent post-request payload-retention points that together caused stepwise RSS growth under large-token traffic. The root cause in each case was a reference being held — in module-level state, a reference cycle through the caching handler, or a long-lived asyncio context var — that prevented Python's refcount from reclaiming the request's messages after the request finished.

• litellm_logging.py: Removes two litellm.error_logs[...] = locals() writes in pre_call/post_call that pinned the full request payload (messages + Logging object) in a module-level dict for the lifetime of the worker; the dict attribute is kept for import compatibility.
• caching_handler.py: Introduces _drop_logging_obj_from_kwargs (applied at all three self.request_kwargs assignment sites) to break the Logging → LLMCachingHandler → kwargs → Logging reference cycle; only the logging object is stripped, all other fields including messages are preserved.
• router.py / io_token_rate_limit_check.py: set_io_token_rate_limit_request_kwargs now accepts store_in_context; for deployments without itpm/otpm limits the context var is overwritten with None (not skipped), while sentinel-stripping continues unconditionally for all deployments.

Confidence Score: 5/5

Safe to merge — all three changes are targeted removals or conditional guards with no new behaviour introduced on existing code paths.

Each fix is minimal and well-scoped: the logging change is a straight deletion of unused writes, the caching-handler change only omits one key from a shallow copy, and the context-var change replaces an unconditional store with a conditional one while keeping sentinel-stripping unconditional. The consumer side (io_token_pre_call_check) already guards against a None context var by bailing early when the deployment has no limits.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/litellm_logging.py	Removes two writes of locals() into module-level litellm.error_logs; the attribute is preserved for import compatibility but nothing ever read from it, so the removal has no functional side-effects.
litellm/caching/caching_handler.py	Introduces _drop_logging_obj_from_kwargs to strip litellm_logging_obj before assigning request_kwargs on LLMCachingHandler; applied at all three assignment sites (constructor + sync/async get-cache paths). Type annotation widening to dict[str, object] is the sole delta from the previously-reviewed commit.
litellm/router_utils/pre_call_checks/io_token_rate_limit_check.py	set_io_token_rate_limit_request_kwargs gains a store_in_context flag; sentinel-stripping still runs unconditionally, and the context var is overwritten with None for non-io-limited deployments so reused contexts don't retain a previous request's kwargs.
litellm/router.py	Single-line change: passes store_in_context=deployment_has_io_token_limits(deployment); deployment_has_io_token_limits is properly imported and checks itpm/otpm across litellm_params and model_info.
tests/test_litellm/test_router/test_io_token_rate_limits.py	Adds TestContextSlotRetention with three tests covering store_in_context=False/True and router integration for plain vs io-limited deployments. All mock-only, no network calls.
tests/test_litellm/caching/test_caching_handler.py	Adds a unit test verifying LLMCachingHandler strips litellm_logging_obj from request_kwargs while preserving other fields.
tests/test_litellm/litellm_core_utils/test_litellm_logging.py	Adds a unit test that calls pre_call and post_call and asserts litellm.error_logs stays empty after both calls.

_{Reviews (2): Last reviewed commit: "fix(logging): stop pinning large request..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai please review the current head 7d53a23 (delta from the reviewed commit is a single type-annotation widening in caching_handler.py)

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_large_payload_logging_retention (7d53a23) with litellm_internal_staging (9121ae3)}