fix(proxy): increase spend counter cache capacity

[wzj1228516103]

TLDR

Problem this solves:

• Spend counters inherit a 200-entry in-memory limit
• Active counters are evicted before their TTL expires

How it solves it:

• Give the dedicated spend counter cache a bounded 10,000-entry limit
• Add a regression test covering 300 active budget scopes

User Flow

Before: a proxy serving more than 200 active budget scopes repeatedly reloads warm spend counters

1. An operator runs the proxy with database-backed budgets and no Redis
2. Applications send POST https://litellm-domain/v1/chat/completions through more than 200 distinct budget scopes
3. Earlier active scopes are evicted before their 60-second TTL and later requests trigger avoidable durable-store reads

After: the same traffic keeps active spend counters available during the configured TTL

1. An operator runs the proxy with database-backed budgets and no Redis
2. Applications send the same POST https://litellm-domain/v1/chat/completions requests through more than 200 distinct budget scopes
3. Active scopes remain cached and budget checks avoid repeated reseeds

Relevant issues

Fixes #40221

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• The targeted pytest command is blocked locally by missing MSVC link.exe while building LiteLLM's Rust/PyO3 editable package
• My PR passes all required CI/CD checks
• My PR's scope is isolated to the spend counter cache capacity
• I have received a Greptile Confidence Score of at least 4/5 before requesting maintainer review

Screenshots / Proof of Fix

After (8cb42ca)

1. py -3.13 -m py_compile litellm/proxy/proxy_server.py tests/test_litellm/proxy/test_proxy_server.py completed successfully
2. git diff --check completed successfully
3. The targeted pytest, Black, and Ruff commands were attempted but blocked before test collection or linting by the missing MSVC link.exe required for the editable Rust/PyO3 build

Type

Bug Fix

Caveats

The cache remains bounded at 10,000 entries and keeps the existing TTL behavior

Final Attestation

• The regression test covers the reported active-counter eviction scenario

[CLAassistant]

All committers have signed the CLA.

[greptile-apps]

Greptile Summary

The PR increases the dedicated in-memory spend-counter cache capacity from 200 to 10,000 entries while retaining existing cache behavior

• Adds an explicit bounded capacity for spend counters
• Adds a regression test covering more than 200 active budget scopes

Confidence Score: 5/5

The PR appears safe to merge and the capacity change is bounded and covered by a focused regression test

No actionable failure remains; the supplied cache preserves prior semantics while increasing the eviction threshold as intended

Important Files Changed

Filename	Overview
litellm/proxy/proxy_server.py	Configures the dedicated spend-counter cache with a bounded 10,000-entry in-memory capacity
tests/test_litellm/proxy/test_proxy_server.py	Adds a focused regression test proving an early counter remains cached after inserting 300 scopes

_{Reviews (1): Last reviewed commit: "fix(proxy): increase spend counter cache..." | Re-trigger Greptile}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing wzj1228516103:litellm_40221_spend_counter_cache (8cb42ca) with litellm_internal_staging (1af7a40)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!