perf: increase default LRU cache size to reduce multi-model thrash

[emerzon]

Summary

Increase DEFAULT_MAX_LRU_CACHE_SIZE from 16 to 64 and sync docs.

• litellm/constants.py: default changed to 64 (still fully env-overridable).
• docs/my-website/docs/proxy/config_settings.md: updated DEFAULT_MAX_LRU_CACHE_SIZE doc default from 16 to 64.

Why

DEFAULT_MAX_LRU_CACHE_SIZE is shared by several @lru_cache hot paths:

• litellm.utils.get_model_info
• litellm.utils._cached_get_model_info_helper
• litellm.router.Router._cached_get_model_group_info
• litellm.cost_calculator._model_contains_known_llm_provider
• litellm.utils._select_tokenizer_helper

With the default of 16, workloads with >16 active model keys (or model groups) can force near-constant eviction and very low cache hit rates.

Benchmark results (local, controlled)

All runs were executed in fresh Python processes with only DEFAULT_MAX_LRU_CACHE_SIZE changed.

1) get_model_info(model, provider) - 50 models, round-robin

• cache 16: 0.086815 ms/op, hit rate 0%
• cache 64: 0.000414 ms/op, hit rate 99.75%
• ~210x faster

2) _cached_get_model_info_helper(model, provider) - 50 models, round-robin

• cache 16: 0.014779 ms/op, hit rate 0%
• cache 64: 0.000286 ms/op, hit rate 99.75%
• ~52x faster

3) Router _cached_get_model_group_info - 50 groups, round-robin

• cache 16: 0.255323 ms/op, hit rate 0%
• cache 64: 0.000169 ms/op, hit rate 99.75%
• large reduction in per-request overhead

4) Realistic random traffic shape (uniform over 50 models)

get_model_info(model, provider):

• cache 16: hit rate 32.17%, 0.068214 ms/op
• cache 64: hit rate 99.9%, 0.000193 ms/op

This reproduces the low-hit-rate pattern expected when active model cardinality is much larger than cache size.

5) completion_cost end-to-end check (openai-only, 50 models)

Median over 5 runs:

• cache 16: 0.216659 ms/op
• cache 64: 0.120326 ms/op
• ~1.8x faster

Why 64 and not 256

64 removes thrash for common 50-model working sets while keeping memory growth modest.

In a local tracemalloc run populating both get_model_info and helper caches:

• size 16: current delta ~766 KB
• size 64: current delta ~937 KB
• size 256: current delta ~1.67 MB

So 64 gives the major performance win with significantly lower memory growth than 256.

Backward compatibility

• No API changes.
• Existing env override remains unchanged: DEFAULT_MAX_LRU_CACHE_SIZE can still be set explicitly.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Feb 13, 2026 7:10pm

[greptile-apps]

Greptile Overview

Greptile Summary

Increases the default LRU cache size (DEFAULT_MAX_LRU_CACHE_SIZE) from 16 to 64 to reduce cache thrashing in multi-model production workloads, and adds a guard test for the new default.

• litellm/constants.py: Default bumped from 16 → 64. This constant is used by 5 lru_cache-decorated hot-path functions (get_model_info, _cached_get_model_info_helper, _cached_get_model_group_info, _model_contains_known_llm_provider, _select_tokenizer_helper). The env-var override remains unchanged.
• tests/test_litellm/test_constants.py: New test test_default_max_lru_cache_size_is_64 guards the production default. Properly clears the env var before reloading the module.
• Documentation gap: docs/my-website/docs/proxy/config_settings.md still references the old default of 16 and should be updated.

Confidence Score: 5/5

• This PR is safe to merge — it changes a single integer constant with no API or behavioral impact.
• The change is minimal (one constant value from 16 → 64), fully backward-compatible (env-var override preserved), and backed by benchmarks showing significant cache-hit improvements. The test is correct and makes no network calls. The only gap is an out-of-scope documentation file that still references the old default.
• No files in the changeset require special attention. However, docs/my-website/docs/proxy/config_settings.md (not in changeset) should be updated to reflect the new default.

Important Files Changed

Filename	Overview
litellm/constants.py	Changed DEFAULT_MAX_LRU_CACHE_SIZE default from 16 to 64. Added helpful inline comment explaining the rationale. Env var override is preserved.
tests/test_litellm/test_constants.py	Added test_default_max_lru_cache_size_is_64 guard test — properly uses monkeypatch to clear the env var before reloading the module. No network calls.

Sequence Diagram

sequenceDiagram
    participant Client
    participant Router as litellm.router
    participant Utils as litellm.utils
    participant CostCalc as litellm.cost_calculator
    participant Constants as litellm.constants
    participant LRU as functools.lru_cache

    Note over Constants: DEFAULT_MAX_LRU_CACHE_SIZE = 64<br/>(was 16, env-overridable)

    Client->>Router: completion request
    Router->>LRU: _cached_get_model_group_info(group)
    LRU-->>Router: cached ModelGroupInfo (if hit)
    Router->>Utils: get_model_info(model, provider)
    Utils->>LRU: lru_cache lookup (maxsize=64)
    LRU-->>Utils: cached ModelInfo (if hit)
    Utils->>Utils: _cached_get_model_info_helper(model, provider)
    Utils->>LRU: lru_cache lookup (maxsize=64)
    LRU-->>Utils: cached ModelInfoBase (if hit)
    Utils->>Utils: _select_tokenizer_helper(model)
    Utils->>LRU: lru_cache lookup (maxsize=64)
    LRU-->>Utils: cached tokenizer (if hit)
    Router->>CostCalc: completion_cost()
    CostCalc->>LRU: _model_contains_known_llm_provider(model)
    LRU-->>CostCalc: cached bool (if hit)
    CostCalc-->>Router: cost result
    Router-->>Client: response

Loading

_{Last reviewed commit: 73e589f}

[greptile-apps]

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (1)

docs/my-website/docs/proxy/config_settings.md
Documentation still references old default. config_settings.md still says "Default is 16" for DEFAULT_MAX_LRU_CACHE_SIZE. This should be updated to 64 to stay consistent with the code change.

| DEFAULT_MAX_LRU_CACHE_SIZE | Default maximum size for LRU cache. Default is 64

[Copilot]

Pull request overview

Increases LiteLLM’s shared functools.lru_cache default maxsize to reduce cache eviction thrash in multi-model workloads, and adds a test to lock in the new production default.

Changes:

• Increase DEFAULT_MAX_LRU_CACHE_SIZE default from 16 to 64 (still env-overridable).
• Add a unit test asserting the default is 64 when the env var is unset.

Reviewed changes

Copilot reviewed 1 out of 1 changed files in this pull request and generated no comments.

File	Description
litellm/constants.py	Updates the default LRU cache size constant and documents rationale inline.
tests/test_litellm/test_constants.py	Adds a regression test to guard the new default value.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.