feat(cache): add a guarded native response-cache resolver foundation

[devin-ai-integration]

TLDR

Problem this solves:

• No bridge path yet reuses a Cache facade's resolved native service
• A stale native runtime could be reused after the facade was reconfigured

How it solves it:

• Adds a resolver that reuses the facade's native service while its guard matches
• Declines native inference when the runtime no longer matches the facade
• Facades without a native runtime keep the Python callback path
• Default catalog stays PYTHON_ONLY; no production route calls the resolver yet

User Flow

This PR is a resolver foundation with no production consumer, so end-user behavior is unchanged in this revision

Before: a configured response cache behaves exactly as today

1. A developer configures litellm.cache = Cache(type="local")
2. They send POST https://litellm-domain/v1/chat/completions twice with the same messages
3. The second response is served from the Python cache path

After: identical

1. A developer configures litellm.cache = Cache(type="local")
2. They send POST https://litellm-domain/v1/chat/completions twice with the same messages
3. The second response is served from the Python cache path; the native resolver added here is only reachable from tests until a later PR wires a route to it

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a LiteLLM maintainer to review my PR (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

No end-to-end proof is possible for this revision: the default catalog keeps every cache rule PYTHON_ONLY and no production route calls _CacheResolver, so a live proxy behaves identically at the merge base and at the tip. The resolver's native-reuse, decline, and Python-fallback behavior is covered by the behavioral tests in tests/test_litellm_rust/test_cache.py (test_inference_resolver_uses_the_configured_native_cache_directly and test_inference_resolver_declines_a_native_runtime_whose_facade_changed), and a live before/after run will be attached by the PR that wires the first production route to the resolver

Type

• Feature

Caveats (if any)

Medium

• After facade.cache is replaced, the facade's own get_cache/add_cache still use the stale _native_cache until the owner rebuilds it; this PR only stops the resolver from reusing it

Low

• A native runtime whose guard could not be captured is never reused by the resolver and declines instead of silently taking the Python path
• Nothing in this PR changes an inference route; enabling native response caching needs a catalog rollout and a route consumer in a later PR

Final Attestation

I have reviewed this PR and confirm that it is ready for review

Link to Devin session: https://app.devin.ai/sessions/6663034c50e246e686b624c794a71fdb
Open in Devin Desktop: https://app.devin.ai/desktop/session/6663034c50e246e686b624c794a71fdb?variant=devin
Requested by: @yujonglee-berri

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; the previously reported stale native backend reuse is guarded and no new actionable failures were identified.

Summary

The PR adds the guarded foundation for resolving an existing native response-cache service without changing production routing.

• Captures facade state when constructing a native cache runtime.
• Reuses the native service only while the facade guard still matches.
• Declines resolution after facade reconfiguration instead of falling back through the changed facade.
• Preserves Python callback behavior for facades without a native runtime.
• Adds behavioral tests for native reuse, Python fallback, and stale-runtime rejection.

_{Reviews (2) · Last reviewed commit: "fix(cache): decline native inference whe..."}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_rust_response_cache_foundation (132e897) with main (5beac4f)¹}

Footnotes

1. No successful run was found on main (966f695) during the generation of this report, so 5beac4f was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[devin-ai-integration]

@greptileai please re-review the new tip: the resolver now declines on a facade mismatch instead of taking the Python callback path