fix: backport five regression fixes to rc/1.103.0 by yuneng-berri · Pull Request #43331 · BerriAI/litellm

yuneng-berri · 2026-09-26T18:44:43Z

TLDR

Problem this solves:

Five regressions present in v1.103.0-rc.1 would ship into the next stable
Fixes already merged on main but were missing from rc/1.103.0

How it solves it:

Cherry-picks five fixes onto rc/1.103.0, one commit each
fix: apply configured cache_control_injection_points beside client cache_control marks #41956: configured cache_control_injection_points apply beside client marks
fix(token_counter): count replayed redacted_thinking blocks so prompt_caching keeps pinning #42069: replayed redacted_thinking blocks no longer break prompt_caching pinning
fix(mcp): return camelCase tool keys from /v1/mcp/tools after the SDK 2 upgrade #42352: /v1/mcp/tools returns camelCase inputSchema and outputSchema again
fix(rust_bridge): keep Messages, token counter and tokenizer routes on Python #42517: with LITELLM_RUST=1, /v1/messages stays on Python
fix(proxy): stop leaking periodic tasks on every DB config reload #42784: DB config reloads stop leaking background tasks

User Flow

Before: a customer on v1.103.0-rc.1 hits one of five breaks that worked on an earlier release

An operator with cache_control_injection_points on an Anthropic deployment sends POST http://localhost:4000/v1/chat/completions with a client-marked system block, and cached_tokens stays at the system block only on every turn
A Claude Code user on a group with prompt_caching replays a redacted_thinking block to POST http://localhost:4000/v1/messages, and each turn lands on a different deployment, paying a fresh cache write
A script calls GET http://localhost:4000/v1/mcp/tools and reads inputSchema, which is missing because the route now returns input_schema
A Claude Code user on a proxy with LITELLM_RUST=1 sends a compaction edit to POST http://localhost:4000/v1/messages and gets 400 context_management: Extra inputs are not permitted
An admin with DB-stored config and Slack alerting watches the proxy's background task count grow by two every 30 seconds

After: the same five flows behave the way they did before the regression

The configured tail checkpoint lands next to the client's mark, so later turns read the conversation from cache
Every replayed turn stays on the deployment that served the first one
GET http://localhost:4000/v1/mcp/tools returns inputSchema and outputSchema
The compaction request returns 200
The task count stays flat across reloads

Affected release

Regressions in v1.103.0-rc.1: #42352 and #42517 since v1.103.0-rc.1, #42069 and #42784 since v1.102.0, #41956 since v1.95.0

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Every touched test file passes on this branch (1873 tests). Each pick's own tests were also run against rc/1.103.0 with its source change removed, and they fail there: #41956 48 failures, #42069 2, #42352 1, #42517 189, #42784 2

Adaptations from the main versions:

fix(token_counter): count replayed redacted_thinking blocks so prompt_caching keeps pinning #42069: test-only conflict, the new test was placed beside rc's existing helper
fix(rust_bridge): keep Messages, token counter and tokenizer routes on Python #42517: rc's route catalog has no token counter or tokenizer routes, so only the Messages rule moves to Python
fix(proxy): stop leaking periodic tasks on every DB config reload #42784: test-only conflict, only this PR's three new tests were carried over
fix(mcp): return camelCase tool keys from /v1/mcp/tools after the SDK 2 upgrade #42352: the one new patch() in its test carries a test-quality-ok reason, since rc/1.103.0 is already over its TQ008 budget

Type

🐛 Bug Fix

Caveats (if any)

Low

Live proxy proof on rc/1.103.0 is not captured yet
No dashboard source changed, so the UI bundle rebuild in chore: rebuild Admin UI bundle for rc/1.103.0 #43328 stays valid

…che_control marks (#41956) (cherry picked from commit d8267d5)

…_caching keeps pinning (#42069) (cherry picked from commit 0e7cf51)

… 2 upgrade (#42352) (cherry picked from commit 5eb4e30)

…th is ready (#42517) rc/1.103.0 has no token counter or tokenizer routes in the catalog, so only the Messages rule moves to PYTHON_ONLY (cherry picked from commit 85ed18e)

…2784) (cherry picked from commit fc87a06)

greptile-apps · 2026-09-26T18:49:34Z

Confidence Score: 3/5

[Medium risk] Backports five regression fixes across integrations and routing.

The PR is not ready to merge because router updates can lose queued RPM increments or leave routing arguments stale.

Findings

Queued increments are lost ▶
Mutated routing arguments stay stale ▶

Summary

This PR backports fixes for cache-control injection, replayed redacted-thinking token counting, MCP tool field aliases, Messages dispatch, and background-task growth. The router retirement and settings changes need correction before merging.

_{Reviews (1) · Last reviewed commit: "fix(proxy): stop leaking periodic tasks ..."}

greptile-apps · 2026-09-26T18:49:37Z

+        try:
+            loop: Final = asyncio.get_running_loop()
+        except RuntimeError:
+            return
+        loop.create_task(self._push_in_memory_increments_to_redis())


Queued increments are lost

If a synchronous Router.update_settings() call replaces a usage-based selector while it has Redis increments queued, retire() cancels its sync task and returns because there is no running event loop. Those increments are never flushed, so later routing decisions can use understated RPM usage.

greptile-apps · 2026-09-26T18:49:37Z

                            rebuild_routing_groups = True
                    elif var == "routing_strategy_args":
-                        routing_args_updated = True
+                        routing_args_updated = value != self.routing_strategy_args


Mutated routing arguments stay stale

If a caller changes the dictionary previously passed as routing_strategy_args and passes that same dictionary to update_settings(), this comparison checks the dictionary against itself. It skips rebuilding the selector, so the router reports the new arguments while routing still uses the old values.

CLAassistant · 2026-09-26T19:30:23Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mateo-berri
❌ devin-ai-integration[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

mateo-berri and others added 5 commits September 26, 2026 11:39

fix: apply configured cache_control_injection_points beside client ca…

bda27a8

…che_control marks (#41956) (cherry picked from commit d8267d5)

fix(token_counter): count replayed redacted_thinking blocks so prompt…

4d9db69

…_caching keeps pinning (#42069) (cherry picked from commit 0e7cf51)

fix(mcp): return camelCase tool keys from /v1/mcp/tools after the SDK…

3d097d7

… 2 upgrade (#42352) (cherry picked from commit 5eb4e30)

fix(rust_bridge): keep the Messages route on Python until the Rust pa…

ab38d71

…th is ready (#42517) rc/1.103.0 has no token counter or tokenizer routes in the catalog, so only the Messages rule moves to PYTHON_ONLY (cherry picked from commit 85ed18e)

fix(proxy): stop leaking periodic tasks on every DB config reload (#4…

528cce8

…2784) (cherry picked from commit fc87a06)

yuneng-berri merged commit 6397b36 into rc/1.103.0 Sep 26, 2026
9 of 10 checks passed

yuneng-berri deleted the litellm_backport_regression_fixes_rc_1_103_0 branch September 26, 2026 18:47

greptile-apps Bot reviewed Sep 26, 2026

View reviewed changes

yuneng-berri mentioned this pull request Sep 26, 2026

fix(caching): stand default cache points down when extra_body hides a direct client mark #43342

Merged

3 of 5 tasks

devin-ai-integration Bot mentioned this pull request Sep 28, 2026

fix(caching): count tool_call cache_control marks in the injection census #43556

Merged

8 of 16 tasks

Navigation Menu

Repository navigation

fix: backport five regression fixes to rc/1.103.0 - #43331

fix: backport five regression fixes to rc/1.103.0#43331
yuneng-berri merged 5 commits into
rc/1.103.0from
litellm_backport_regression_fixes_rc_1_103_0

yuneng-berri commented Sep 26, 2026

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 26, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Sep 26, 2026

Uh oh!

greptile-apps Bot Sep 26, 2026

Uh oh!

CLAassistant commented Sep 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

yuneng-berri commented Sep 26, 2026

TLDR

User Flow

Affected release

Pre-Submission checklist

Type

Caveats (if any)

Low

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 3/5

Findings

Summary

Uh oh!

greptile-apps Bot Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 26, 2026

Choose a reason for hiding this comment

Uh oh!

CLAassistant commented Sep 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

greptile-apps Bot commented Sep 26, 2026 •

edited

Loading