fix(proxy): keep runtime-injected deployments the config reconcile cannot see

[Pawan-Shahane]

TLDR

Problem this solves:

• Models added to a running proxy vanish after 30 seconds
• Nothing is logged when they go
• Plugins and callbacks are unusable with store_model_in_db on

How it solves it:

• A deployment can name its owner in model_info.managed_by
• The periodic config reload leaves owned deployments alone
• Anything naming no owner is still removed, exactly as before

User Flow

Before: an operator whose plugin adds a model to the running proxy watches it disappear half a minute later

1. They start the proxy with store_model_in_db: true and a plugin of their own that adds a deployment named injected-by-plugin once the proxy is up
2. They send GET https://litellm-domain/model/info and see both their config.yaml model and injected-by-plugin
3. They wait out one config reload interval and send the same GET again. Only the config.yaml model comes back
4. They send POST https://litellm-domain/v1/chat/completions with "model": "injected-by-plugin" and get HTTP 400, Invalid model name passed in model=injected-by-plugin
5. They search the proxy logs for that model name at every log level and find nothing, so there is no way to tell what removed it

After: the same model stays put and keeps serving traffic

1. They start the proxy the same way, with the plugin now tagging its deployment managed_by: <their plugin's name> in model_info
2. They send GET https://litellm-domain/model/info and see both models
3. They wait out the same interval and send the same GET again. Both models still come back
4. They send POST https://litellm-domain/v1/chat/completions with "model": "injected-by-plugin" and get HTTP 200 with a real completion
5. With debug logging on, each reload prints one line naming the model it kept and the owner that claimed it

Relevant issues

Fixes #42211

Affected release

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both sides run the same proxy, the same plugin and the same commands against a real Postgres, and the last step is a real billed OpenRouter call.

qa_config.yaml, with the reload interval shortened to 10s so the wait is short:

model_list:
  - model_name: config-model
    litellm_params:
      model: openrouter/mistralai/mistral-nemo
      api_key: os.environ/OPENROUTER_API_KEY
    model_info:
      id: config-model-id

general_settings:
  master_key: sk-litellm-dev-1234
  store_model_in_db: true
  proxy_config_reload_interval_seconds: 10

litellm_settings:
  callbacks: injector.proxy_handler_instance

injector.py next to it, standing in for any plugin that adds a deployment to a running proxy:

import asyncio
import os

from litellm.integrations.custom_logger import CustomLogger
from litellm.types.router import Deployment, LiteLLM_Params


class RuntimeModelInjector(CustomLogger):
    def __init__(self) -> None:
        super().__init__()
        asyncio.get_event_loop().create_task(self._inject())

    async def _inject(self) -> None:
        from litellm.proxy import proxy_server

        while proxy_server.llm_router is None:
            await asyncio.sleep(0.5)
        await asyncio.sleep(5)

        proxy_server.llm_router.upsert_deployment(
            Deployment(
                model_name="injected-by-plugin",
                litellm_params=LiteLLM_Params(
                    model="openrouter/mistralai/mistral-nemo",
                    api_key=os.environ["OPENROUTER_API_KEY"],
                ),
                model_info={
                    "id": "injected-by-plugin-id",
                    "managed_by": "runtime-model-injector",
                },
            )
        )
        print("INJECTOR: injected-by-plugin is now in the router", flush=True)


proxy_handler_instance = RuntimeModelInjector()

managed_by is accepted on both commits, so the plugin is byte for byte identical on the two sides. On the Before commit it is simply an extra field nothing reads.

Start the proxy:

export DATABASE_URL=postgresql://litellm:litellm@localhost:5433/litellm
python litellm/proxy/proxy_cli.py --config qa_config.yaml --port 4055 2>&1 | tee proxy.log

Both sides then run these three, waiting for the plugin's line in the log first:

curl -s http://localhost:4055/model/info -H 'Authorization: Bearer sk-litellm-dev-1234' | jq -c '[.data[].model_info.id] | sort'
sleep 25
curl -s http://localhost:4055/model/info -H 'Authorization: Bearer sk-litellm-dev-1234' | jq -c '[.data[].model_info.id] | sort'
curl -s -w 'HTTP %{http_code}\n' -X POST http://localhost:4055/v1/chat/completions -H 'Authorization: Bearer sk-litellm-dev-1234' -H 'Content-Type: application/json' -d '{"model":"injected-by-plugin","messages":[{"role":"user","content":"say hi"}],"max_tokens":10}'

Before (aa0ffee)

1. Right after the plugin added its deployment, both models are served:

['config-model-id', 'injected-by-plugin-id']

2. After waiting past one 10s reload tick, the plugin's model is gone:

['config-model-id']

3. Calling it now fails:

HTTP 400
{"error":{"message":"/chat/completions: Invalid model name passed in model=injected-by-plugin. Call `/v1/models` to view available models for your key.","type":"invalid_request_error","param":null,"code":"400"}}

4. Nothing in the log accounts for it, at any level:

$ grep -c "injected-by-plugin-id" proxy.log
0

After (0bb58ee)

1. Right after the plugin added its deployment, both models are served:

['config-model-id', 'injected-by-plugin-id']

2. After the same wait, both are still there:

['config-model-id', 'injected-by-plugin-id']

3. The call goes through to OpenRouter and is really billed:

HTTP 200
{"id":"gen-1789981794-dU0n2SWrAyOs7qzAJtfv","created":1789981794,"model":"injected-by-plugin","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"Hello! How can I assist you today?","role":"assistant"}}],"usage":{"completion_tokens":10,...

4. Re-running the same proxy with --detailed_debug, each reload says what it kept and who claimed it:

$ grep -o "Config reconcile kept.*" proxy.log
Config reconcile kept 1 deployment(s) absent from db + config because they declare an owner: {'injected-by-plugin-id': 'runtime-model-injector'}

Same proxy, same plugin, same commands on both sides, so the only thing that changed is whether the model survives the reload

Type

🐛 Bug Fix

Caveats (if any)

Medium

• Opt in: a deployment that sets no owner is still evicted
  • Existing injectors keep breaking until they tag their deployments
  • Protecting every untagged orphan instead would mean never evicting a stale model again, including one another pod deleted from the db
• Owned deployments are never reaped by the proxy
  • Whoever injected one is on the hook for removing it
  • An owner that leaks deployments leaks them until restart

Low

• The kept-deployment line logs at info, so default WARNING logging hides it
  • It is the steady state once you opt in, and the reload runs every 30s by default, so a louder level would be constant noise
  • The capture above turns on --detailed_debug to show it
• A deployment the reload cannot read counts as unowned and is evicted, same as before this change
• combined_id_list picked up a list[str] annotation, which drops 3 basedpyright findings in the file

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

🤖 Generated with Claude Code

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing Pawan-Shahane:litellm_preserve_runtime_injected_deployments (0bb58ee) with main (1cac8bd)}

[codecov]

Codecov Report

❌ Patch coverage is 75.00000% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/proxy_server.py	73.91%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!

[chilkotiKartik]

LGTM! Reviewed fix(proxy): keep runtime-injected deployments the config reconcile cannot see. Preserving runtime-injected router deployments during background config reconciliation prevents race conditions where dynamic endpoints get unintentionally pruned before synchronization completes.

[chilkotiKartik]

PR Review: Gemini Streaming Grounding Metadata Normalization

Problem & Solution:

When streaming from Gemini 2.5 / Vertex AI with search grounding enabled, the final response chunk emits groundingMetadata structures with variable field names (webSearchQueries, groundingChunks). This PR normalizes the metadata into the standard OpenAI-compatible choice.delta.extra_content payload.

Key Verification:

1. Streaming Delta Aggregation:
   • Verified that intermediate chunk iterators pass through text deltas without stalling while buffering the terminal grounding metadata dictionary.
2. Schema Uniformity:
   • search_entry_point and grounding_supports conform to standard downstream RAG consumption structures.

LGTM! 🚀