Repository navigation
fix(proxy): read through to the DB on registry misses so just-created models, guardrails, and agents resolve on sibling replicas - #36263
Merged
mateo-berri merged 8 commits intoAug 19, 2026
Conversation
… models, guardrails, and agents resolve on sibling replicas
Contributor
Greptile SummaryThis PR adds bounded database read-through recovery for model, guardrail, and agent registry misses across replicas while hardening concurrent registry reconciliation.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| litellm/proxy/common_utils/registry_read_through.py | Introduces bounded, negative-cached, per-registry database read-through with reconciliation locking and supported-object gating. |
| litellm/proxy/route_llm_request.py | Retries model routing once after a recoverable local-registry miss while exempting non-model A2A failures. |
| litellm/proxy/proxy_server.py | Extracts database-object gating and serializes guardrail and agent snapshot reconciliation. |
| litellm/proxy/management_endpoints/model_access_group_management_endpoints.py | Hydrates missing models before validation and evaluates reload degradation against the reload’s atomic outcome. |
| litellm/router.py | Restores a previously serving deployment when a replacement fails under ignored-invalid-deployment handling. |
| litellm/proxy/agent_endpoints/a2a_routing.py | Adds agent registry read-through and prevents unresolved agent failures from consuming the model recovery budget. |
| litellm/proxy/guardrails/guardrail_endpoints.py | Resolves guardrails through the bounded read-through path before returning not found. |
| tests/test_litellm/proxy/common_utils/test_registry_read_through.py | Covers negative caching, budget behavior, registry recovery, object gating, status filtering, and reconciliation locking. |
Reviews (8): Last reviewed commit: "fix: gate guardrail read-through to acti..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
Contributor
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 2 · PR risk: 0/10 |
…itellm_replica_registry_read_through # Conflicts: # litellm/proxy/management_endpoints/model_access_group_management_endpoints.py # ruff.toml # tests/test_litellm/proxy/test_route_llm_request.py
Resync registry misses with single-row DB fetches (guardrail by unique name, agent by unique id or name, model by name then id) instead of full-table loads, and bound them with a global budget of 20 resyncs per 5s window per registry that fails closed without negative-caching the key. Access group create/update now trust the reconcile outcome snapshot captured under the reload lock instead of a post-lock router read, so a concurrent reconcile can no longer surface a false degraded-serving 500. Router.upsert_deployment restores the previously served deployment when the replacement add fails under ignore_invalid_deployments, so a bad update no longer silently drops a healthy deployment from serving.
Contributor
Author
Contributor
Author
|
bugbot run |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for all 3 issues found in the latest run.
- ✅ Fixed: Read-through skips reconcile lock
- Both model and agent read-through resyncs now acquire MODEL_RECONCILE_LOCK across the DB fetch and registry write, serializing them against the periodic reload's reset/evict path.
- ✅ Fixed: A2A misses spend model budget
- route_request now re-raises ProxyModelNotFoundError immediately for a2a/ models instead of consuming the model registry read-through budget on agent-not-found failures.
- ✅ Fixed: Read-through ignores object-type gates
- _db_backed_registries_enabled now takes an object_type and consults proxy_config._should_load_db_object so read-through skips types excluded from supported_db_objects.
Or push these changes by commenting:
@cursor push 55aa29f53d
Preview (55aa29f53d)
diff --git a/litellm/proxy/common_utils/registry_read_through.py b/litellm/proxy/common_utils/registry_read_through.py
--- a/litellm/proxy/common_utils/registry_read_through.py
+++ b/litellm/proxy/common_utils/registry_read_through.py
@@ -95,17 +95,20 @@
return found
-def _db_backed_registries_enabled() -> bool:
+def _db_backed_registries_enabled(object_type: str) -> bool:
from litellm.proxy import proxy_server
- return proxy_server.prisma_client is not None and proxy_server.store_model_in_db is True
+ if proxy_server.prisma_client is None or proxy_server.store_model_in_db is not True:
+ return False
+ return proxy_server.proxy_config._should_load_db_object(object_type=object_type)
async def _resync_model_deployments(model_name: str) -> bool:
from litellm.proxy import proxy_server
+ from litellm.proxy.proxy_server import MODEL_RECONCILE_LOCK
from litellm.repositories.model_repository import ModelRepository
- if not _db_backed_registries_enabled():
+ if not _db_backed_registries_enabled(object_type="models"):
return False
prisma_client: Final = proxy_server.prisma_client
assert prisma_client is not None
@@ -120,8 +123,11 @@
prisma_client=prisma_client, proxy_logging_obj=proxy_server.proxy_logging_obj
)
return proxy_server.llm_router is not None
- proxy_server.proxy_config._add_deployment(db_models=rows)
- proxy_server.llm_model_list = proxy_server.llm_router.get_model_list()
+ async with MODEL_RECONCILE_LOCK:
+ if proxy_server.llm_router is None:
+ return False
+ proxy_server.proxy_config._add_deployment(db_models=rows)
+ proxy_server.llm_model_list = proxy_server.llm_router.get_model_list()
return True
@@ -132,7 +138,7 @@
GuardrailRegistry,
)
- if not _db_backed_registries_enabled():
+ if not _db_backed_registries_enabled(object_type="guardrails"):
return False
prisma_client: Final = proxy_server.prisma_client
assert prisma_client is not None
@@ -151,9 +157,10 @@
agents_table,
global_agent_registry,
)
+ from litellm.proxy.proxy_server import MODEL_RECONCILE_LOCK
from litellm.types.agents import AgentResponse
- if not _db_backed_registries_enabled():
+ if not _db_backed_registries_enabled(object_type="agents"):
return False
if _agent_from_registry(agent_id_or_name) is not None:
return True
@@ -163,12 +170,15 @@
id_filter: Final[LiteLLM_AgentsTableWhereUniqueInput] = {"agent_id": agent_id_or_name}
name_filter: Final[LiteLLM_AgentsTableWhereUniqueInput] = {"agent_name": agent_id_or_name}
include_permission: Final[LiteLLM_AgentsTableInclude] = {"object_permission": True}
- row: Final = await table.find_unique(where=id_filter, include=include_permission) or await table.find_unique(
- where=name_filter, include=include_permission
- )
- if row is None:
- return False
- global_agent_registry.register_agent(agent_config=AgentResponse.model_validate(row.model_dump()))
+ async with MODEL_RECONCILE_LOCK:
+ if _agent_from_registry(agent_id_or_name) is not None:
+ return True
+ row: Final = await table.find_unique(where=id_filter, include=include_permission) or await table.find_unique(
+ where=name_filter, include=include_permission
+ )
+ if row is None:
+ return False
+ global_agent_registry.register_agent(agent_config=AgentResponse.model_validate(row.model_dump()))
return True
diff --git a/litellm/proxy/route_llm_request.py b/litellm/proxy/route_llm_request.py
--- a/litellm/proxy/route_llm_request.py
+++ b/litellm/proxy/route_llm_request.py
@@ -441,6 +441,11 @@
requested_model: Final = data.get("model", "")
if not isinstance(requested_model, str) or not requested_model:
raise
+ # A2A routing already runs its own agent read-through; retrying against the
+ # model registry would burn its budget on `a2a/<name>` keys and can fail-closed
+ # a sibling-created model for the rest of the window.
+ if _is_a2a_agent_model(requested_model):
+ raise
from litellm.proxy import proxy_server
from litellm.proxy.common_utils.registry_read_through import (
model_registry_read_through,You can send follow-ups to the cloud agent here.
The model resync now mutates the router under MODEL_RECONCILE_LOCK, and the agent resync shares the new AGENT_RECONCILE_LOCK with the periodic agent reload, so a reconcile built from a pre-write DB snapshot can no longer evict or duplicate what a read-through just registered. Every resync checks should_load_db_object for its object type, keeping read-through consistent with what the replica is configured to load, and the a2a raise sites tag ProxyModelNotFoundError as non-retryable so an agent miss no longer burns the model resync budget.
Contributor
Author
Contributor
Author
|
bugbot run |
Contributor
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit afeed48. Configure here.
…/litellm into litellm_replica_registry_read_through # Conflicts: # tests/test_litellm/proxy/test_proxy_server.py
Contributor
Author
Contributor
Author
|
bugbot run |
…outer coverage gate
Contributor
Author
Contributor
Author
|
bugbot run |
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Guardrail hydrate races reload eviction
- Added a GUARDRAIL_RECONCILE_LOCK held across both the periodic reload's snapshot-and-reconcile and the read-through's DB fetch-plus-sync, matching the model and agent reconcile-lock pattern so a just-hydrated sibling-replica guardrail can no longer be evicted.
Or push these changes by commenting:
@cursor push fc18a726f3
Preview (fc18a726f3)
diff --git a/litellm/proxy/common_utils/registry_read_through.py b/litellm/proxy/common_utils/registry_read_through.py
--- a/litellm/proxy/common_utils/registry_read_through.py
+++ b/litellm/proxy/common_utils/registry_read_through.py
@@ -132,21 +132,27 @@
async def _resync_guardrails(guardrail_name: str) -> bool:
from litellm.proxy import proxy_server
from litellm.proxy.guardrails.guardrail_registry import (
+ GUARDRAIL_RECONCILE_LOCK,
IN_MEMORY_GUARDRAIL_HANDLER,
GuardrailRegistry,
)
if not _db_backed_registries_enabled("guardrails"):
return False
+ if _initialized_guardrail(guardrail_name) is not None:
+ return True
prisma_client: Final = proxy_server.prisma_client
assert prisma_client is not None
- row: Final = await GuardrailRegistry().get_guardrail_by_name_from_db(
- guardrail_name=guardrail_name, prisma_client=prisma_client
- )
- if row is None:
- return False
- IN_MEMORY_GUARDRAIL_HANDLER.sync_guardrail_from_db(guardrail=row)
- return _initialized_guardrail(guardrail_name) is not None
+ async with GUARDRAIL_RECONCILE_LOCK:
+ if _initialized_guardrail(guardrail_name) is not None:
+ return True
+ row: Final = await GuardrailRegistry().get_guardrail_by_name_from_db(
+ guardrail_name=guardrail_name, prisma_client=prisma_client
+ )
+ if row is None:
+ return False
+ IN_MEMORY_GUARDRAIL_HANDLER.sync_guardrail_from_db(guardrail=row)
+ return _initialized_guardrail(guardrail_name) is not None
async def _resync_agents(agent_id_or_name: str) -> bool:
diff --git a/litellm/proxy/guardrails/guardrail_registry.py b/litellm/proxy/guardrails/guardrail_registry.py
--- a/litellm/proxy/guardrails/guardrail_registry.py
+++ b/litellm/proxy/guardrails/guardrail_registry.py
@@ -1,5 +1,6 @@
# litellm/proxy/guardrails/guardrail_registry.py
+import asyncio
import importlib
import os
from collections.abc import Callable, Iterator, Mapping
@@ -813,4 +814,5 @@
# In Memory Guardrail Handler for LiteLLM Proxy
########################################################
IN_MEMORY_GUARDRAIL_HANDLER: Final = InMemoryGuardrailHandler()
+GUARDRAIL_RECONCILE_LOCK: Final = asyncio.Lock()
########################################################
diff --git a/litellm/proxy/proxy_server.py b/litellm/proxy/proxy_server.py
--- a/litellm/proxy/proxy_server.py
+++ b/litellm/proxy/proxy_server.py
@@ -7090,38 +7090,40 @@
async def _init_guardrails_in_db(self, prisma_client: PrismaClient):
from litellm.proxy.guardrails.guardrail_registry import (
+ GUARDRAIL_RECONCILE_LOCK,
IN_MEMORY_GUARDRAIL_HANDLER,
Guardrail,
GuardrailRegistry,
)
try:
- guardrails_in_db: Final[list[Guardrail]] = await GuardrailRegistry.get_all_guardrails_from_db(
- prisma_client=prisma_client
- )
- verbose_proxy_logger.debug("guardrails from the DB %s", str(guardrails_in_db))
- db_guardrail_ids: Final[set] = set()
- for guardrail in guardrails_in_db:
- guardrail_id = guardrail.get("guardrail_id")
- if guardrail_id:
- db_guardrail_ids.add(guardrail_id)
- try:
- IN_MEMORY_GUARDRAIL_HANDLER.sync_guardrail_from_db(
- guardrail=cast(Guardrail, guardrail),
- )
- except Exception as e: # noqa: BLE001 # one unloadable row must not stop the remaining guardrails
- verbose_proxy_logger.error(
- "litellm.proxy.proxy_server.py::ProxyConfig:_init_guardrails_in_db - "
- "skipping guardrail '%s' (ID: %s): %s: %s",
- guardrail.get("guardrail_name"),
- guardrail_id,
- type(e).__name__,
- e,
- )
+ async with GUARDRAIL_RECONCILE_LOCK:
+ guardrails_in_db: Final[list[Guardrail]] = await GuardrailRegistry.get_all_guardrails_from_db(
+ prisma_client=prisma_client
+ )
+ verbose_proxy_logger.debug("guardrails from the DB %s", str(guardrails_in_db))
+ db_guardrail_ids: Final[set] = set()
+ for guardrail in guardrails_in_db:
+ guardrail_id = guardrail.get("guardrail_id")
+ if guardrail_id:
+ db_guardrail_ids.add(guardrail_id)
+ try:
+ IN_MEMORY_GUARDRAIL_HANDLER.sync_guardrail_from_db(
+ guardrail=cast(Guardrail, guardrail),
+ )
+ except Exception as e: # noqa: BLE001 # one unloadable row must not stop the remaining guardrails
+ verbose_proxy_logger.error(
+ "litellm.proxy.proxy_server.py::ProxyConfig:_init_guardrails_in_db - "
+ "skipping guardrail '%s' (ID: %s): %s: %s",
+ guardrail.get("guardrail_name"),
+ guardrail_id,
+ type(e).__name__,
+ e,
+ )
- # Drop in-memory DB-backed entries whose row was deleted on another
- # pod. Config-loaded entries are never touched.
- IN_MEMORY_GUARDRAIL_HANDLER.reconcile_db_guardrails(db_guardrail_ids=db_guardrail_ids)
+ # Drop in-memory DB-backed entries whose row was deleted on another
+ # pod. Config-loaded entries are never touched.
+ IN_MEMORY_GUARDRAIL_HANDLER.reconcile_db_guardrails(db_guardrail_ids=db_guardrail_ids)
except Exception as e:
verbose_proxy_logger.exception("litellm.proxy.proxy_server.py::ProxyConfig:_init_guardrails_in_db - %s", e)
diff --git a/tests/test_litellm/proxy/common_utils/test_registry_read_through.py b/tests/test_litellm/proxy/common_utils/test_registry_read_through.py
--- a/tests/test_litellm/proxy/common_utils/test_registry_read_through.py
+++ b/tests/test_litellm/proxy/common_utils/test_registry_read_through.py
@@ -422,3 +422,37 @@
assert await resync_task is True
assert len(clean_agent_registry.agent_list) == 1
+
+
+@pytest.mark.asyncio
+async def test_resync_guardrails_waits_for_guardrail_reload_and_skips_duplicate_sync(monkeypatch):
+ from unittest.mock import AsyncMock, MagicMock
+
+ import litellm.proxy.proxy_server as proxy_server
+ from litellm.proxy.common_utils.registry_read_through import _resync_guardrails
+ from litellm.proxy.guardrails.guardrail_registry import (
+ GUARDRAIL_RECONCILE_LOCK,
+ IN_MEMORY_GUARDRAIL_HANDLER,
+ )
+
+ guardrail_id: Final = "reload-race-guardrail-id"
+ guardrail_name: Final = "reload-race-guardrail"
+ prisma_client: Final = MagicMock()
+ prisma_client.db.litellm_guardrailstable.find_unique = AsyncMock(
+ side_effect=AssertionError("db hit while the guardrail reload held the reconcile lock")
+ )
+ monkeypatch.setattr(proxy_server, "prisma_client", prisma_client)
+ monkeypatch.setattr(proxy_server, "store_model_in_db", True)
+
+ try:
+ async with GUARDRAIL_RECONCILE_LOCK:
+ resync_task: Final = asyncio.ensure_future(_resync_guardrails(guardrail_name))
+ await asyncio.sleep(0.05)
+ assert not resync_task.done()
+ IN_MEMORY_GUARDRAIL_HANDLER.sync_guardrail_from_db(
+ guardrail=dict(FakeGuardrailRow(guardrail_id, guardrail_name))
+ )
+
+ assert await resync_task is True
+ finally:
+ IN_MEMORY_GUARDRAIL_HANDLER.delete_in_memory_guardrail(guardrail_id)You can send follow-ups to the cloud agent here.
… the reload reconcile
Contributor
Author
|
bugbot run |
Contributor
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit f8a23aa. Configure here.
Contributor
Author
mateo-berri
enabled auto-merge
August 19, 2026 09:38
Contributor
Author
tin-berri
approved these changes
Aug 19, 2026
tin-berri
left a comment
tin-berri
left a comment
Contributor
There was a problem hiding this comment.
LGTM — this is a well-tested fix for a real multi-replica consistency problem, and I verified the core mechanisms independently:
- Read-through + budget/negative-cache:
RegistryReadThroughinlitellm/proxy/common_utils/registry_read_through.pyis instantiated separately per object type (model/guardrail/agent), giving genuinely independent 20-per-5s budgets — confirmed by the A2A-miss-doesn't-starve-model-resync test. Budget exhaustion correctly skips negative-caching a throttled key rather than poisoning it as a false miss. - Access-group snapshot: verdicts now read
reload_outcome.still_desired/live_aftercaptured atomically under the reload lock, rather than a fresh post-lock router read — directly demonstrated by a test simulating a concurrently-wiped router. - Reconcile locks: for guardrails/agents, both the full periodic reload and the miss-triggered resync take the same lock, verified with an interleaved-asyncio race test. For models,
MODEL_RECONCILE_LOCKisn't defined in this diff, but I confirmed it's pre-existing shared infrastructure inproxy_server.pyalready used to serialize model delete/reconcile — this PR's resync path correctly reuses it rather than introducing a parallel lock. - Fail-safe refresh:
_restore_deployment_after_failed_upsertonly fires on an upsert failure for an existing model_id; legitimate deletions go through a separate path, so a real delete won't get incorrectly kept alive. - Scope: the large
route_llm_request.pydiff is a mechanicalLiteral→RouteTypeextraction to support the new retry wrapper's type signature, not unrelated drift.
One acknowledged, bounded tradeoff worth naming rather than a bug: the 2s negative cache means a miss checked just before a real create lands on another replica can still defeat the read-through fix for up to that window — not airtight for the tightest possible race, but a reasonable DB-protection tradeoff.
3 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
How it solves it:
supported_db_objectsand take the reload's reconcile locksUser Flow
Before: an admin running two gateway replicas behind one URL creates an object, and the very next call that uses it fails whenever the load balancer picks the other replica; access group writes racing a reload can also return a false 500 and knock a serving model off a replica
POST https://litellm-domain/model/newwith the admin key and{"model_name": "team-gpt", "litellm_params": {"model": "openai/gpt-4o-mini", ...}}and get 200 with the new deployment idPOST https://litellm-domain/v1/chat/completionswith{"model": "team-gpt", ...}and, when the other replica answers, get400 "Invalid model name passed in model=team-gpt. Call /v1/models to view available models for your key."(/v1/responsesand/v1/messagesfail the same way)POST https://litellm-domain/access_group/newwith{"access_group": "prod-models", "model_names": ["team-gpt"]}and get400 {"error": "Model(s) not found: team-gpt"}POST https://litellm-domain/guardrailsregisteringpii-filterand get 200 with a guardrail idPOST https://litellm-domain/guardrails/apply_guardrailwith{"guardrail_name": "pii-filter", "text": "..."}and get404 "Guardrail 'pii-filter' not found. Please ensure the guardrail is configured in your LiteLLM proxy."POST https://litellm-domain/v1/agentsregistering an agent and get 200 with anagent_idlike6d27a7e1-0aa6-...POST https://litellm-domain/a2a/{agent_id}with JSON-RPCmessage/sendand get{"error": {"code": -32000, "message": "Agent '6d27a7e1-...' not found"}}POST https://litellm-domain/access_group/newnaming a model the answering replica already serves and occasionally get500 {"error": "Access group 'prod-models' was saved to the database, but model id(s) [...] that this pod was serving are no longer live after the reload it triggered. ..."}even though the write saved, and the named model can then stop serving on that replica (chat calls on it 400) until the next background reloadAfter: the same admin runs the same sequence and every immediate follow-up call succeeds no matter which replica answers, and access group writes racing a reload stay 200 with no serving gap
POST https://litellm-domain/model/newwith the admin key and{"model_name": "team-gpt", "litellm_params": {"model": "openai/gpt-4o-mini", ...}}and get 200 with the new deployment idPOST https://litellm-domain/v1/chat/completionswith{"model": "team-gpt", ...}and get 200 with a real completion from the new deployment, on whichever replica the call lands (/v1/responsesand/v1/messageslikewise)POST https://litellm-domain/access_group/newwith{"access_group": "prod-models", "model_names": ["team-gpt"]}and get200 {"access_group": "prod-models", ..., "models_updated": 1}POST https://litellm-domain/guardrailsregisteringpii-filterand get 200 with a guardrail idPOST https://litellm-domain/guardrails/apply_guardrailand get200 {"response_text": "..."}POST https://litellm-domain/v1/agentsregistering an agent and get 200 with anagent_idPOST https://litellm-domain/a2a/{agent_id}with JSON-RPCmessage/sendand get a JSON-RPC result carrying the agent's real reply200 {..., "models_updated": 1}every time, and the models they name keep serving throughoutRelevant issues
Linear ticket
Resolves LIT-5791
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Setup: two proxies booted from the same checkout sharing one fresh Postgres (
store_model_in_db: true, no Redis), replica A and replica B on separate high ports. Every "create on A" is immediately followed by a "use on B" curl, which is exactly the cross-replica race from the 2-replica e2e suite. All LLM calls hit the real OpenAI API withgpt-4o-miniand cost real $Before (6d32d40)
model created on A, immediate /v1/chat/completions on B
curl -X POST $A/model/new -H "$K" -H "$J" -d '{"model_name": "lit5791-miss-2272", "litellm_params": {"model": "openai/gpt-4o-mini", "api_key": "os.environ/OPENAI_API_KEY"}}'returns 200 with"model_id": "a3d0f437-8e53-4bee-936b-0511f3c4732c"curl -X POST $B/chat/completions -H "$K" -H "$J" -d '{"model": "lit5791-miss-2272", "messages": [{"role": "user", "content": "say hi"}]}'returns HTTP 400:/chat/completions: Invalid model name passed in model=lit5791-miss-2272. Call /v1/models to view available models for your key.model created on A, immediate /v1/responses on B
curl -X POST $A/model/new ...(fresh namelit5791-before-responses-971) returns 200curl -X POST $B/v1/responses -H "$K" -H "$J" -d '{"model": "lit5791-before-responses-971", "input": "Reply with exactly: read-through works"}'returns HTTP 400:/responses: Invalid model name passed in model=lit5791-before-responses-971. ...model created on A, immediate /v1/messages on B
curl -X POST $A/model/new ...(fresh namelit5791-before-messages-591) returns 200curl -X POST $B/v1/messages -H "$K" -H "$J" -d '{"model": "lit5791-before-messages-591", "max_tokens": 32, "messages": [{"role": "user", "content": "Reply with exactly: read-through works"}]}'returns HTTP 400:anthropic_messages: Invalid model name passed in model=lit5791-before-messages-591. ...guardrail created on A, applied on B
curl -X POST $A/guardrails -H "$K" -H "$J" -d '{"guardrail": {"guardrail_name": "lit5791-guard-7957", "litellm_params": {"guardrail": "litellm_content_filter", "mode": "pre_call", "default_on": false, "blocked_words": [{"keyword": "zebra", "action": "BLOCK"}]}}}'returns 200curl -X POST $B/guardrails/apply_guardrail -H "$K" -H "$J" -d '{"guardrail_name": "lit5791-guard-7957", "text": "hello world"}'returns HTTP 404:Guardrail 'lit5791-guard-7957' not found. Please ensure the guardrail is configured in your LiteLLM proxy.A2A agent created on A, message/send on B
curl -X POST $A/v1/agents -H "$K" -H "$J" -d '{"agent_name": "lit5791-agent-13316", "agent_card_params": {...}, "litellm_params": {"custom_llm_provider": "openai", "model": "gpt-4o-mini"}}'returns 200 with"agent_id": "ff8a3a61-4c0c-4acd-b904-4928768e5dd1"curl -X POST $B/a2a/lit5791-agent-13316 -H "$K" -H "$J" -d '{"jsonrpc": "2.0", "id": "1", "method": "message/send", "params": {"message": {"role": "user", "parts": [{"kind": "text", "text": "Reply with exactly: a2a works"}], "messageId": "m1"}}}'returns HTTP 404:{"jsonrpc":"2.0","id":"1","error":{"code":-32000,"message":"Agent 'lit5791-agent-13316' not found"}}model created on A, access group naming it on B
curl -X POST $A/model/new ...(fresh namelit5791-xag-14571) returns 200curl -X POST $B/access_group/new -H "$K" -H "$J" -d '{"access_group": "lit5791-xg", "model_names": ["lit5791-xag-14571"]}'returns HTTP 400:{"detail":{"error":"Model(s) not found: lit5791-xag-14571"}}unknown model on B keeps failing with 400
curl -X POST $B/chat/completions -H "$K" -H "$J" -d '{"model": "lit5791-never-created", "messages": [{"role": "user", "content": "hi"}]}'returns HTTP 400:/chat/completions: Invalid model name passed in model=lit5791-never-created. ...concurrent access-group writes during reload churn
curl -X POST $A/access_group/new ...writes, each naming its own modelAccess group '...' was saved to the database, but model id(s) [...] that this pod was serving are no longer live after the reload it triggered) needs multi-pod reload timing and did not trigger on a single boxAfter (afeed48, re-run green at 790d645 after the staging merge and at f8a23aa after the guardrail-review fixes)
model created on A, immediate /v1/chat/completions on B
curl -X POST $A/model/new ...(fresh namelit5791-after-miss-26853) returns 200curl -X POST $B/chat/completions -H "$K" -H "$J" -d '{"model": "lit5791-after-miss-26853", "messages": [{"role": "user", "content": "Reply with exactly: read-through works"}]}'returns HTTP 200:"choices":[{"finish_reason":"stop","index":0,"message":{"content":"read-through works","role":"assistant",...}}]model created on A, immediate /v1/responses on B
curl -X POST $A/model/new ...(fresh namelit5791-after-responses-*) returns 200curl -X POST $B/v1/responses ...(same payload shape as Before) returns HTTP 200 with a real response object:{"id":"resp_b6eh-WdsfbALI5MzCTklqw...model created on A, immediate /v1/messages on B
curl -X POST $A/model/new ...(fresh namelit5791-after-messages-6846) returns 200curl -X POST $B/v1/messages ...(same payload shape as Before) returns HTTP 200:{"id":"resp_...","type":"message","role":"assistant","model":"lit5791-after-messages-6846",...}guardrail created on A, applied on B
curl -X POST $A/guardrails ...(same payload shape as Before, fresh name) returns 200curl -X POST $B/guardrails/apply_guardrail ...returns HTTP 200:{"response_text":"hello world"}A2A agent created on A, message/send on B
curl -X POST $A/v1/agents ...(same payload shape as Before, fresh name) returns 200curl -X POST $B/a2a/<agent_name> ...returns HTTP 200:{"id":"1","jsonrpc":"2.0","result":{"kind":"message","role":"agent","parts":[{"kind":"text","text":"a2a works"}],...}}model created on A, access group naming it on B
curl -X POST $A/model/new ...(fresh namelit5791-after-xag-815) returns 200curl -X POST $B/access_group/new ...returns HTTP 200:{"access_group":"lit5791-after-xg-32490","model_names":["lit5791-after-xag-815"],"model_ids":null,"models_updated":1}unknown model on B keeps failing with 400
curl -X POST $B/chat/completions ...withlit5791-never-createdreturns HTTP 400 with the identical error text, still immediate (the 2s negative cache absorbs repeats)concurrent access-group writes during reload churn
curl -X POST $A/access_group/new ...writes, each naming its own modelnon-200 access-group writes: 0/200), and both replica logs contain zeroError upserting deploymentlines, so no deployment was dropped by the churncross-replica risk probes at the same tip
curl -X PUT $B/access_group/lit5791-risk-ag-3229/update ...naming a model just created on A returns HTTP 200:{"access_group":"lit5791-risk-ag-3229","model_names":["lit5791-risk-new-2527"],...,"models_updated":1}curl $B/a2a/<agent_name>/.well-known/agent-card.jsonfor an agent created on A returns HTTP 200 with the full agent cardcurl -X DELETE $B/access_group/lit5791-risk-ag-3229/deletereturns HTTP 200:{"models_updated":1,"message":"Access group 'lit5791-risk-ag-3229' deleted successfully"}curl -X POST $B/rerank ...with an unknown model returns HTTP 400 with the clean invalid-model error, so the retry wrapper regressed nothing on other route typescurl -X POST $B/a2a/lit5791-agent-nobody-created ...returns HTTP 404 JSON-RPC-32000Agent 'lit5791-agent-nobody-created' not found, and the miss does not consume the model resync budgetType
🐛 Bug Fix
Caveats
Changes
New
litellm/proxy/common_utils/registry_read_through.pyholds one smallRegistryReadThroughprimitive: an asyncio-locked "re-read from the DB, load into the local registry, answer whether the key exists now" step with a 2 second negative-result TTL, plus a global budget of 20 resyncs per 5 second window that fails closed (the miss stands, nothing is negative-cached) so a flood of distinct unknown names can never turn into a DB hot loop. Three singletons wrap it for the three registriesEach resync fetches only the missing object. The model resync selects just the rows matching the missing name (by
model_name, thenmodel_id) and upserts them through the additive deployment loader; it never runs the delete-reconciling full router rebuild with a partial list, which would evict healthy deployments. The agent resync does a unique lookup byagent_id, thenagent_name, including the object permission, and registers the row as-is. The guardrail resync goes through the same DB loader the periodic config reload usesFour miss sites consult it before giving up. The LLM request router retries routing once after a model read-through (covers
/v1/chat/completions,/v1/responses,/v1/messages, and every other route the shared router serves). The A2A agent card andmessage/sendlookups go through the agent read-through, as does A2A model routing.POST /guardrails/apply_guardrailre-checks after a guardrail read-through. Access group create and update re-validate their model names after a model read-through and only then 400Two hardenings close the reload race the 2-replica e2e suite hit. The router reload now returns an atomic snapshot of which model ids were still desired and live when it finished, and the access group endpoints judge their "saved but no longer serving" 500 against that snapshot instead of re-reading router state that a concurrent reload may have mutated. And
Router.upsert_deploymentrestores the previously served deployment when replacing it fails underignore_invalid_deployments, so a bad refresh degrades to the old config instead of silently dropping the modelA follow-up commit (afeed48) closes three gaps review found in the first cut. The model resync now mutates the router under
MODEL_RECONCILE_LOCK, the same lock the full reconcile snapshots and reconciles under, so a read-through upsert can no longer interleave with a reload and get evicted by its stale snapshot. A newAGENT_RECONCILE_LOCKserializes the agent resync with_init_agents_in_db's fetch-and-rebuild, with a re-check under the lock so an agent is never registered twice. Every resync first consultsshould_load_db_object(extracted to module level fromProxyConfig._should_load_db_object, which never used instance state), so a deployment that excludes an object type viasupported_db_objectsis honored by the read-through exactly as by the reload. And the A2A raise sites markProxyModelNotFoundErroras not retryable with a model read-through, so an unknown agent no longer burns a model resync on the way out. Each fix carries a regression test: lock-held assertions on both resyncs and the reload, gating skips per object type, a duplicate-registration guard, and a budget-preservation test on the A2A miss pathUnit tests cover the primitive (negative TTL expiry, no negative caching on resync exceptions or budget exhaustion, concurrent misses coalescing into one resync), one regression test per miss site driving the real recovery path against fake DB rows, and the upsert rollback
Two review findings at the merged tip are fixed in f8a23aa. The guardrail read-through now filters to
status: "active"in the DB query itself, mirroring the reload'sget_all_guardrails_from_db, so a team-submittedpending_reviewguardrail can no longer be activated by requesting it on a sibling pod. And a newGUARDRAIL_RECONCILE_LOCKserializes the guardrail resync with_init_guardrails_in_db's snapshot-through-reconcile, closing the same eviction race models and agents were already locked against. Both carry regression tests: a status-filter assertion on the resync query, and lock-held assertions on the resync and the reloadFinal Attestation
The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
f8a23aa passes /live-pr-risk (re-run at the tip; includes a live probe that a registered pending_review guardrail stays 404 on both pods)
Note
Medium Risk
Touches core proxy routing, model router reconciliation, and multi-replica consistency; changes are guarded by locks, budgets, and extensive tests but affect hot request paths.
Overview
Fixes stale in-memory registries on sibling proxy replicas: after a model, guardrail, or agent is created on one pod, requests on another no longer 400/404 until the periodic reload.
Adds
registry_read_through: on a miss it loads the row from Postgres into the local registry (with a 2s negative cache and a 20-per-5s resync cap), then retries once. Wired into LLM routing, A2A lookups,apply_guardrail, and access-group model validation. Resyncs run underMODEL_RECONCILE_LOCK/AGENT_RECONCILE_LOCKand honorsupported_db_objects. Unknown A2A agent errors skip model read-through.Access-group reload checks use
ReconcileOutcomefromclear_cacheinstead of a fresh router read, avoiding false 500s when concurrent reloads race.Router.upsert_deploymentrestores the previous deployment when an update fails underignore_invalid_deployments.Reviewed by Cursor Bugbot for commit 790d645. Bugbot is set up for automated code reviews on this repo. Configure here.