feat(memory): add gateway memory with separate admin saving and recall controls

backend/routes/allowlist.py

@@ -71,6 +71,7 @@
     "/v1/workflows/",
     "/project/",
     "/memory/",
+    "/v2/memory/",
     "/mcp/",
     # Control plane (see the List Endpoints + Tables standard). Every resource
     # eventually moves under this prefix, so allowlist it once rather than

deploy/memory-pilot/README.md

@@ -0,0 +1,122 @@
+# Memory gateway pilot on Render
+
+Run this branch as an isolated forwarding gateway. Colleagues keep their existing
+upstream LiteLLM key and model name and change only their gateway base URL. They
+need no plugin or client-side memory tools. Choosing the pilot URL opts them into
+the pilot; returning to the original URL stops using and collecting pilot memory.
+
+Every model call uses that caller's upstream key. The upstream
+gateway continues to enforce its model permissions, budgets, rate limits, and
+guardrails. The pilot checks the key against the upstream model catalog, then
+registers its hash as a local virtual key so LiteLLM's normal authentication and
+memory authorization still apply. No upstream key or provider credential is
+configured on Render. The administrator credential belongs only to this pilot.
+
+The forwarding pilot isolates memories by virtual key. Upstream management APIs
+may deny ordinary keys access to user/team/org details, so the pilot does not
+infer those identities from client metadata. Install the feature directly in an
+organization's gateway to use its existing user/team/project/org policies.
+A regular gateway deployment reuses its existing PostgreSQL database with normal
+schema migrations. It does not need a separate memory database or vector service.
+This forwarding pilot has a separate database for isolation. Its memories are not
+automatically available on the original gateway. Sharing requires both deployments
+to run this feature against the same database and authenticated namespace; the
+pilot must not be connected to an older gateway's production database.
+
+## Create the service
+
+1. Create a separate Render Postgres 16 database in the same region as the web
+   service. Restrict public database access; use its internal connection URL.
+2. Create a Python web service from this repository and the Memory V2 branch.
+   A Standard service and the smallest paid Postgres plan are sufficient starting
+   points for a small pilot. They incur Render hosting charges.
+3. Set the build command to `bash deploy/memory-pilot/build.sh`, the start command
+   to `bash deploy/memory-pilot/start.sh`, and the health path to
+   `/health/readiness`. The build includes the dashboard from this branch.
+4. Set these environment variables in Render:
+
+   | Variable | Value |
+   | --- | --- |
+   | `DATABASE_URL` | The new database's internal connection URL |
+   | `UPSTREAM_LITELLM_BASE_URL` | Your original gateway URL, without `/v1` |
+   | `LITELLM_MASTER_KEY` | A new random `sk-` administrator key |
+   | `LITELLM_SALT_KEY` | A separate random encryption secret; preserve it across deploys |
+   | `PYTHON_VERSION` | `3.12.14` |
+   | `NODE_VERSION` | `24.19.0` |
+   | `NEXT_TELEMETRY_DISABLED` | `1` |
+   | `PORT` | `4000` |
+
+5. After deployment, open `/ui/memory`, sign in as `admin` using the pilot's master
+   key, and save a policy for **Whole gateway**, **Enabled automatically**,
+   **Private to each virtual key**. This policy persists across restarts. Memory
+   stays disabled until an administrator enables it.
+
+Equivalent activation through the API, with secrets supplied in shell variables:
+
+```bash
+curl --fail-with-body "$PILOT_URL/v2/memory/policies" \
+  -H "Authorization: Bearer $PILOT_ADMIN_KEY" \
+  -H 'Content-Type: application/json' \
+  -X PUT \
+  -d '{"target_type":"gateway","target_id":"*","activation":"automatic","scope":"key"}'
+```
+
+Administrators can instead require opt-in, disable a particular registered key,
+or disable the whole gateway. Under an opt-in policy, callers set their preference
+with `PUT /v2/memory/preference` and `{"enabled":true}` using their own key.
+
+## Try it
+
+Set an OpenAI-compatible client's base URL to `https://YOUR-SERVICE.onrender.com/v1`.
+For Claude Code, set `ANTHROPIC_BASE_URL` to `https://YOUR-SERVICE.onrender.com`.
+Retain the same gateway key and model setting.
+
+In one conversation, say “Remember that my demo project is Cobalt Heron and its
+staging port is 8347.” In a **new conversation**, ask “What is my demo project and
+its staging port?” Check actual saved entries with `GET /v2/memory/entries` using
+the same key. An unrelated key must not see them. Administrators can inspect,
+correct, or delete entries in Memory; callers can use the self-service API.
+
+## Behavior and limits
+
+- Supported surfaces: Chat Completions, Responses, and Anthropic Messages,
+  including their native streaming responses and client tool continuation.
+- The selected model must support function calling. The actual answering model
+  receives catalog, fuzzy search, full-read, and observation-capture tools beside
+  its normal client tools. The gateway executes only its own memory tools.
+- A request allows at most eight model rounds and sixteen memory calls per round.
+  One final reflection round can acknowledge an empty observation batch. Additional
+  rounds use the same model and caller budget, and add latency and token spend.
+- Captures are immediately visible after a confirmed save. Each observation keeps
+  its title, relevance guidance, scope, kind, certainty, evidence, source, and actor.
+  Corrections append observations. Agents receive no memory deletion tool.
+- Search uses weighted fuzzy matching over the authorized scope. There is no vector
+  database, extraction model, or nightly consolidation.
+- Fixed instructions and tool definitions preserve prompt-prefix caching after
+  warm-up. Dynamic catalogs and checkpoint IDs stay at the conversation tail.
+  Complete-response caching is bypassed for memory rounds on both gateways so
+  permission checks, retrieval, and capture execute against current state.
+- Hidden tool continuations expire after 24 hours, hold at most one megabyte each,
+  and are limited to 1,000 per key and scope. They contain gateway-added fragments,
+  not another copy of the complete incoming transcript. Responses retrieval and
+  continuation use gateway-owned response IDs; deleting one removes its model
+  responses and temporary continuation records, not saved memories.
+- Foreground requests with one completion are supported. Use modern tools instead
+  of legacy functions. The special Cursor conversion route, background responses,
+  multiple completions, and WebSocket inference are outside this implementation.
+- On gateway/backend deployments without shared Redis, first-time activation
+  can take up to 30 seconds to reach another process. Policy revocation is
+  checked against the primary database before memory operations.
+- Each memory scope can hold up to 1,000 entries. Creation checks this limit
+  atomically; correction and deletion remain available when the scope is full.
+- Stored references are untrusted data. They cannot grant API permissions or
+  change the namespace derived from authentication. Current user corrections
+  take precedence. Replacements require the current revision.
+- Invalid tool arguments return errors to the model. Infrastructure and model
+  failures fail the request or stream instead of reporting a successful save. Administrators can disable memory to restore ordinary calls.
+- Switching away or disabling memory stops automatic use; it does not delete
+  existing entries. Delete memories explicitly through Memory or the API.
+- Shared upstream keys share a pilot namespace. Give each person a distinct key
+  when their memories must be private from each other.
+- Other API surfaces are outside this forwarding pilot. Use the original gateway
+  for embeddings, images, realtime, batches, and administration.

deploy/memory-pilot/build.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+set -euo pipefail
+# Render's preinstalled Rust toolchain directory is read-only. Maturin needs
+# writable homes to install the repository's pinned toolchain during uv sync.
+export RUSTUP_HOME="$PWD/.memory-pilot-rustup"
+export CARGO_HOME="$PWD/.memory-pilot-cargo"
+export PRISMA_BINARY_CACHE_DIR="$PWD/.memory-pilot-prisma"
+python -m pip install uv==0.11.7
+export UV_PROJECT_ENVIRONMENT="$PWD/.memory-pilot-venv"
+uv sync --frozen --extra proxy --extra extra_proxy --no-default-groups
+export PATH="$UV_PROJECT_ENVIRONMENT/bin:$PATH"
+prisma generate --schema schema.prisma
+(
+  cd ui/litellm-dashboard
+  npm ci
+  npm run build
+)
+rm -rf litellm/proxy/_experimental/out
+cp -R ui/litellm-dashboard/out litellm/proxy/_experimental/out

deploy/memory-pilot/hooks.py

@@ -0,0 +1 @@
+from pilot import forward_credential as forward_credential

deploy/memory-pilot/pilot.py

@@ -0,0 +1,168 @@
+"""An isolated office pilot that preserves upstream gateway credentials."""
+
+import asyncio
+import hashlib
+import os
+import secrets
+from contextvars import ContextVar
+from typing import Final
+
+import httpx
+from starlette.exceptions import HTTPException
+from starlette.requests import Request
+from starlette.responses import JSONResponse
+from starlette.types import ASGIApp, Receive, Scope, Send
+
+from litellm.caching.caching import DualCache
+from litellm.integrations.custom_logger import CustomLogger
+from litellm.litellm_core_utils.prompt_templates.server_tool_responses import object_value
+from litellm.llms.custom_httpx.http_handler import get_async_httpx_client
+from litellm.proxy._types import UI_TEAM_ID, UserAPIKeyAuth
+from litellm.proxy.auth.auth_checks import ExperimentalUIJWTToken
+from litellm.proxy.auth.user_api_key_auth import _get_bearer_token_or_received_api_key
+from litellm.proxy.memory.transport import in_gateway_round
+from litellm.repositories.verification_token_repository import VerificationTokenRepository
+from litellm.types.llms.custom_http import httpxSpecialProvider
+from litellm.types.utils import CallTypesLiteral
+
+_UPSTREAM: Final = os.environ["UPSTREAM_LITELLM_BASE_URL"].rstrip("/")
+_CREDENTIAL: Final[ContextVar[str | None]] = ContextVar("memory_pilot_credential", default=None)
+_INFERENCE: Final = frozenset(
+    ("/chat/completions", "/v1/chat/completions", "/responses", "/v1/responses", "/v1/messages")
+)
+_SELF_SERVICE: Final = frozenset(("/v2/memory/status", "/v2/memory/preference", "/v2/memory/entries"))
+
+
+class ForwardCredential(CustomLogger):
+    async def async_pre_call_hook(
+        self, user_api_key_dict: UserAPIKeyAuth, cache: DualCache, data: dict[str, object], call_type: CallTypesLiteral
+    ) -> dict[str, object]:
+        credential: Final = _CREDENTIAL.get()
+        if credential is None:
+            raise HTTPException(status_code=403, detail="Use your upstream gateway key for model calls")
+        return {  # mutable-ok: The gateway hook returns native provider request JSON.
+            **data,
+            "api_key": credential,
+            "api_base": _UPSTREAM,
+            **(
+                {  # mutable-ok: The second gateway must receive its own cache controls in the provider body.
+                    "extra_body": {  # mutable-ok: The proxy provider forwards this JSON unchanged.
+                        **object_value(data.get("extra_body")),
+                        "cache": {
+                            "no-cache": True,
+                            "no-store": True,
+                        },  # mutable-ok: Native upstream gateway cache controls.
+                    },
+                }
+                if in_gateway_round()
+                else {}
+            ),  # mutable-ok: Hook payload is native JSON.
+        }
+
+
+forward_credential: Final = ForwardCredential()
+
+
+class PilotGateway:
+    def __init__(self, app: ASGIApp) -> None:
+        self.app = app
+        self.validation_slots = asyncio.Semaphore(16)
+        self.upstream = get_async_httpx_client(
+            httpxSpecialProvider.PassThroughEndpoint,
+            params={"timeout": 20, "client_alias": "memory-pilot-upstream"},
+        )
+
+    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+        if scope["type"] == "lifespan":
+            try:
+                await self.app(scope, receive, send)
+            finally:
+                await self.upstream.close()
+            return
+        if scope["type"] != "http":
+            await self.app(scope, receive, send)
+            return
+        request: Final = Request(scope, receive)
+        credential: Final = _get_bearer_token_or_received_api_key(
+            request.headers.get("x-litellm-api-key")
+            or request.headers.get("authorization")
+            or request.headers.get("x-api-key")
+            or ""
+        )
+        from litellm.proxy.proxy_server import master_key, prisma_client
+
+        if not credential or master_key and secrets.compare_digest(credential, master_key):
+            await self.app(scope, receive, send)
+            return
+        if prisma_client is None:
+            await JSONResponse({"error": "Pilot database unavailable"}, status_code=503)(scope, receive, send)
+            return
+        digest: Final = hashlib.sha256(credential.encode()).hexdigest()
+        tokens: Final = VerificationTokenRepository(prisma_client)
+        local_key: Final = await tokens.find_by_id(digest)
+        if local_key and local_key.team_id == UI_TEAM_ID:
+            await self.app(scope, receive, send)
+            return
+        if (
+            not credential.startswith("sk-")
+            and ExperimentalUIJWTToken.get_key_object_from_ui_hash_key(credential) is not None
+        ):
+            await self.app(scope, receive, send)
+            return
+        path: Final = request.url.path.rstrip("/")
+        memory_response: Final = request.method in ("GET", "DELETE") and path.startswith(
+            ("/v1/responses/resp_litellm_memory_", "/responses/resp_litellm_memory_")
+        )
+        if (
+            path not in _INFERENCE | _SELF_SERVICE | {"/models", "/v1/models"}
+            and not path.startswith("/v2/memory/entries/")
+            and not memory_response
+        ):
+            await JSONResponse(
+                {"error": "Upstream keys can only use inference and their own memories"}, status_code=403
+            )(scope, receive, send)
+            return
+        try:
+            await asyncio.wait_for(self.validation_slots.acquire(), timeout=0.05)
+        except TimeoutError:
+            await JSONResponse(
+                {"error": "Pilot credential validation is busy; retry shortly"},
+                status_code=503,
+                headers={"Retry-After": "1"},
+            )(scope, receive, send)
+            return
+        try:
+            models: Final = await self.upstream.get(
+                _UPSTREAM + "/v1/models", headers={"Authorization": "Bearer " + credential}
+            )
+        except httpx.HTTPError:
+            await JSONResponse({"error": "Upstream gateway unavailable"}, status_code=503)(scope, receive, send)
+            return
+        finally:
+            self.validation_slots.release()
+        if models.is_error:
+            await JSONResponse({"error": "Upstream gateway rejected this key"}, status_code=models.status_code)(
+                scope, receive, send
+            )
+            return
+        if path in ("/models", "/v1/models"):
+            await JSONResponse(models.json())(scope, receive, send)
+            return
+        await tokens.table.upsert(
+            where={"token": digest},
+            data={
+                "create": {"token": digest, "models": [], "key_alias": "Memory pilot " + digest[:8]},
+                "update": {},
+            },
+        )
+        token: Final = _CREDENTIAL.set(credential)
+        try:
+            await self.app(scope, receive, send)
+        finally:
+            _CREDENTIAL.reset(token)
+
+
+def create_app() -> PilotGateway:
+    from litellm.proxy.proxy_server import app
+
+    return PilotGateway(app)

deploy/memory-pilot/proxy_config.yaml

@@ -0,0 +1,15 @@
+model_list:
+  - model_name: "*"
+    litellm_params:
+      model: litellm_proxy/*
+      api_base: os.environ/UPSTREAM_LITELLM_BASE_URL
+    model_info:
+      supports_function_calling: true
+litellm_settings:
+  callbacks:
+    - hooks.forward_credential
+  drop_params: true
+  turn_off_message_logging: true
+general_settings:
+  master_key: os.environ/LITELLM_MASTER_KEY
+  store_model_in_db: true

deploy/memory-pilot/start.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+set -euo pipefail
+export PATH="$PWD/.memory-pilot-venv/bin:$PATH"
+export PRISMA_BINARY_CACHE_DIR="$PWD/.memory-pilot-prisma"
+export PRISMA_CLI_PATH="$PRISMA_BINARY_CACHE_DIR/node_modules/.bin/prisma"
+prisma migrate deploy --schema litellm-proxy-extras/litellm_proxy_extras/schema.prisma
+export WORKER_CONFIG="$PWD/deploy/memory-pilot/proxy_config.yaml"
+export PYTHONPATH="$PWD/deploy/memory-pilot${PYTHONPATH:+:$PYTHONPATH}"
+exec uvicorn pilot:create_app --factory --host 0.0.0.0 --port "${PORT:-4000}"

gateway/routes/allowlist.py

@@ -74,6 +74,7 @@
     "/containers",
     "/v1/evals",
     "/v1/memory",
+    "/v2/memory/entries",
     "/queue/chat/",
     # Google data plane (v1beta is the Google AI Studio version)
     "/v1beta/",
@@ -123,6 +124,8 @@
         "/redoc",
         "/test",
         "/debug/memory/summary",
+        "/v2/memory/status",
+        "/v2/memory/preference",
     }
 )
 

litellm-proxy-extras/litellm_proxy_extras/migrations/20260912000000_gateway_memory_v2/migration.sql

@@ -0,0 +1,26 @@
+ALTER TABLE "LiteLLM_MemoryTable" ADD COLUMN IF NOT EXISTS "namespace" TEXT;
+
+CREATE INDEX IF NOT EXISTS "LiteLLM_MemoryTable_namespace_updated_at_idx"
+ON "LiteLLM_MemoryTable"("namespace", "updated_at");
+
+CREATE TABLE IF NOT EXISTS "LiteLLM_MemoryPolicy" (
+    "policy_id" TEXT NOT NULL,
+    "target_type" TEXT NOT NULL,
+    "target_id" TEXT NOT NULL,
+    "activation" TEXT NOT NULL,
+    "scope" TEXT NOT NULL,
+    "created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    "updated_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    "updated_by" TEXT NOT NULL,
+    CONSTRAINT "LiteLLM_MemoryPolicy_pkey" PRIMARY KEY ("policy_id")
+);
+
+CREATE UNIQUE INDEX IF NOT EXISTS "LiteLLM_MemoryPolicy_target_type_target_id_key"
+ON "LiteLLM_MemoryPolicy"("target_type", "target_id");
+
+CREATE TABLE IF NOT EXISTS "LiteLLM_MemoryPreference" (
+    "subject" TEXT NOT NULL,
+    "enabled" BOOLEAN NOT NULL DEFAULT false,
+    "updated_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    CONSTRAINT "LiteLLM_MemoryPreference_pkey" PRIMARY KEY ("subject")
+);