feat(memory): add automatic gateway memory with admin policies

[moe-berri]

TLDR

Problem this solves:

• Persistent memory currently requires applications to orchestrate memory tools.
• Gateway administrators cannot centrally choose automatic memory or opt-in.

How it solves it:

• Adds gateway-managed saving, search, recall, and memory management.
• Reuses existing authentication, organization hierarchy, billing, and database.
• Provides an isolated Render pilot using callers' existing gateway keys.

User Flow

Before: a developer loses their saved project preference in a fresh conversation.

1. They send POST http://localhost:4019/v1/chat/completions with “Remember my demo project's staging port is 8753.”
2. They start a fresh conversation and ask for that project's port.
3. The response says unknown.
4. They open http://localhost:4019/ui/memory and see a deprecation banner and manual memory administration.

After: the developer's enabled gateway remembers the preference across conversations.

1. They send POST http://localhost:4018/v1/chat/completions with “Remember my demo project's staging port is 8753.” An administrator has enabled automatic, private-per-key memory at http://localhost:4018/ui/memory.
2. They start a fresh conversation and ask for that project's port.
3. The streamed response says 8753 using the same gateway key.
4. They open http://localhost:4018/ui/memory to inspect, correct, or delete their memories. Administrators can choose automatic, opt-in, or disabled policies for existing gateway, organization, team, project, user, and key targets.
5. Another key cannot retrieve or delete their private memories, including a sibling key belonging to the same user.

The Render forwarding pilot preserves each caller's original upstream key and model. Colleagues change only the base URL. Its memories are private per key; full organization sharing requires installing this feature in the gateway that owns those identities.

Relevant issues

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally
• My PR passes all required CI/CD checks
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Real provider: vertex_ai/claude-sonnet-5 through the existing upstream gateway. Before uses a clean merge-base checkout on port 4019 with the memory-test alias; After uses this branch's forwarding pilot on port 4018 with the original model name and caller key. Each case uses a fresh random project name and two independent conversations. Keys are supplied through a private curl configuration, never embedded in the evidence.

Shared command shape, with $BODY containing the endpoint-native JSON:

curl --config "$PRIVATE_AUTH_CONFIG" --no-buffer \
  -H 'Content-Type: application/json' \
  "$BASE_URL$ENDPOINT" --data-binary "$BODY"

Chat/Messages bodies use messages: [{role: "user", content: PROMPT}] and max_tokens: 250. Responses uses input: PROMPT, max_output_tokens: 250, and store: false. Seed uses stream: false; fresh recall uses stream: true. Seed asks to remember the project's staging port, 8753. Recall asks for that project's port without repeating it.

Before (9071ca5)

Chat Completions

1. Send the seed to /v1/chat/completions: HTTP 200, acknowledges the port for this conversation.
2. Send fresh streamed recall: HTTP 200, cannot recall the port (unknown).

Responses

1. Send the seed to /v1/responses: HTTP 200, acknowledges the port for this conversation.
2. Send fresh streamed recall: HTTP 200, cannot recall the port (unknown).

Anthropic Messages

1. Send the seed to /v1/messages: HTTP 200, acknowledges the port for this conversation.
2. Send fresh streamed recall: HTTP 200, cannot recall the port (unknown).

Memory dashboard

1. Log in through the actual /ui/login form and open /ui/memory.
2. Observe the deprecation banner and absence of automatic gateway memory settings. Screenshot below.

After (ecf23d7)

Chat Completions

1. Send the seed to /v1/chat/completions: HTTP 200, confirms saving the project port.
2. Send fresh streamed recall: HTTP 200, 8753; no gateway memory tool events appear in the stream.

Responses

1. Send the seed to /v1/responses: HTTP 200, confirms saving the project port.
2. Send fresh streamed recall: HTTP 200, 8753; no gateway memory tool events appear in the stream.

Anthropic Messages

1. Send the seed to /v1/messages: HTTP 200, acknowledges the project port.
2. Send fresh streamed recall: HTTP 200, 8753; no gateway memory tool events appear in the stream.

Memory dashboard

1. Log in through the actual /ui/login form and open /ui/memory.
2. Select Whole gateway, Enabled automatically, Private to each virtual key, and Save memory policy.
3. Observe the success confirmation, persisted policy, saved entries, and removal of the memory deprecation banner. Screenshot below.

Additional verification: 12 real-provider E2E cases passed against both the monolithic gateway and separate gateway/backend processes. The same 12 cases also passed with gpt-5.5 and claude-haiku-4-5, matching CI's model selection, with tracing enabled and a Redis cluster. 95 focused Python tests, 219 capability tests, 3 dashboard integration tests, and 41 navigation tests passed. make check, the dashboard production build, and fresh Postgres migrations passed. A real PostgreSQL concurrency test seeded 999 entries and raced five creates: exactly one succeeded, four returned 429, and storage remained at 1000 entries. Correction at capacity and deletion followed by a new save both passed. Real Claude Code saved a preference, recalled 8629 in a new process, and used its normal Read tool while recalling the same preference.

The Render pilot is deployed at https://litellm-memory-pilot.onrender.com. The deployed revision is ecf23d761d. It passed all three real-provider streamed recall checks and the hosted dashboard login, search, correction, and deletion flow. Real Claude Code also passed on this revision: independent processes saved and recalled port 8629, and its normal Read tool remained usable while recalling that preference. It has automatic key-private memory enabled. No upstream provider credential is stored in its configuration.

Final revision validation: all 33 branch-required checks passed. The stage-mirror changed-test gate passed 40 tests three consecutive times (231.45s, 227.38s, 225.46s). Greptile scored the current commit 5/5, Bugbot found no new issues, and Veria reported both findings addressed with no open security issues. No review threads remain unresolved. The separate, non-required Buildkite check is still pending. Human approvals, including the migration reviewer requirement, remain necessary before merge.

Type

New Feature

Caveats (if any)

Severe

• Enabling memory adds billed preparation calls before visible responses.
• Preparation failures fail enabled requests; administrators can disable memory.

Medium

• Memory repository operations intentionally run in the request path. Existing authentication supplies identity; fresh primary policy checks prevent revocation during preparation from permitting later saves. The existing identity helpers do not store or retrieve memory records.

• Without shared Redis, first-time activation can take up to 30 seconds to reach another worker.

• Models must support function calling; preparation has a 60-second deadline.

• Long conversations repeat prompt tokens during bounded preparation.

• Pilot namespaces follow keys; shared keys share memories.

• Upstream organization policies require deployment within the authoritative gateway.

• Keyword search only; no embeddings or nightly consolidation.

• Each memory namespace holds up to 1000 entries; corrections remain available at capacity.

• Forwarding pilot supports three inference protocols and memory management.

QA runbook

Prerequisites: a live Postgres-backed gateway, administrator key, and a real tool-capable model configured as CHEAP_OPENAI_MODEL and CHEAP_ANTHROPIC_MODEL. Create two users, one team, two keys for the first user and a separate key for the second user. Limit each key to one concurrent request. Remove created entries, policies, preferences, keys, team, and users after checking.

• tests/e2e/management/test_memory_v2_e2e.py::TestMemoryV2::test_gateway_stores_and_recalls_without_client_memory_tools proves capture and fresh recall across six protocol/stream combinations.

  • PUT /v2/memory/policies as admin with {"target_type":"team","target_id":"TEAM","activation":"automatic","scope":"key"}.
  • POST /v1/chat/completions as the owner, asking to remember a unique project codename. GET /v2/memory/entries and find the exact word.
  • Ask for the codename in a fresh request to each of /v1/chat/completions, /v1/responses, /v1/messages, with streaming both false and true; expect HTTP 200 and that word. Streaming must emit multiple chunks without memory tool events.
  • GET entries with sibling and unrelated keys; expect empty lists.
  • Sanity check: use a fresh random word and real provider; no previous conversation supplies the answer.

• tests/e2e/management/test_memory_v2_e2e.py::TestMemoryV2::test_admin_selects_opt_in_or_automatic_and_key_override_wins proves activation precedence.

  • Change the team policy to opt_in; GET /v2/memory/status reports inactive. A remember request creates no entry.
  • PUT /v2/memory/preference as owner with {"enabled":true}; both that user's team keys become active, unrelated key stays inactive.
  • Save an entry; set an owner-key policy disabled using its SHA256 key ID. Owner becomes inactive, sibling remains active.
  • Change owner-key policy to automatic and preference to false; owner stays active, sibling becomes inactive.
  • Sanity check: inspect status with each actual credential after every policy change.

• tests/e2e/management/test_memory_v2_e2e.py::TestMemoryV2::test_private_entries_are_isolated_even_for_sibling_keys_and_legacy_api proves tenant isolation.

  • POST /v2/memory/entries as owner with key, title, content, and evidence; record its ID.
  • Sibling and unrelated keys GET entries: empty. DELETE that ID and GET entries with key_id=OWNER_HASH: denied.
  • GET legacy /v1/memory as sibling: either denied or excludes the private entry. Owner's content remains unchanged.
  • Sanity check: sibling key must belong to the same user and team, so this checks key privacy beyond user isolation.

• tests/e2e/management/test_memory_v2_e2e.py::TestMemoryV2::test_members_cannot_enable_or_broaden_memory proves admin-only policy writes.

  • As an ordinary key, PUT automatic, team-shared policies targeting gateway, team, and user.
  • Expect each attempt denied and the caller's status unchanged.
  • Sanity check: use a real non-admin key and compare status before and after.

• tests/e2e/management/test_memory_v2_e2e.py::TestMemoryV2::test_corrections_require_current_revision_and_deleted_memory_is_not_recalled proves correction safety and deletion.

  • Save an entry and replace it using its updated_at as expected_revision; expect the same ID and corrected content.
  • Repeat using the stale revision; expect HTTP 409 and unchanged corrected content.
  • DELETE the entry; GET entries returns empty. Ask the model for the codename in a fresh conversation; the deleted word must not appear.
  • Sanity check: verify both storage state and the real model answer.

• tests/e2e/management/test_memory_v2_e2e.py::TestMemoryV2::test_client_tool_call_and_continuation_remain_owned_by_client proves client tool compatibility.

  • Store a unique codename; send Chat Completions with client tool verify_release, required word, and tool_choice: "required".
  • Expect one verify_release call containing the saved word and a tool call ID.
  • Return a unique tool result under that ID and continue; expect the result in the model answer.
  • Sanity check: the gateway must not execute or swallow the client's tool call.

• tests/e2e/management/test_memory_v2_e2e.py::TestMemoryV2::test_preparation_and_answer_are_charged_once_to_the_calling_key proves accounting.

  • Send one remember request as a fresh owner key and verify the entry exists.
  • Inspect /spend/logs for that key: expect 2–4 records with distinct request IDs, correct key hash, user, team, positive spend, and nonzero prompt/completion tokens.
  • Sanity check: include preparation and visible answer, exclude other calls, and keep max_parallel_requests at one.

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Hooks the main inference pre-call path with extra billed LLM rounds and new authorization/namespace rules; failures or misconfiguration can affect latency, spend, and recall isolation for enabled keys.

Overview
Introduces gateway Memory V2: administrators can enable automatic or opt-in persistent memory for Chat Completions, Responses, and Anthropic Messages without client-side memory tools. Policies target gateway/org/team/project/user/key with configurable sharing scope; users can opt in when required.

Inference path: Before the visible model call, prepare_gateway_memory may run bounded server-side tool rounds (search/read/capture) via in-process ASGI dispatch, then injects retrieved facts as untrusted reference context while leaving the client’s original tools and streaming settings intact. Storage: Postgres gains namespace on memory rows plus LiteLLM_MemoryPolicy / LiteLLM_MemoryPreference; entries are isolated by namespace with keyword search and revision-checked updates. API/UI: New /v2/memory/* routes (policies backend-only; status/preference/entries on gateway too) and dashboard sections for policies, preference, and per-key saved memories; legacy /v1/memory reads exclude namespaced rows.

Ops: Parallel-request limiter v3 defers the next internal prep round until the prior call releases its slot. deploy/memory-pilot adds a Render forwarding gateway that validates upstream keys, maps them to local virtual keys, and proxies inference for an isolated pilot.

^{Reviewed by Cursor Bugbot for commit 142088d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[moe-berri]

@veria-ai Please review the current branch for authorization boundaries, memory isolation, protocol compatibility, accounting, and deployment correctness.

[moe-berri]

bugbot run

Please review the current branch, especially memory policy enforcement, request recursion, tenant isolation, and client tool continuation.

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_memory_v2 (ecf23d7) with litellm_internal_staging (9071ca5)}

[codecov]

Codecov Report

❌ Patch coverage is 94.60967% with 29 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/memory/gateway.py	90.47%	10 Missing ⚠️
...itellm_core_utils/prompt_templates/server_tools.py	84.31%	8 Missing ⚠️
litellm/proxy/memory/policy.py	91.91%	8 Missing ⚠️
litellm/types/memory_v2.py	96.49%	2 Missing ⚠️
litellm/proxy/memory/store.py	98.64%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[moe-berri]

@greptileai Please review the updated branch, including the Render build fixes, memory authorization, protocol handling, and request accounting.

[greptile-apps]

Greptile Summary

This PR introduces gateway-managed persistent memory, including policy-based activation and namespace isolation, protocol-aware preparation for three inference APIs, management endpoints and dashboard controls, database persistence, request accounting coordination, and an isolated forwarding pilot. Since the previous review, it additionally enforces an atomic 1,000-entry limit per memory namespace using a transaction-scoped advisory lock.

• Serializes namespace quota checks and creation in one PostgreSQL transaction.
• Keeps correction and deletion available when a namespace reaches its limit.
• Adds focused coverage for full-scope rejection, correction, deletion, and reclaimed capacity.
• Documents the new per-scope limit in the pilot README.

Confidence Score: 5/5

The updated PR appears safe to merge; no new actionable failure was found and no previous blocking finding remains outstanding.

The new quota path serializes count-and-create operations per stable namespace, uses established Prisma transaction APIs, and leaves correction and deletion outside the creation cap. moe-berri accepted the documented feature-specific critical-path database tradeoff because bounded memory operations require fresh primary data; that resolved finding does not remain outstanding. The FastAPI dependency and lowercase-bearer threads were manually resolved without explanation.

Important Files Changed

Filename	Overview
litellm/proxy/memory/store.py	Adds an atomic, namespace-scoped entry quota while preserving revision-based corrections and authorization revalidation.
tests/test_litellm/proxy/memory/test_memory_v2_boundaries.py	Covers full-scope rejection and verifies that corrections, deletion, and subsequent creation remain available.
tests/test_litellm/proxy/memory/test_memory_v2_management.py	Updates management-test database fixtures for transaction, count, and advisory-lock behavior.
deploy/memory-pilot/README.md	Documents the atomic 1,000-entry limit and recovery through deletion.
litellm/proxy/memory/gateway.py	Performs bounded protocol-aware memory preparation with per-operation authorization revalidation and final access checking.
deploy/memory-pilot/pilot.py	Validates preserved upstream credentials while delegating inference to the standard gateway application.

_{Reviews (7): Last reviewed commit: "fix(memory): enforce an atomic namespace..." | Re-trigger Greptile}

[cursor]

Stale Bugbot comment from a previous run.

[moe-berri]

@greptileai Please review the updated branch, including the Render build fixes, memory authorization, protocol handling, and request accounting.

[moe-berri]

@veria-ai Please review the current branch for authorization boundaries, memory isolation, protocol compatibility, accounting, and deployment correctness.

[moe-berri]

bugbot run

Please review the current branch, especially memory policy enforcement, request recursion, tenant isolation, and client tool continuation.

[cursor]

Stale Bugbot comment from a previous run.

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 2 · PR risk: 0/10

[moe-berri]

@greptileai Please review the updated branch, including the Render build fixes, memory authorization, protocol handling, and request accounting.

[moe-berri]

@veria-ai Please review the current branch for authorization boundaries, memory isolation, protocol compatibility, accounting, and deployment correctness.

[moe-berri]

bugbot run

Please review the current branch, especially memory policy enforcement, request recursion, tenant isolation, and client tool continuation.

[cursor]

Stale Bugbot comment from a previous run.

[moe-berri]

@greptileai Please review the updated branch, including the Render build fixes, memory authorization, protocol handling, and request accounting.

[moe-berri]

@veria-ai Please review the current branch for authorization boundaries, memory isolation, protocol compatibility, accounting, and deployment correctness.

[moe-berri]

bugbot run

Please review the current branch, especially memory policy enforcement, request recursion, tenant isolation, and client tool continuation.

[cursor]

Stale Bugbot comment from a previous run.

[moe-berri]

@greptileai Please review the updated branch, including the Render build fixes, memory authorization, protocol handling, and request accounting.

[moe-berri]

@veria-ai Please review the current branch for authorization boundaries, memory isolation, protocol compatibility, accounting, and deployment correctness.

[moe-berri]

bugbot run

Please review the current branch, especially memory policy enforcement, request recursion, tenant isolation, and client tool continuation.

[cursor]

Stale Bugbot comment from a previous run.

[moe-berri]

@greptileai Please review the updated branch, including the Render build fixes, memory authorization, protocol handling, and request accounting.

[moe-berri]

@veria-ai Please review the current branch for authorization boundaries, memory isolation, protocol compatibility, accounting, and deployment correctness.

[moe-berri]

bugbot run

Please review the current branch, especially memory policy enforcement, request recursion, tenant isolation, and client tool continuation.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit ecf23d7. Configure here.}

[moe-berri]

Closing because this implementation confines memory tools to preparation. The replacement will expose them to the active model throughout gateway-managed tool execution.