feat(memory): add gateway memory with separate admin saving and recall controls

[moe-berri]

TLDR

Problem this solves:

• Persistent memory currently requires application-side orchestration
• Administrators cannot centrally control saving and agent recall separately

How it solves it:

• Adds gateway-managed memory, with saving and recall defaulting off
• Admins enable each independently for everyone or selected users
• Existing clients use gateway-owned capture, search, and read tools
• Existing user and team permissions determine which records are visible
• Admins customize what assistants should remember

User Flow

Before: a user asks an assistant to remember a project port, but a fresh conversation cannot recall it

1. They send POST https://gateway.example/v1/chat/completions, asking to remember the project's staging port
2. They start a fresh conversation and ask for the same port
3. The assistant cannot recall it unless their application supplies memory

After: the administrator enables saving and recall, and the same client can remember across conversations

1. They send POST https://gateway.example/v1/chat/completions, asking to remember the project's staging port
2. They start a fresh conversation and ask for the same port
3. The assistant searches saved memories and returns the port
4. Their other keys can recall the same fact under the same permissions
5. At https://gateway.example/ui/memory/, they see recent memories, contributors, and separate saving/recall states

At Memory > Administration, the proxy admin controls Save memories and Use saved memories independently. Each has an on/off switch and an Everyone/Selected users selector. The selection follows the user across their keys. No end-user opt-in, per-key preference, client hook, or local installation is needed. With no V2 configuration, saving and recall are off. Existing V2 settings in the database persist across deployments; V1 settings do not enable V2

Admin settings for a user	What the agent can do
Saving off, recall off	No automatic memory tools
Saving on, recall off	Save new facts; cannot retrieve saved memories
Saving off, recall on	Search/read existing memories; cannot save new ones
Saving on, recall on	Save, search, and read

These controls govern automatic agent memory. They do not delete records or change human dashboard/API visibility. Users see their own records; team admins can also see their team's records. Ordinary members can receive team memory access through the existing Member Permissions editor. Proxy admins can see all V2 records. When enabled for a proxy-admin inference identity, recall can therefore send records from any user or organization to that model. Read permission does not grant editing or deletion, and agents have no deletion tool. V1 behavior is unchanged

What should memory remember?

Memory > Administration now includes an editable What should memory remember? field. A proxy admin can set gateway-wide capture guidance, such as “Remember architectural decisions and deployment requirements; skip routine updates and personal preferences.” Click Save changes to apply it to future saves without a restart. Reset to default restores the draft to the original guidance; save again to apply it. Discard changes cancels unsaved edits

The default is: “Save durable new facts, decisions or corrections when useful, without waiting for an explicit request to remember.” Existing settings inherit that text. Editing it does not enable memory, change visibility, or rewrite existing records. The technical tool workflow, evidence requirements, and server-enforced permissions stay separate. This is model guidance, not a guaranteed content filter. It uses the existing configuration record and adds no further migration

Implementation

Existing components reused: memory/configuration tables, authenticated identities and team permissions, repositories, the request processor and router, spend/budget/limiter hooks, DualCache and cross-worker invalidation, APScheduler, and PodLockManager. New code covers memory selection/capture, private tool execution across three protocols, and bounded saved Responses transcripts. LiteLLM already has a shared AgenticLoopPlan framework across Chat, Responses, and Anthropic, including private-tool hold-back streaming. Its current follow-up executors call the SDK (litellm.acompletion / litellm.aresponses) directly. They do not re-enter the gateway admission and accounting path used here. Reusing that framework requires a gateway-aware round executor that preserves per-round RPM, budgets, guardrails, cost completion, and permission checks. This PR still has separate gateway orchestration; that is a consolidation opportunity, not a claim that the framework is missing

Enrollment settings are cached for 30 seconds and invalidated on admin updates. Unenrolled inference requests return before user/team database lookups. Cached enrollment never authorizes a memory operation: eligible requests and memory reads/writes still resolve current permissions. Service-key contributors return actor: null and actor_name: "Service key"; internal ownership identifiers remain private

The model chooses when to call capture, search, or read; only enabled tools are injected. Capture requires an exact evidence quote from an incoming user message or application tool result. There is no forced checkpoint, catalog, nightly work, or mandatory reflection call. Capture is best-effort and can miss facts if the model does not save them

Hidden rounds reuse the proxy request processor, hooks, router, budgets, token limits, concurrency limits, and spend tracking. Each physical model call is billed and logged under the same request trace. Each upstream round consumes RPM, including hidden continuations; requests stop with 429 when a limit is reached. The memory loop permits up to eight rounds; structured-output requests and requests capped below 4,096 output tokens use a separate final-answer call, which also counts against limits. Clients cannot force a private memory tool through tool_choice. Streaming rounds are buffered until private activity is distinguished from the final answer, with keepalives while waiting

With automatic tool choice, short-budget requests give memory preparation a 4,096-token output allowance and restore the caller's exact max_tokens, max_completion_tokens, or max_output_tokens for the final answer. Application tools and output formatting are also restored for that final call. Larger unconstrained requests keep their existing flow. This prevents a 40-token answer limit from truncating private tool arguments. If preparation exhausts its budget, its unfinished response is discarded and the final pass still runs with the caller's cap, using only completed tool results and explicit failure guidance. Outside preparation, incomplete tool responses remain incomplete with store:false. Unmatched private calls and incomplete application calls cannot become resumable history. Previously poisoned histories are rejected with HTTP 409 before a provider call

Storage reuses the existing memory and configuration tables. The only new table holds complete Responses transcripts and public response objects, readable for up to 24 hours with eviction at capacity. Follow-ups replay that authenticated transcript instead of reconstructing it from spend logs, so prompt logging is optional. Tool calls and results remain paired, and branching starts from the selected response. The existing one-megabyte per-response, 256-response per-key, and 32-megabyte per-namespace limits still apply. store:false disables this retention. Public retrieval returns only the public response; private input history remains unavailable

Responses saved before this fix require a fresh conversation (HTTP 409). An existing non-memory provider parent remains subject to its normal provider/session replay behavior

Expired rows are removed by bounded hourly cleanup using the existing scheduler and Redis pod lock; without Redis, each worker performs its own bounded sweep. Expiry checks do not depend on cleanup timing. Turning recall off blocks resuming a saved Responses conversation with previously retrieved memory. Already-returned content in client history cannot be removed; start a fresh conversation after changing recall

Instructions and tool definitions stay stable between admin edits, preserving provider prompt-cache eligibility. Editing capture guidance changes the prompt prefix and can cause an expected cache miss until the new prefix warms. Saved records are fetched on demand, not injected into every system prompt. Whole-response caching is bypassed for memory rounds to avoid replaying side effects. No E2E scripts are committed

With prompt logging enabled, recalled memory is part of the full request log. Someone authorized to view that request can see it there. Direct memory browsing/search still uses memory permissions; this is the accepted logging contract

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

• Meaningful regression tests added
• Focused backend/UI tests and make check pass locally
• Current-tip required CI and coverage pass
• Scope is gateway-managed memory
• Current-tip Greptile confidence is at least 4/5 (5/5 on f0995fc)

Screenshots / Proof of Fix

Real provider calls against isolated PostgreSQL databases: merge base on 127.0.0.1:4019, PR tip on 127.0.0.1:4018. memory-test routes to vertex_ai/claude-sonnet-5; Messages uses that provider directly. No QA scripts or screenshot branches are part of the PR

The save-only configuration is PUT /memory/v2/settings with {"enabled":true,"everyone":true,"read":{"enabled":false}}. Recall-only uses {"enabled":false,"read":{"enabled":true,"everyone":true}}. Custom guidance uses the same endpoint and adds capture_instructions

Before (1ca4579)

Chat Completions

1. Send a request asking the assistant to remember project Lark-80ff6b2d uses port 8359
2. Start a fresh conversation and ask for that project's port: HTTP 200, but the model reports it has no saved memory and cannot recall the port

Anthropic Messages

1. Send a request asking the assistant to remember project Lark-c37c8ac9 uses port 8359
2. Start a fresh conversation and ask for that project's port: HTTP 200, but the model reports it has no saved memory and cannot recall the port

Responses

1. Send a request asking the assistant to remember project Lark-7823dee2 uses port 8359
2. Start a fresh conversation and ask for that project's port: HTTP 200, but the model reports it has no saved memory and cannot recall the port

Memory dashboard and administration

1. Open the memory page: only the existing /v1/memory table is available
2. There are no gateway saving/recall switches or editable capture instructions

After (f0995fc)

Chat Completions

1. With saving on and recall off, send POST /v1/chat/completions with {"model":"memory-test","messages":[{"role":"user","content":"Remember project AdminRecall-2fb672 uses staging port 8764."}],"max_tokens":1200}: HTTP 200; the record appears in /memory/v2/entries

2. Ask a fresh conversation using a sibling key for the port: HTTP 200, UNKNOWN

3. Switch to saving off and recall on, then repeat: HTTP 200, 8764, with streaming both enabled and disabled

4. Set capture guidance to save deployment ports, prefix titles with Deployment note:, and skip food preferences. Turn saving on, then send “Project Guidance-ca3c01be uses staging port 9284. My favorite snack is blueberries.”: HTTP 200; the saved entry has the requested title prefix and port, without the snack preference

5. Set max_tokens:40 and ask to remember project Budget-198f0d34 uses port 8567, then reply exactly ok: HTTP 200, exact ok, memory saved. Both streaming modes pass

Anthropic Messages

1. With saving on and recall off, send POST /v1/messages with {"model":"vertex_ai/claude-sonnet-5","messages":[{"role":"user","content":"Remember project AdminRecall-e67975 uses staging port 8764."}],"max_tokens":1200}: HTTP 200; the record appears in /memory/v2/entries

2. Ask a fresh conversation using a sibling key for the port: HTTP 200, UNKNOWN

3. Switch to saving off and recall on, then repeat: HTTP 200, 8764, with streaming both enabled and disabled

4. Set capture guidance to save deployment ports, prefix titles with Deployment note:, and skip food preferences. Turn saving on, then send “Project Guidance-d3570bf8 uses staging port 9284. My favorite snack is blueberries.”: HTTP 200; the saved entry has the requested title prefix and port, without the snack preference

5. Set max_tokens:40 and ask to remember project Budget-7ef8d177 uses port 8567, then reply exactly ok: HTTP 200, exact ok, memory saved. Both streaming modes pass

Responses

1. With saving on and recall off, send POST /v1/responses with {"model":"memory-test","input":"Remember project AdminRecall-34fbd0 uses staging port 8764.","max_output_tokens":1200,"store":false}: HTTP 200; the record appears in /memory/v2/entries

2. Ask a fresh conversation using a sibling key for the port: HTTP 200, UNKNOWN

3. Switch to saving off and recall on, then repeat: HTTP 200, 8764, with streaming both enabled and disabled

4. Set capture guidance to save deployment ports, prefix titles with Deployment note:, and skip food preferences. Turn saving on, then send “Project Guidance-f61eaad0 uses staging port 9284. My favorite snack is blueberries.”: HTTP 200; the saved entry has the requested title prefix and port, without the snack preference

5. With saving and recall enabled, run curl -sS http://127.0.0.1:4018/v1/responses -H "Authorization: Bearer $ALICE_KEY" -H 'Content-Type: application/json' -d '{"model":"vertex_ai/claude-sonnet-5","input":"Find the staging port for AdminRecall-2fb672 in memory. Return only the port.","store":true,"max_output_tokens":1600}': HTTP 200, output text exactly 8764, ID starting with resp_litellm_memory_

6. Send another POST http://127.0.0.1:4018/v1/responses with that ID as previous_response_id, input: "Repeat your previous answer exactly. Only the port.", and stream:true: HTTP 200, streamed text exactly 8764, completed response text exactly 8764

7. Repeat with prompt logging disabled and enabled, and with native gpt-5.5: all four combinations pass. Continue with a temporary random code and tools disabled, then ask for the preceding answer: exact code recovered. Branch from the first response: 8764, without later branch context

8. Call an application function, submit its function_call_output with previous_response_id, and stream the answer: exact temporary code returned for both models and logging modes. These are 28 real requests with exact-output assertions, not substring matches

9. Start a native OpenAI conversation with memory off, enable memory, and resume twice: the original temporary code survives both transitions. Other keys cannot retrieve its memory-owned response (404), private input_items stays unavailable (501), store:false leaves no retained response (404), and turning recall off blocks resumption (403)

10. Set max_output_tokens:40 and ask to remember project Budget-d76a7728 uses port 8567, then reply exactly ok: HTTP 200, exact ok, memory saved. Both streaming modes pass; the saved previous_response_id also resumes successfully. A 20-worker burst saved all 20 facts, returned 20 exact ok replies, and completed all 20 follow-ups. A poisoned response captured before this fix now returns 409 instead of OpenAI's permanent missing-tool-output 400

11. Ask to capture eight long, exact public label sequences with max_output_tokens:512, exhausting the 4,096-token preparation allowance. The final pass still returns HTTP 200 with the requested failure marker FAILED, and a follow-up also returns FAILED. Inspection confirmed two model calls and no retained partial function calls

Memory dashboard and administration

1. Open /ui/memory/: the table shows recent memories, contributor, team, search, and separate saving/recall status
2. Open Administration: independent saving/recall controls offer Everyone or Selected users
3. Edit What should memory remember?, save, and reload: the text persists. A non-admin update returns 403; blank guidance returns 422
4. Change the guidance to architectural decisions with an Architecture note: title prefix: a new real model request saves the PostgreSQL decision with that prefix without restarting the gateway
5. Click Reset to default: the draft changes. Click Save changes and reload: the original guidance is restored

Additional current-tip checks: recall-only requests did not create memories; turning recall off blocked a stored Responses continuation with HTTP 403. Both controls off returned an ordinary greeting with 8 input tokens. Local validation passed: 375 memory/Responses-authorization tests (2 skipped) and make check. Dashboard source is unchanged in this token-budget fix; existing screenshots show the same layout

A stable provider prefix still produced 12,619 cache-read input tokens across gateway rounds after adding a saved memory. This enrollment cache is separate from provider prompt caching

Search currently scans every visible record in 128-row pages, retaining only the best results in memory. That bounds RAM, not total database and CPU work. A SQL candidate search with indexed fuzzy matching would improve large stores. Enrollment caching protects unenrolled traffic; enabled operations still read fresh permissions before and after access to avoid stale grants. Replacing those reads with cached user/team helpers requires reliable revocation invalidation

The primary pin deliberately fails closed instead of falling back to a potentially stale replica for permissions. prisma_transaction is a typed adapter over db.tx(), not another transaction implementation. V1 and V2 share the existing table, with V2 namespaces separating their rows and visibility rules. The two spend-writer P2028 errors in the reported load run remain unattributed; this patch does not change pool settings or claim to fix them

Type

New Feature

Caveats

Severe

• Enabling memory adds model calls, latency, and spend
• Database migration adds columns, indexes, and a continuation table
• Enable V2 after every gateway replica has been upgraded
• Full request logs include recalled memory when prompt logging is enabled
• Stored Responses retain transcripts independently of prompt logging

Medium

• Short output caps require an extra final-answer call
• Final output caps include reasoning and can still truncate
• Search scans visible rows in bounded Python pages
• Enabled memory operations recheck permissions against the primary database
• Pre-fix saved Responses require a new conversation
• Streaming waits until private tool activity is classified
• Capture guidance is best-effort, not guaranteed content filtering
• Memory text remains untrusted; prompt-injection resistance is not guaranteed
• Changing recall requires fresh Responses conversations
• Custom proxy model aliases need accurate local pricing
• Interactive headful coding-tool sessions remain unverified

Low

• Current-tip Greptile review is 5/5; Veria review is pending, and a fresh Bugbot review has been requested but is not yet confirmed

Final Attestation

• Final attestation pending current-tip CI and automated review results

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_memory_active_tools (c530e50) with main (24153b5)}

[codecov]

Codecov Report

❌ Patch coverage is 91.17647% with 129 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/memory/gateway.py	82.28%	45 Missing ⚠️
litellm/proxy/common_request_processing.py	31.25%	22 Missing ⚠️
litellm/proxy/memory/knowledge.py	84.74%	9 Missing ⚠️
..._core_utils/prompt_templates/server_tool_stream.py	95.65%	8 Missing ⚠️
litellm/proxy/memory/responses.py	84.90%	8 Missing ⚠️
litellm/proxy/memory/store.py	95.39%	7 Missing ⚠️
...re_utils/prompt_templates/server_tool_responses.py	93.24%	5 Missing ⚠️
litellm/proxy/memory/management.py	95.00%	4 Missing ⚠️
litellm/proxy/memory/transport.py	96.22%	4 Missing ⚠️
...itellm_core_utils/prompt_templates/server_tools.py	96.00%	3 Missing ⚠️
... and 9 more

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR adds gateway-managed memory with independently administered saving and recall, permission-aware memory operations, private tool orchestration across Chat Completions, Responses, and Anthropic Messages, bounded Responses continuations, and dashboard controls. The latest revision fixes preparation exhaustion by discarding truncated private calls, restoring the caller’s final-answer constraints, and persisting only resumable Responses history

• Truncated preparation now always proceeds to a separately capped final answer
• Partial private calls are removed before final-answer generation and continuation persistence
• Completed earlier tool calls and results remain paired in retained Responses transcripts
• Regression coverage exercises all three protocols in streaming and non-streaming modes

Confidence Score: 5/5

The current revision appears safe to merge, with the previous preparation-exhaustion failure fixed and no new actionable issue established

Truncated preparation now bypasses partial tool execution, restores the client’s exact output constraints for a final model call, and persists a completed Responses result without unmatched private calls. The earlier duplicate-directive finding was manually resolved without explanation; all other previous findings were fixed, withdrawn, accepted as intentional behavior, or resolved with supporting context

Important Files Changed

Filename	Overview
litellm/proxy/memory/gateway.py	Detects truncated preparation rounds, discards their incomplete state, and always executes the restored final-answer phase
litellm/litellm_core_utils/prompt_templates/server_tool_responses.py	Centralizes detection of incomplete Responses, Anthropic, and Chat Completions outputs
tests/test_litellm/proxy/memory/test_memory_v2_boundaries.py	Adds cross-protocol regression coverage for preparation exhaustion, partial-call removal, restored limits, and resumable Responses output

_{Reviews (41): Last reviewed commit: "fix(memory): finish replies after prepar..." | Re-trigger Greptile}

[veria-ai]

PR overview

This PR adds gateway-backed memory with separate controls for administrators to save memories and for callers to recall them. Recalled results can be incorporated into subsequent model requests through server-tool continuation.

Seven issues have been addressed, with one conditional data-exposure issue remaining. When prompt logging is enabled, recalled memory content may appear in spend logs, allowing team members with spend-log access but without memory-read permission to view it. The issue depends on a specific logging configuration and permission combination.

Open issues (1)

• Low: Recalled memories bypass memory-read permissions via spend logs — litellm/proxy/memory/gateway.py:329

Fixed/addressed: 7 · PR risk: 4/10

[moe-berri]

@greptileai @veria-ai @cursor review Please review 610a5a4, especially per-round RPM enforcement, JSON updates, error preservation, and disabled-path isolation

[cursor]

Stale Bugbot comment from a previous run.

[veria-ai]

Low: Recalled memories bypass memory-read permissions via spend logs

continue_server_tools embeds full memory results in the next model request, which follows the normal spend-logging path and is attributed to the caller's team. When prompt logging is enabled, a team member with /spend/logs but without /memory/v2/entries can read memory content they are not authorized to access; keep these private tool results out of logging snapshots, or replace them with a redacted logging-only representation while retaining billing metadata.

[moe-berri]

This is intended. Access to a user's full request logs includes recalled memories within those requests. Separate memory permissions govern direct browsing and search

[moe-berri]

@greptileai

[moe-berri]

bugbot run

[moe-berri]

@veria-ai "Low: Recalled memories bypass memory-read permissions via spend logs" - this is intended - all recorded memories are already available as part of logs - memory read permissions are for agentic read/write

[cursor]

Stale Bugbot comment from a previous run.

[moe-berri]

@greptileai @veria-ai @cursor review Please review the current changes, including admin capture guidance, permission enforcement, default behavior, and prompt cache stability

[moe-berri]

@cursor review Please check the final provider-field fix and capture guidance. @greptileai @veria-ai please refresh your reviews for this revision

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 13d0a57. Configure here.}

[moe-berri]

@greptileai @veria-ai @cursor review Please re-review the latest commit covering enrollment caching, coordinated continuation cleanup, and service-key identifier redaction

[moe-berri]

@greptileai @veria-ai @cursor review Please review 396b18b, including the cleanup lock expiry margin and regression covering the next scheduled run

[moe-berri]

@greptileai @veria-ai @cursor review Please review c530e50: saved Responses now replay complete authenticated transcripts independently of spend logs

[moe-berri]

@greptileai @veria-ai @cursor review Please review 06860d0 for isolated memory tool budgets, exact final-answer caps, and rejection of incomplete continuations

[moe-berri]

@greptileai @veria-ai @cursor review Please review f0995fc, including preparation exhaustion, partial-call removal, restored final-answer limits, and safe Responses replay