feat: add Phase 5D memory scoring and pruning - #111
Conversation
Phase 5D adds deterministic, model-free memory usefulness scoring, review-first stale-memory pruning, and confidence-weighted retrieval ranking. Safe-by-default and review-first: scoring/pruning are gated off by default, pruning never hard-deletes, and namespaces stay isolated (AU never contaminates real_life). - Scoring service: usefulness/freshness/feedback/contradiction factors combine into a clamped 0..1 retrieval_weight. Secrets are always blocked to zero and never retained. Durable preferences outrank transient logs; current production facts outrank stale ones. - Pruning service: produces keep/stale/superseded/low_value/unsafe/duplicate recommendations (review-only) plus a non-destructive maintenance runner. - Retrieval: recent events are ranked by retrieval_weight with null-safe fallback; semantic/embeddings/model gates remain disabled and deterministic retrieval stays primary. - Candidate review UI shows plain usefulness/freshness labels. - Additive migration: nullable score columns on memory_events/candidates plus a review-only memory_pruning_candidates table. - Optional gates (scoring/pruning/mode/version) are catalogued as managed but never required provider envs, so they cannot reintroduce the RED drift false-positive. - Admin status route and internal-token maintenance job (dry-run by default). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013izDrMYfyugjRhhNpL3dJW
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
More reviews will be available in 42 minutes and 55 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
📝 WalkthroughWalkthroughIntroduces Phase 5D memory lifecycle management: a deterministic usefulness scoring engine, review-first pruning recommendation service, additive database migration, two new API routes (maintenance job and admin status), env-broker registration for Phase 5D flags, hybrid retrieval ranking by retrieval weight, and admin UI display of scoring fields. ChangesPhase 5D Memory Usefulness Scoring & Review-First Pruning
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4b387e2feb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| // Safe-by-default: any run without an explicit dryRun:false stays a non-destructive dry run. | ||
| const dryRun = parsed.data.dryRun ?? parsed.data.dry_run ?? true; | ||
| const user_id = parsed.data.user_id ?? process.env.PANDORA_MEMORY_BRIDGE_USER_ID; |
There was a problem hiding this comment.
Because this route uses the bridge admin client below, accepting parsed.data.user_id lets any holder of PANDORA_INTERNAL_JOB_TOKEN choose an arbitrary account and, with dryRun:false, update that user's memory_events and insert pruning candidates. This bypasses the server/RLS-derived identity requirement for memory writes; restrict the job to the configured bridge user or derive the user from an authenticated operator/session instead of the request body.
Useful? React with 👍 / 👎.
| const upd = await client | ||
| .from("memory_events") | ||
| .update({ usefulness_score: breakdown.usefulness_score, confidence_score: breakdown.confidence_score, freshness_score: breakdown.freshness_score, contradiction_score: breakdown.contradiction_score, retrieval_weight: breakdown.retrieval_weight, stale_status: stale, scoring_version: config.scoringVersion, scored_at: new Date(now).toISOString() }) |
There was a problem hiding this comment.
When dryRun:false and PANDORA_ENABLE_MEMORY_USEFULNESS_SCORING=true, this directly changes persisted memory_events metadata (retrieval_weight, stale_status, scored_at, etc.) but the maintenance path never appends corresponding memory_patches or audit_logs records. The memory browser/proof flow relies on those patch/audit rows to explain persisted changes, so Phase 5D score writes become untraceable once the gate is enabled.
Useful? React with 👍 / 👎.
| .from("memory_pruning_candidates") | ||
| .insert({ user_id: scope.user_id, namespace: scope.namespace, memory_id: candidate.memory_id, pruning_category: candidate.category, recommendation: candidate.recommendation, reason: candidate.reason, stale_status: candidate.stale_status, retrieval_weight: candidate.retrieval_weight, superseded_by_memory_id: candidate.superseded_by_memory_id ?? null, scoring_version: config.scoringVersion, status: "open" }) |
There was a problem hiding this comment.
When pruning is enabled, each non-dry-run maintenance call executes a blind insert for every candidate, and the migration adds no unique constraint/upsert to collapse (user_id, namespace, memory_id, pruning_category) duplicates. Re-running the job after a retry or schedule will create multiple open review rows for the same stale/unsafe memory, so the review queue is no longer idempotent.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/api/admin/memory/phase-5d/status/route.ts`:
- Around line 23-36: The status aggregation in the route handler is only
counting stale, superseded, and low_value candidates, so it drops unsafe and
duplicate pruning categories from the admin response. Update the totals building
logic in the memory phase-5d status route to include every category emitted by
the pruning service, especially unsafe and duplicate, and make sure the response
object and any related type definitions in this handler reflect the added
counts.
- Around line 37-39: The scoring summary logic in the status route is only
checking a single row, so `totals.scored` is not an actual count and
`lastScoringRun` may come from a null value. Update the query in the status
handler to fetch the latest non-null `scored_at` separately for
`lastScoringRun`, and use a count-based query for scored rows in the same
`route.ts` flow so `totals.scored` reflects the exact number of scored events.
In `@lib/services/memory-hybrid-retrieval-service.ts`:
- Around line 6-10: The recent events query in memory-hybrid-retrieval-service
is not selecting the fields used by rankMemoriesByRetrievalWeight, so ranking
falls back to incomplete scores. Update the memory_events select in the hybrid
retrieval flow to include the Phase 5D ranker inputs it reads, especially
retrieval_weight, confidence, freshness timestamps, feedback counts, and
contradiction markers, so recent_events are ordered using the intended weights
instead of defaults.
In `@lib/services/memory-pruning-service.ts`:
- Around line 246-250: The insert path in memory-pruning-service.ts is creating
duplicate open rows for the same memory/category on repeated runs. Update the
logic around the client.from("memory_pruning_candidates").insert(...) flow to
first check for an existing open candidate for the same user_id, namespace,
memory_id, and pruning_category, or switch to a uniqueness-backed upsert so only
one open row exists per candidate. Keep the fix localized to the pruning
candidate creation path and preserve the existing select("*").single() behavior
for the inserted/updated row.
- Around line 79-81: The superseding check in memory-pruning-service’s
sameNamespacePeers.find currently compares created_at as strings, which can let
null or invalid timestamps win over real ones. Update the logic in the newer
selection so it only considers peers with valid created_at values and compares
actual timestamps (or parsed dates) before treating a production_fact as newer
than memory. Keep the change localized to the find predicate and the created_at
handling around classifyContent and projectSubject.
In `@lib/services/memory-usefulness-scoring-service.ts`:
- Around line 75-81: Move the structured `memory_type` checks in
`classifyContent` ahead of the blank-text early return so empty text still
respects typed memories. In `memory-usefulness-scoring-service.ts`, make sure
`secret_or_credential` is classified as `"secret"` and other typed categories
are evaluated before `raw.trim()` can return `"general"`, so `attachScore()` in
`admin-memory-candidate-loader.ts` will correctly keep blocked secrets at zero
usefulness.
In `@supabase/migrations/20260629000000_phase_5d_memory_scoring.sql`:
- Line 44: The `memory_pruning_candidates.stale_status` column is missing the
same restricted `stale_status` constraint used elsewhere, so update the Phase 5D
migration to enforce the app’s `StaleStatus` contract instead of allowing
free-form text. Apply the same constraint pattern already used for the other
Phase 5D columns in this migration, and keep the change localized to the
`memory_pruning_candidates` table definition so the schema stays consistent.
In `@tests/phase5d-memory-scoring.test.ts`:
- Around line 227-243: The mock cleanup in the phase-5d maintenance test is not
guaranteed to run if the import or assertions fail, leaving
`PANDORA_INTERNAL_JOB_TOKEN`, `vi.doMock`, and `@/lib/supabase/bridge-admin`
state polluted for later tests. Wrap the test body around the `route.POST`
import/assertions in a try/finally and move the env restore,
`vi.doUnmock("`@/lib/supabase/bridge-admin`")`, and `vi.resetModules()` into the
finally block so teardown always runs.
- Around line 155-163: The test is mutating process.env directly and restoring
it inline, which can leak state if an assertion throws and can leave absent keys
present after cleanup. Update the affected test in getHybridMemoryContext to
save whether each PANDORA_ENABLE_* key existed, restore the previous values
inside a finally block, and delete keys that were originally absent; apply the
same cleanup pattern to the other process.env.PANDORA_INTERNAL_JOB_TOKEN
mutations in the suite.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 7b269a28-ff66-481e-a65e-64a6e4039c24
📒 Files selected for processing (12)
app/admin/memory/candidates/page.tsxapp/api/admin/memory/phase-5d/status/route.tsapp/api/memory/jobs/phase-5d-maintenance/route.tslib/config/phase-5d-config.tslib/services/admin-memory-candidate-loader.tslib/services/env-discovery-service.tslib/services/env-validation-service.tslib/services/memory-hybrid-retrieval-service.tslib/services/memory-pruning-service.tslib/services/memory-usefulness-scoring-service.tssupabase/migrations/20260629000000_phase_5d_memory_scoring.sqltests/phase5d-memory-scoring.test.ts
- Maintenance job derives the user from server config only (no body user_id), so a job-token holder can never target another account's memory writes (P1). - Score writes append an audit_logs trail for traceability (memory_patches is FK-bound to memory_items, so audit_logs is the correct table for memory_events) (P1). - Pruning candidate inserts are idempotent: existence check + partial unique index on open (user, namespace, memory_id, category) so re-runs don't inflate the queue. - classifyContent checks structured memory_type before the blank-text fast path, keeping redacted blocked_secret candidates at zero usefulness. - Supersede check parses timestamps instead of string-comparing created_at. - Hybrid retrieval selects the score fields the ranker reads. - Status route reports all pruning categories (incl. unsafe/duplicate) and an exact scored count from non-null scored_at. - Migration: stale_status CHECK constraint + partial unique index on open candidates. - Tests: env/mocks teardown moved into finally; added an idempotency test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013izDrMYfyugjRhhNpL3dJW
2 participants
1. Phase 5D summary
Phase 5D adds deterministic, model-free memory usefulness scoring, review-first stale-memory pruning, and confidence-weighted retrieval ranking. It is safe-by-default and review-first:
retrieval_weight = usefulness*0.40 + confidence*0.25 + freshness*0.20 + feedback*0.10 − contradiction*0.20, clamped to 0..1. Per-category freshness half-lives mean durable preferences decay slowly while operational/transient details decay fast — so current production facts outrank stale ones and durable preferences outrank transient logs.keep / stale / superseded / low_value / unsafe / duplicaterecommendations and never hard-deletes. High-confidence durable memories are protected; newer same-subject verified facts supersede older ones.retrieval_weightwith a null-safe fallback for unscored rows. Deterministic structured retrieval stays primary; semantic/embeddings/model gates remain disabled.real_lifescoring, pruning, or retrieval.2. Files changed (12, +1031 / −5)
New:
lib/services/memory-usefulness-scoring-service.ts— deterministic scoring (scoreMemoryUsefulness,scoreFreshness,scoreFeedback,scoreContradictionRisk,computeRetrievalWeight,classifyStaleness, labels + ranking helpers)lib/services/memory-pruning-service.ts—evaluatePruningRecommendation,findPruningCandidates,runPhase5dMaintenancelib/config/phase-5d-config.ts— optional gate resolution + gates summaryapp/api/memory/jobs/phase-5d-maintenance/route.ts— internal-job-token maintenance job (dry-run by default)app/api/admin/memory/phase-5d/status/route.ts— operator-protected status routesupabase/migrations/20260629000000_phase_5d_memory_scoring.sql— additive migrationtests/phase5d-memory-scoring.test.ts— 19 testsEdited (additive, non-breaking):
lib/services/memory-hybrid-retrieval-service.ts— confidence-weighted ranking of recent eventslib/services/admin-memory-candidate-loader.ts+app/admin/memory/candidates/page.tsx— usefulness/freshness labels in candidate reviewlib/services/env-discovery-service.ts+lib/services/env-validation-service.ts— catalog Phase 5D gates as managed/optional and classify_MODE/_VERSIONconfig keys3. Migration filename
supabase/migrations/20260629000000_phase_5d_memory_scoring.sql— fully additive (nullable score columns onmemory_eventsandmemory_capture_candidates; new review-onlymemory_pruning_candidatestable with user-scoped RLS and namespace check). No drops, no deletes.4. Test results
npm run typecheck✓ (exit 0) ·npm run lint✓ (0 errors) ·npm run build✓ (both routes registered) ·npm run env:policy✓ (89 keys).5. Production deploy required?
Not from this PR by itself — no runtime behavior changes until the migration is applied and the optional scoring gate is enabled. A redeploy is part of the post-merge steps below so the new routes and migration take effect. Default behavior writes nothing and mutates no memory.
6. Exact post-merge steps
main.20260629000000_phase_5d_memory_scoring.sql.main; wait for READY).real_life:POST /api/memory/jobs/phase-5d-maintenancewith the internal job token, body{ "dryRun": true, "namespace": "real_life" }.au:POST /api/memory/jobs/phase-5d-maintenancewith the internal job token, body{ "dryRun": true, "namespace": "au" }.PANDORA_ENABLE_MEMORY_USEFULNESS_SCORING=true(the Env Broker stays GREEN — this flag is optional, not a required provider env).PANDORA_ENABLE_MEMORY_PRUNINGunset/false andPANDORA_MEMORY_PRUNING_MODE=review_onlyunless pruning is explicitly approved.Guardrails for this rollout: do not enable pruning by default, do not mutate production memory yet (dry-run only), and do not print or expose secrets (the job uses the internal token from env only and returns redacted summaries with
rawTokenReturned-style safety; no raw values are rendered).Checklist
PANDORA_ENABLE_MEMORY_USEFULNESS_SCORING,PANDORA_ENABLE_MEMORY_PRUNING,PANDORA_MEMORY_PRUNING_MODE,PANDORA_MEMORY_SCORING_VERSION).NEXT_PUBLIC_values are added or modified.🤖 Generated with Claude Code
Generated by Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Chores