Skip to content

Add Phase 5C Daily Compaction and Profile Synthesis #107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
besfeng23 merged 2 commits into from
Jun 28, 2026
Merged

Add Phase 5C Daily Compaction and Profile Synthesis #107

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ffaa19d
Add Phase 5C daily compaction
besfeng23 Jun 28, 2026
7fc579b
fix: prevent generic compaction signals from polluting named profiles
besfeng23 Jun 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions app/admin/memory/compaction/page.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
import { resolvePandoraRuntimeSafetyConfig } from "@/lib/config/pandora-runtime-safety-config";
export default function MemoryCompactionAdminPage(){const r=resolvePandoraRuntimeSafetyConfig(); const safe=[
Comment on lines +1 to +2

@coderabbitai coderabbitai Bot Jun 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

Next.js 15 App Router page static rendering force-dynamic route segment config

💡 Result:

In Next.js 15, the dynamic route segment configuration option remains the primary mechanism to explicitly control the rendering mode of a page or layout [1][2]. To force a route to be rendered dynamically (on every request), you can export the following from your page.tsx, layout.tsx, or Route Handler: export const dynamic = 'force-dynamic'; Effects of 'force-dynamic': - It forces the entire route segment to be rendered at request time [2][3]. - It is equivalent to setting all fetch requests within that segment to { cache: 'no-store' } [2][4]. - It ensures the route is not pre-rendered at build time [5]. Distinction between Rendering and Caching: It is important to distinguish between server-side rendering (the generation of the page) and the Next.js Router Cache (a client-side cache) [4][6]. While export const dynamic = 'force-dynamic' forces the page to be rendered dynamically on the server for each request, the client-side Router Cache may still store the resulting page navigation to improve perceived performance [7][6]. If you observe that a page seems to be "cached" despite using force-dynamic, you may be encountering the client-side Router Cache [6]. You can control the duration of this cache in your next.config.js via the experimental.staleTimes configuration [7]. Additionally, in Next.js 15, you can opt into dynamic behavior at a more granular level by invoking dynamic APIs [8][3]. The connection() function from next/server can be used within an asynchronous function to signal that the execution should be dynamic, ensuring the route is rendered at request time [9]. Note: The dynamic configuration takes precedence over individual fetch options within the same segment [4]. If you need a more precise approach, prefer using { cache: 'no-store' } on specific fetch calls rather than forcing the entire route to be dynamic [4].

Citations:

Force this admin page to render dynamically. Add export const dynamic = "force-dynamic"; so the runtime safety gates and since timestamp aren’t frozen at build time.

Proposed fix 
 import { resolvePandoraRuntimeSafetyConfig } from "`@/lib/config/pandora-runtime-safety-config`";
+
+export const dynamic = "force-dynamic";
+
 export default function MemoryCompactionAdminPage(){const r=resolvePandoraRuntimeSafetyConfig(); const safe=[
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
import { resolvePandoraRuntimeSafetyConfig } from "@/lib/config/pandora-runtime-safety-config";
export default function MemoryCompactionAdminPage(){const r=resolvePandoraRuntimeSafetyConfig(); const safe=[
import { resolvePandoraRuntimeSafetyConfig } from "`@/lib/config/pandora-runtime-safety-config`";
export const dynamic = "force-dynamic";
export default function MemoryCompactionAdminPage(){const r=resolvePandoraRuntimeSafetyConfig(); const safe=[
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/admin/memory/compaction/page.tsx` around lines 1 - 2, The
MemoryCompactionAdminPage is currently rendered statically, which can freeze the
runtime safety gate data and the since timestamp at build time. Add a dynamic
rendering export alongside the existing MemoryCompactionAdminPage component so
this admin page always evaluates resolvePandoraRuntimeSafetyConfig at request
time and keeps its runtime checks current.

["Distillation gate",r.config.memoryDistillationEnabled],
["Model calls",r.config.modelCallsEnabled],
["Embeddings",r.config.embeddingsEnabled],
["Semantic retrieval",r.config.semanticRetrievalEnabled],
["Public memory read",r.config.publicMemoryReadEnabled],
["Public persistence",r.config.publicMemoryPersistenceEnabled],
]; return <main style={{padding:24,maxWidth:900}}><h1>Phase 5C Memory Compaction</h1><p>Daily compaction deterministically turns reviewed/captured memory into versioned profiles, open loops, and daily context packs. It does not call models, create embeddings, enable retrieval, or expose public reads/writes.</p><h2>Safety status</h2><ul>{safe.map(([label,value])=><li key={String(label)}><strong>{label}:</strong> {value?"enabled":"disabled"}</li>)}</ul><h2>Internal job</h2><p>POST <code>/api/memory/jobs/daily-digest</code> with <code>Authorization: Bearer $PANDORA_INTERNAL_JOB_TOKEN</code>.</p><pre>{JSON.stringify({namespace:"real_life",dry_run:true,since:new Date(Date.now()-86400000).toISOString()},null,2)}</pre><h2>Operator notes</h2><ul><li>Keep model calls, embeddings, semantic retrieval, public read/write, and permanent auto-capture off.</li><li>Run dry runs first, inspect profiles/open loops/context pack output, then run per namespace.</li></ul></main>}
8 changes: 8 additions & 0 deletions app/api/memory/jobs/daily-digest/route.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import { NextRequest, NextResponse } from "next/server";
import { z } from "zod";
import { resolvePandoraRuntimeSafetyConfig } from "@/lib/config/pandora-runtime-safety-config";
import { createSupabaseBridgeAdminClient } from "@/lib/supabase/bridge-admin";
import { runDailyMemoryCompaction } from "@/lib/services/memory-compaction-service";
const schema=z.object({namespace:z.enum(["real_life","au"]),since:z.string().datetime().optional(),dry_run:z.boolean().optional(),user_id:z.string().min(1).optional()});

@coderabbitai coderabbitai Bot Jun 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

Zod 3.23.8 z.string().datetime offset true default

💡 Result:

In Zod 3.23.8, the default value for the offset option in z.string.datetime({ offset:... }) is false [1][2][3]. When calling z.string.datetime, Zod enforces ISO 8601 strings and, by default, strictly requires UTC format (usually ending in "Z") and prohibits timezone offsets [1][2][3]. Setting { offset: true } explicitly enables the validation to accept strings that include timezone offsets (e.g., +02:00) [1][2][3]. Summary of datetime validation behavior: - Default: No timezone offsets are allowed [1][3]. - Enabled: { offset: true } allows timezone offsets [1][2]. - Precision: Arbitrary sub-second decimal precision is supported by default [1][3]. - Unqualified/Local: To allow datetimes without a timezone or "Z" designator, you must use the { local: true } option, which was added in Zod 3.23.0 [4][5][6].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate Zod version declarations and the target route
git ls-files | rg '(^|/)(package\.json|pnpm-lock\.yaml|package-lock\.json|yarn\.lock|bun\.lockb|app/api/memory/jobs/daily-digest/route\.ts)$'

printf '\n--- package.json mentions ---\n'
rg -n '"zod"|zod@' package.json **/package.json 2>/dev/null || true

printf '\n--- lockfile mentions ---\n'
rg -n 'zod' pnpm-lock.yaml package-lock.json yarn.lock bun.lockb 2>/dev/null || true

printf '\n--- route file ---\n'
cat -n app/api/memory/jobs/daily-digest/route.ts

Repository: besfeng23/Memory

Length of output: 3179

Allow ISO since values with offsets
z.string().datetime() rejects offset timestamps by default, so inputs like 2026-06-28T10:00:00-07:00 fail here. If this field is meant to accept ISO datetimes, switch to datetime({ offset: true }).

Proposed fix 
-const schema=z.object({namespace:z.enum(["real_life","au"]),since:z.string().datetime().optional(),dry_run:z.boolean().optional(),user_id:z.string().min(1).optional()});
+const schema=z.object({namespace:z.enum(["real_life","au"]),since:z.string().datetime({offset:true}).optional(),dry_run:z.boolean().optional(),user_id:z.string().min(1).optional()});
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const schema=z.object({namespace:z.enum(["real_life","au"]),since:z.string().datetime().optional(),dry_run:z.boolean().optional(),user_id:z.string().min(1).optional()});
const schema=z.object({namespace:z.enum(["real_life","au"]),since:z.string().datetime({offset:true}).optional(),dry_run:z.boolean().optional(),user_id:z.string().min(1).optional()});
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/api/memory/jobs/daily-digest/route.ts` at line 6, The daily digest input
schema rejects ISO timestamps with timezone offsets because the `schema`
definition uses `z.string().datetime()` for `since`. Update the `since`
validator in the `route.ts` schema to accept offset-aware ISO datetimes by using
the datetime option with offsets enabled, keeping the existing optional behavior
intact.

function bearer(request:NextRequest){const [scheme,token]=(request.headers.get("authorization")??"").split(" "); return scheme?.toLowerCase()==="bearer"?token:undefined;}
export async function POST(request:NextRequest){const token=process.env.PANDORA_INTERNAL_JOB_TOKEN; if(!token||bearer(request)!==token)return NextResponse.json({ok:false,blockers:["internal_job_token_required"]},{status:401}); const runtime=resolvePandoraRuntimeSafetyConfig(); if(!runtime.config.memoryDistillationEnabled)return NextResponse.json({ok:false,blockers:["memoryDistillationEnabled_disabled"],warnings:[],next_step:"Set PANDORA_ENABLE_MEMORY_DISTILLATION=true in a reviewed environment."},{status:403}); const parsed=schema.safeParse(await request.json().catch(()=>({}))); if(!parsed.success)return NextResponse.json({ok:false,blockers:["invalid_request"],issues:parsed.error.flatten()},{status:400}); const user_id=parsed.data.user_id??process.env.PANDORA_MEMORY_BRIDGE_USER_ID; if(!user_id)return NextResponse.json({ok:false,blockers:["job_user_id_required"],next_step:"Send user_id with the internal token or configure PANDORA_MEMORY_BRIDGE_USER_ID."},{status:400}); const result=await runDailyMemoryCompaction(createSupabaseBridgeAdminClient() as never,{user_id,namespace:parsed.data.namespace,since:parsed.data.since,dry_run:parsed.data.dry_run}); return NextResponse.json({ok:result.ok,namespace:result.namespace,dry_run:result.dry_run,daily_pack:result.daily_pack,updated_profiles:result.updated_profiles,opened_or_updated_loops:result.opened_or_updated_loops,warnings:result.warnings,blockers:result.blockers,next_step:result.next_step});}

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Do not accept compaction user_id from the request body

When the internal job token is accepted, this route falls back to parsed.data.user_id and then uses the bridge admin client, so any caller with that token can choose another user's id and write profiles, open loops, and context packs for that user. That bypasses the AGENTS.md requirement that memory writes be tied to server-derived or RLS-derived identity; use only a server-side configured user/session mapping instead of a request-body user_id.

Useful? React with 👍 / 👎.

@coderabbitai coderabbitai Bot Jun 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Return a failure HTTP status when compaction fails.

Line 8 always returns default HTTP 200, even when runDailyMemoryCompaction returns ok:false with blockers such as compaction_failed. Job schedulers and alerts will treat failed compactions as successful.

Proposed fix 
- const result=await runDailyMemoryCompaction(createSupabaseBridgeAdminClient() as never,{user_id,namespace:parsed.data.namespace,since:parsed.data.since,dry_run:parsed.data.dry_run}); return NextResponse.json({ok:result.ok,namespace:result.namespace,dry_run:result.dry_run,daily_pack:result.daily_pack,updated_profiles:result.updated_profiles,opened_or_updated_loops:result.opened_or_updated_loops,warnings:result.warnings,blockers:result.blockers,next_step:result.next_step});}
+ const result=await runDailyMemoryCompaction(createSupabaseBridgeAdminClient() as never,{user_id,namespace:parsed.data.namespace,since:parsed.data.since,dry_run:parsed.data.dry_run}); return NextResponse.json({ok:result.ok,namespace:result.namespace,dry_run:result.dry_run,daily_pack:result.daily_pack,updated_profiles:result.updated_profiles,opened_or_updated_loops:result.opened_or_updated_loops,warnings:result.warnings,blockers:result.blockers,next_step:result.next_step},{status:result.ok?200:500});}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
export async function POST(request:NextRequest){const token=process.env.PANDORA_INTERNAL_JOB_TOKEN; if(!token||bearer(request)!==token)return NextResponse.json({ok:false,blockers:["internal_job_token_required"]},{status:401}); const runtime=resolvePandoraRuntimeSafetyConfig(); if(!runtime.config.memoryDistillationEnabled)return NextResponse.json({ok:false,blockers:["memoryDistillationEnabled_disabled"],warnings:[],next_step:"Set PANDORA_ENABLE_MEMORY_DISTILLATION=true in a reviewed environment."},{status:403}); const parsed=schema.safeParse(await request.json().catch(()=>({}))); if(!parsed.success)return NextResponse.json({ok:false,blockers:["invalid_request"],issues:parsed.error.flatten()},{status:400}); const user_id=parsed.data.user_id??process.env.PANDORA_MEMORY_BRIDGE_USER_ID; if(!user_id)return NextResponse.json({ok:false,blockers:["job_user_id_required"],next_step:"Send user_id with the internal token or configure PANDORA_MEMORY_BRIDGE_USER_ID."},{status:400}); const result=await runDailyMemoryCompaction(createSupabaseBridgeAdminClient() as never,{user_id,namespace:parsed.data.namespace,since:parsed.data.since,dry_run:parsed.data.dry_run}); return NextResponse.json({ok:result.ok,namespace:result.namespace,dry_run:result.dry_run,daily_pack:result.daily_pack,updated_profiles:result.updated_profiles,opened_or_updated_loops:result.opened_or_updated_loops,warnings:result.warnings,blockers:result.blockers,next_step:result.next_step});}
export async function POST(request:NextRequest){const token=process.env.PANDORA_INTERNAL_JOB_TOKEN; if(!token||bearer(request)!==token)return NextResponse.json({ok:false,blockers:["internal_job_token_required"]},{status:401}); const runtime=resolvePandoraRuntimeSafetyConfig(); if(!runtime.config.memoryDistillationEnabled)return NextResponse.json({ok:false,blockers:["memoryDistillationEnabled_disabled"],warnings:[],next_step:"Set PANDORA_ENABLE_MEMORY_DISTILLATION=true in a reviewed environment."},{status:403}); const parsed=schema.safeParse(await request.json().catch(()=>({}))); if(!parsed.success)return NextResponse.json({ok:false,blockers:["invalid_request"],issues:parsed.error.flatten()},{status:400}); const user_id=parsed.data.user_id??process.env.PANDORA_MEMORY_BRIDGE_USER_ID; if(!user_id)return NextResponse.json({ok:false,blockers:["job_user_id_required"],next_step:"Send user_id with the internal token or configure PANDORA_MEMORY_BRIDGE_USER_ID."},{status:400}); const result=await runDailyMemoryCompaction(createSupabaseBridgeAdminClient() as never,{user_id,namespace:parsed.data.namespace,since:parsed.data.since,dry_run:parsed.data.dry_run}); return NextResponse.json({ok:result.ok,namespace:result.namespace,dry_run:result.dry_run,daily_pack:result.daily_pack,updated_profiles:result.updated_profiles,opened_or_updated_loops:result.opened_or_updated_loops,warnings:result.warnings,blockers:result.blockers,next_step:result.next_step},{status:result.ok?200:500});}
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/api/memory/jobs/daily-digest/route.ts` at line 8, The POST handler in
daily-digest/route.ts always responds with HTTP 200 even when
runDailyMemoryCompaction returns ok:false, so update the final NextResponse.json
call to set an error status whenever the result indicates failure. Use the
existing runDailyMemoryCompaction result fields (ok, blockers, next_step) to
choose a non-2xx status for failed compactions while keeping success responses
unchanged.

19 changes: 19 additions & 0 deletions docs/pandora-memory-autopilot.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,22 @@ Phase 5A is deterministic and keyword-based. It does not implement the future re
## Phase 5B Candidate Review

Phase 5A queues durable memory candidates automatically, but it does not make them permanent memory by default. Phase 5B adds the operator review console at `/admin/memory/candidates`, where Joven can approve, reject, edit, mark duplicate, or capture candidates. This keeps Pandora teachable: autopilot surfaces what might matter, while human review prevents noisy, sensitive, wrong, duplicate, or wrong-namespace items from polluting durable memory.

## Phase 5C Daily Compaction

Phase 5A queues candidates. Phase 5B reviews and captures candidates. Phase 5C compacts reviewed/captured memory into versioned profiles, open loops, and daily context packs. This is the deterministic compounding layer that lets Pandora improve over time without enabling model calls, embeddings, semantic retrieval, public reads, public writes, or permanent auto-capture.

Recommended environment posture:

```bash
PANDORA_ENABLE_MEMORY_DISTILLATION=true
PANDORA_INTERNAL_JOB_TOKEN=<server-only-secret>
PANDORA_ENABLE_MODEL_CALLS=false
PANDORA_ENABLE_EMBEDDINGS=false
PANDORA_ENABLE_SEMANTIC_RETRIEVAL=false
PANDORA_ENABLE_PUBLIC_MEMORY_READ=false
PANDORA_ENABLE_PUBLIC_MEMORY_PERSISTENCE=false
PANDORA_ENABLE_AUTO_CAPTURE=false
```

`PANDORA_INTERNAL_JOB_TOKEN` must remain server-only and must never be exposed as a `NEXT_PUBLIC_*` variable. Do not enable model calls or embeddings just for compaction, and do not enable permanent auto-capture yet.
94 changes: 94 additions & 0 deletions docs/pandora-memory-compaction.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Pandora Memory Phase 5C: Daily Compaction

Phase 5C adds deterministic daily compaction for Pandora Memory. It turns reviewed and captured source-backed memory into clean operating context: versioned profiles, open loops, and daily context packs.

## Why compaction matters

Phase 5A queues candidates. Phase 5B lets an operator approve, reject, edit, duplicate, or capture them. Phase 5C makes the reviewed/captured layer compound over time instead of becoming an infinite event log.

## Data layers

- **Candidates** are proposed memory items awaiting review or capture.
- **Memory events** are captured, source-backed memory rows.
- **Session digests** summarize sessions and can feed compaction wrappers.
- **Profiles** store durable operating, project, risk, relationship, people, deployment, and AU canon state.
- **Open loops** track unresolved blockers, risks, review backlog, and follow-up actions.
- **Context packs** are generated daily or master summaries for downstream private context use.

## Deterministic profile synthesis

`runDailyMemoryCompaction` scans recent rows by `user_id`, `namespace`, and ISO `since`. It uses deterministic keyword signals only. It does not call models, create embeddings, or use semantic retrieval.

Profile updates are versioned. Existing active profiles are marked `superseded`, and a new active row is inserted with merged evidence references.

## Feedback-aware compaction

Approved and captured candidates are eligible profile/open-loop inputs. Rejected and duplicate candidates remain visible in the compaction result but are not promoted as source material. Edited candidate fields are preferred because compaction reads the reviewed candidate title/summary/type/sensitivity fields.

Blocked secrets are excluded and redacted before pack creation.

## Open-loop updates

The open-loop engine creates or updates loops by `user_id`, `namespace`, `loop_type`, and `subject_key`. Existing open/acknowledged loops are updated, evidence references are merged, and severity can increase but not downgrade.

## Daily context packs

Daily compaction creates a `memory_context_packs` row with `pack_type=daily`, key points, projects, people, decisions, risks, open loops, and generated event ids. Older active daily packs for the same user/namespace are superseded when possible.

## Internal job endpoint

`POST /api/memory/jobs/daily-digest`

Headers:

```http
Authorization: Bearer <PANDORA_INTERNAL_JOB_TOKEN>
```

Body:

```json
{ "namespace": "real_life", "since": "2026-06-27T00:00:00.000Z", "dry_run": true }
```

The endpoint requires `PANDORA_ENABLE_MEMORY_DISTILLATION=true` and is internal-token protected. If needed for the first single-user rollout, send `user_id` only through this internal-token route or configure `PANDORA_MEMORY_BRIDGE_USER_ID` server-side.

## Env posture

```bash
PANDORA_ENABLE_MEMORY_DISTILLATION=true
PANDORA_INTERNAL_JOB_TOKEN=<server-only-secret>
PANDORA_ENABLE_MODEL_CALLS=false
PANDORA_ENABLE_EMBEDDINGS=false
PANDORA_ENABLE_SEMANTIC_RETRIEVAL=false
PANDORA_ENABLE_PUBLIC_MEMORY_READ=false
PANDORA_ENABLE_PUBLIC_MEMORY_PERSISTENCE=false
PANDORA_ENABLE_AUTO_CAPTURE=false
```

Never expose `PANDORA_INTERNAL_JOB_TOKEN` as `NEXT_PUBLIC_*`.

## Safety boundaries

- deterministic only
- no model calls required
- no embeddings required
- no semantic retrieval required
- no public memory reads/writes
- blocked secrets excluded/redacted
- `real_life` and `au` remain separated
- permanent auto-capture remains disabled by default

## Rollout

1. Deploy with `PANDORA_INTERNAL_JOB_TOKEN` configured.
2. Keep model calls, embeddings, semantic retrieval, public read/write, and auto-capture off.
3. Enable `PANDORA_ENABLE_MEMORY_DISTILLATION=true`.
4. Run the daily digest endpoint with `dry_run=true`.
5. Inspect profile, loop, and pack output.
6. Run for `real_life`.
7. Run for `au`.
8. Verify daily context packs appear.
9. Verify active profiles version correctly.
10. Verify open loops update without duplicate spam.
11. Only then schedule the job externally.
30 changes: 30 additions & 0 deletions lib/services/memory-compaction-service.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
/* eslint-disable @typescript-eslint/no-explicit-any */
import type { MemoryBridgeDbClient, MemoryBridgeNamespace, MemoryEvent } from "@/lib/services/memory-bridge-service";
import { buildDailyContextPack } from "@/lib/services/memory-distillation-service";
import { upsertVersionedMemoryProfile } from "@/lib/services/memory-profile-service";
import { upsertOpenLoop } from "@/lib/services/memory-open-loop-service";
import { redactSecrets } from "@/lib/services/memory-redaction-service";
export type MemoryCompactionScope={user_id:string;namespace:MemoryBridgeNamespace;since?:string;dry_run?:boolean};
export type MemoryCompactionResult={ok:boolean;namespace:MemoryBridgeNamespace;dry_run:boolean;daily_pack?:unknown;updated_profiles:unknown[];opened_or_updated_loops:unknown[];reviewed_candidates:unknown[];captured_events:unknown[];feedback_events:unknown[];blockers:string[];warnings:string[];next_step?:string};
const projects=["Pandora Memory","PLP","Hatid","Speedcash","SpeedyPay","RetargetOS","AI GrowthOS","Red-Apple"];
const personNames=["Patty","Melodee","Mang Rudy","Mang Bert","Ate Dhes"];
const blocked="[REDACTED_SECRET]";
const clean=(s:string)=>redactSecrets(s).replace(/\s+/g," ").trim().slice(0,500);
const textOf=(x:any)=>clean([x.extracted_summary,x.summary,x.title,x.redacted_excerpt,x.raw_text,x.raw_excerpt].filter(Boolean).join(" — "));
const has=(t:string,words:string[])=>words.some(w=>t.toLowerCase().includes(w.toLowerCase()));
const ref=(kind:string,x:any)=>({kind,id:x.id,source:x.source,source_ref:x.source_ref,created_at:x.created_at??x.reviewed_at});
const subject=(s:string)=>s.toLowerCase().replace(/[^a-z0-9]+/g,"_").replace(/^_|_$/g,"")||"global";
async function list(client:MemoryBridgeDbClient,table:string,input:MemoryCompactionScope,statuses?:string[]){const q:any=client.from(table).select("*").eq("user_id",input.user_id).eq("namespace",input.namespace).order("created_at",{ascending:false}).limit(100);const r=await (q as Promise<{data:any[]|null;error:{message:string}|null}>); if(r.error)throw new Error(`${table}: ${r.error.message}`); return (r.data??[]).filter(x=>(!input.since||String(x.created_at??x.reviewed_at??"")>=input.since!)&&(!statuses||statuses.includes(x.status))).filter(x=>x.status!=="blocked_secret"&&textOf(x)!==blocked);}

@coderabbitai coderabbitai Bot Jun 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Apply source filters before the 100-row limit.

Line 17 fetches the latest 100 rows first, then filters by since and status in memory. Busy tables with many rejected/blocked/out-of-window rows can cause eligible records in the compaction window to be skipped. Push since/status filtering into the query before limit(100).

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/services/memory-compaction-service.ts` at line 17, The list() helper in
memory-compaction-service is applying since/status filtering after limit(100),
which can skip eligible rows on busy tables. Move the source filters into the
query built from client.from(table) before the limit is applied, using the same
MemoryCompactionScope fields and statuses constraints, and keep the existing
blocked_secret/textOf filtering as needed for any remaining in-memory checks.

function profileInputs(user_id:string,namespace:MemoryBridgeNamespace,sources:any[]){const out:any[]=[];const add=(profile_type:string,subject_key:string,title:string,items:any[],bucket:string)=>{if(!items.length)return; out.push({user_id,namespace,profile_type,subject_key,title,summary:items.map(textOf).filter(Boolean).slice(0,8).join("\n"),[bucket]:items.map((x:any)=>({text:textOf(x),ref:ref(x.raw_text?"event":"candidate",x)})),evidence_refs:items.map((x:any)=>ref(x.raw_text?"event":"candidate",x)),confidence:Math.min(0.95,0.55+items.length*0.05)});};
add("operating_profile","global","Global operating profile",sources.filter(x=>has(textOf(x),["I prefer","from now on","always","don't","dont","be blunt","be concise","don't overpraise","check repo first","analyze automatically","maximize Codex task"])),"preferences");
add("style_profile","global","Global style profile",sources.filter(x=>has(textOf(x),["tone","style","format","writing","prompt style","direct","Taglish","cinematic"])),"preferences");
add("risk_profile","global","Global risk profile",sources.filter(x=>has(textOf(x),["gambling","money risk","reputation risk","avoidance","fantasy-vs-execution","legal risk","deployment claim not verified","fake progress","risk"])),"risks");
for(const p of projects){add("project_profile",subject(p),`${p} project profile`,sources.filter(x=>has(textOf(x),[p])||has(textOf(x),["GitHub","PR","merge","Vercel","Supabase","migration","deployment","build","CI","repo"])),"facts"); add("technical_deployment_profile",subject(p),`${p} deployment profile`,sources.filter(x=>has(textOf(x),[p])&&has(textOf(x),["deployed","failed","pending","merged","blocked","production","Vercel","GitHub Actions","CodeRabbit","Supabase migration"])),"facts");}

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Require a project match before writing project profiles

For every configured project, this predicate also accepts generic repo/deployment words even when the source text does not mention p, so one "GitHub PR" or "Vercel" memory is copied into PLP, Hatid, Speedcash, and all other project profiles. That pollutes unrelated durable project profiles during compaction; require the project name or explicit project metadata before writing a per-project profile.

Useful? React with 👍 / 👎.

for(const n of personNames){if(namespace==="real_life"&&["Melodee","Mang Rudy","Mang Bert","Ate Dhes"].includes(n))continue; add(namespace==="au"?"people_profile":"relationship_loop_profile",subject(n),`${n} profile`,sources.filter(x=>has(textOf(x),[n,"relationship loop","emotional loop","relapse","unresolved emotional pattern"])),"patterns");}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
if(namespace==="au")add("writing_canon_profile","global","AU writing canon",sources.filter(x=>has(textOf(x),["AU","canon","chapter","story","fictional continuity","Melodee","Mang Rudy","Mang Bert","Ate Dhes"])),"facts");
return out.filter(p=>p.summary&&!p.summary.includes(blocked));}
function loopInputs(user_id:string,namespace:MemoryBridgeNamespace,sources:any[]){const specs=[ ["deployment_blocker","deployment","Deployment blocker",["deployment blocked","Vercel failed","CI failed","failed migration","blocked"],8], ["review_backlog","candidate_review","Candidate review backlog",["candidate review backlog","pending review"],5], ["risk","global","Risk follow-up",["gambling","money risk","reputation risk","legal risk"],9], ["relationship_loop","relationship","Relationship/emotional loop",["relationship loop","emotional loop","relapse","unresolved"],7], ["project_followup","project","Project follow-up",["what next","next action","todo","follow up","pending"],5] ] as const; return specs.flatMap(([loop_type,sk,title,words,severity])=>{const items=sources.filter(x=>has(textOf(x),words as unknown as string[]));return items.length?[{user_id,namespace,loop_type,subject_key:sk,title,description:items.map(textOf).slice(0,5).join("\n"),severity,evidence_refs:items.map((x:any)=>ref(x.raw_text?"event":"candidate",x)),next_action:"Review and close when source evidence is resolved."}]:[];});}
export async function runDailyMemoryCompaction(client:MemoryBridgeDbClient,input:MemoryCompactionScope):Promise<MemoryCompactionResult>{const since=input.since??new Date(Date.now()-24*60*60*1000).toISOString(); const base={ok:false,namespace:input.namespace,dry_run:!!input.dry_run,updated_profiles:[],opened_or_updated_loops:[],reviewed_candidates:[],captured_events:[],feedback_events:[],blockers:[],warnings:[]} as MemoryCompactionResult; try{const events=await list(client,"memory_events",{...input,since},["captured","reviewed","promoted"]);const candidates=await list(client,"memory_capture_candidates",{...input,since},["approved","captured","rejected","duplicate"]);const feedback=await list(client,"memory_feedback_events",{...input,since});base.reviewed_candidates=candidates;base.captured_events=events;base.feedback_events=feedback;const usable=[...events,...candidates.filter(c=>["approved","captured"].includes(c.status))]; if(!usable.length){return {...base,ok:true,warnings:["no_source_data_available"],next_step:"Capture or approve memory before compaction."};}
for(const p of profileInputs(input.user_id,input.namespace,usable)){const r=await upsertVersionedMemoryProfile(client,{...p,dry_run:input.dry_run}); if(r.ok)base.updated_profiles.push(r.profile); else base.warnings.push(...r.warnings,...r.blockers);}
for(const l of loopInputs(input.user_id,input.namespace,usable)){const r=await upsertOpenLoop(client,{...l,dry_run:input.dry_run}); if(r.ok)base.opened_or_updated_loops.push(r.open_loop); else base.warnings.push(...r.warnings,...r.blockers);}
const safeEvents=events.map((e:MemoryEvent)=>({...e,raw_text:clean(e.raw_text),extracted_summary:e.extracted_summary?clean(e.extracted_summary):e.extracted_summary})); const pack={...buildDailyContextPack(input.namespace,input.user_id,safeEvents),open_loops:base.opened_or_updated_loops,risks:(base.updated_profiles as any[]).filter(p=>p.profile_type==="risk_profile")}; if(input.dry_run)base.daily_pack=pack; else {await client.from("memory_context_packs").update({status:"superseded",updated_at:new Date().toISOString()}).eq("user_id",input.user_id).eq("namespace",input.namespace).eq("pack_type","daily").eq("status","active").select("*").single(); const ins=await client.from("memory_context_packs").insert({...pack,status:"active"}).select("*").single(); if(ins.error||!ins.data)base.warnings.push(ins.error?.message??"context_pack_write_failed"); else base.daily_pack=ins.data;} return {...base,ok:true};}catch(e){return {...base,blockers:["compaction_failed"],warnings:[e instanceof Error?e.message:"unknown failure"],next_step:"Check compaction source table schemas and RLS."};}}
Comment on lines +28 to +30

@coderabbitai coderabbitai Bot Jun 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Propagate upsert blockers instead of returning success.

Lines 28-29 append r.blockers into base.warnings, and Line 30 returns ok: true even when profile/open-loop writes failed. Preserve blockers in base.blockers and stop before writing the daily pack when required upserts fail.

Proposed direction 
- if(r.ok)base.updated_profiles.push(r.profile); else base.warnings.push(...r.warnings,...r.blockers);
+ if(r.ok)base.updated_profiles.push(r.profile);
+ else {
+   base.warnings.push(...r.warnings);
+   base.blockers.push(...r.blockers);
+ }

- if(r.ok)base.opened_or_updated_loops.push(r.open_loop); else base.warnings.push(...r.warnings,...r.blockers);
+ if(r.ok)base.opened_or_updated_loops.push(r.open_loop);
+ else {
+   base.warnings.push(...r.warnings);
+   base.blockers.push(...r.blockers);
+ }

+ if (base.blockers.length) {
+   return {
+     ...base,
+     ok: false,
+     next_step: "Resolve profile/open-loop write failures before writing the context pack.",
+   };
+ }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for(const p of profileInputs(input.user_id,input.namespace,usable)){const r=await upsertVersionedMemoryProfile(client,{...p,dry_run:input.dry_run}); if(r.ok)base.updated_profiles.push(r.profile); else base.warnings.push(...r.warnings,...r.blockers);}
for(const l of loopInputs(input.user_id,input.namespace,usable)){const r=await upsertOpenLoop(client,{...l,dry_run:input.dry_run}); if(r.ok)base.opened_or_updated_loops.push(r.open_loop); else base.warnings.push(...r.warnings,...r.blockers);}
const safeEvents=events.map((e:MemoryEvent)=>({...e,raw_text:clean(e.raw_text),extracted_summary:e.extracted_summary?clean(e.extracted_summary):e.extracted_summary})); const pack={...buildDailyContextPack(input.namespace,input.user_id,safeEvents),open_loops:base.opened_or_updated_loops,risks:(base.updated_profiles as any[]).filter(p=>p.profile_type==="risk_profile")}; if(input.dry_run)base.daily_pack=pack; else {await client.from("memory_context_packs").update({status:"superseded",updated_at:new Date().toISOString()}).eq("user_id",input.user_id).eq("namespace",input.namespace).eq("pack_type","daily").eq("status","active").select("*").single(); const ins=await client.from("memory_context_packs").insert({...pack,status:"active"}).select("*").single(); if(ins.error||!ins.data)base.warnings.push(ins.error?.message??"context_pack_write_failed"); else base.daily_pack=ins.data;} return {...base,ok:true};}catch(e){return {...base,blockers:["compaction_failed"],warnings:[e instanceof Error?e.message:"unknown failure"],next_step:"Check compaction source table schemas and RLS."};}}
for(const p of profileInputs(input.user_id,input.namespace,usable)){const r=await upsertVersionedMemoryProfile(client,{...p,dry_run:input.dry_run}); if(r.ok)base.updated_profiles.push(r.profile); else {base.warnings.push(...r.warnings); base.blockers.push(...r.blockers);}
for(const l of loopInputs(input.user_id,input.namespace,usable)){const r=await upsertOpenLoop(client,{...l,dry_run:input.dry_run}); if(r.ok)base.opened_or_updated_loops.push(r.open_loop); else {base.warnings.push(...r.warnings); base.blockers.push(...r.blockers);}
if (base.blockers.length) { return { ...base, ok: false, next_step: "Resolve profile/open-loop write failures before writing the context pack.", }; }
const safeEvents=events.map((e:MemoryEvent)=>({...e,raw_text:clean(e.raw_text),extracted_summary:e.extracted_summary?clean(e.extracted_summary):e.extracted_summary})); const pack={...buildDailyContextPack(input.namespace,input.user_id,safeEvents),open_loops:base.opened_or_updated_loops,risks:(base.updated_profiles as any[]).filter(p=>p.profile_type==="risk_profile")}; if(input.dry_run)base.daily_pack=pack; else {await client.from("memory_context_packs").update({status:"superseded",updated_at:new Date().toISOString()}).eq("user_id",input.user_id).eq("namespace",input.namespace).eq("pack_type","daily").eq("status","active").select("*").single(); const ins=await client.from("memory_context_packs").insert({...pack,status:"active"}).select("*").single(); if(ins.error||!ins.data)base.warnings.push(ins.error?.message??"context_pack_write_failed"); else base.daily_pack=ins.data;} return {...base,ok:true};}catch(e){return {...base,blockers:["compaction_failed"],warnings:[e instanceof Error?e.message:"unknown failure"],next_step:"Check compaction source table schemas and RLS."};}}
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/services/memory-compaction-service.ts` around lines 28 - 30, The
compaction flow in memory-compaction-service is masking failed upserts by
pushing r.blockers into warnings and still returning ok:true. Update the profile
and loop loops to preserve blockers in base.blockers (not warnings), and if any
required upsert fails, stop before buildDailyContextPack and the
memory_context_packs write so the function returns failure instead of a
successful daily pack.

@coderabbitai coderabbitai Bot Jun 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Handle daily-pack supersede failures before inserting.

Line 30 ignores the result of superseding the existing active daily pack. If that update fails for reasons other than “no existing row”, the subsequent insert can leave multiple active daily packs for the same user/namespace. Check the update result and only continue on a true no-op/no-row case.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/services/memory-compaction-service.ts` at line 30, The daily pack
replacement flow in the compaction service ignores failures from the supersede
step, which can allow multiple active packs to exist. Update the logic around
the client.from("memory_context_packs").update(...) call in the compaction path
to inspect its result and stop/record a warning on any real failure, only
proceeding to the insert when the update truly affected no rows. Keep the fix
localized to the daily-pack handling branch that builds pack and writes to
memory_context_packs.

Loading
Loading