docs(readme): pivot to four-layer model + Auto Dream vindication framing - #22
Merged
Conversation
…backend Phase 0 Task 0.2 of the pgvector + AGE migration plan. Decision: WAIT for upstream MemPalace#665 (skuznetsov's PostgreSQL backend) to merge, then cherry-pick — rather than fork-port. 1839 LOC of working code on top of the MemPalace#995/RFC 001 BaseBackend contract; ~51 LOC of moderate merge conflict (palace.py 32, test_backends.py 19, plus trivial README + uv.lock). pg_sorted_heap codepath is gated by extension availability so our apache/age:PG16 + pgvector substrate runs the fallback path which is exactly what we want. Plan-B triggers documented to revisit (4 weeks of maintainer silence, unresolved blocking review concern, or a Phase 1 blocker that requires patching MemPalace#665 internals). On any trigger, switch Phase 1 from Task 1.A.1 to Task 1.B.1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Substantial README rewrite (+137 / -100 net) reflecting three things that have happened since the last refresh (a67be3f, 2026-05-10 sync): 1. **Four-layer model promoted to the lede** (storage / encoder / retrieval / consumption). Anchored on the SME jp-realm-v0.1 numbers: RLM-Qwen-7B and RLM-Llama-70B both ceiling at 46.67% recall while Familiar's deterministic pipeline hits 78.33% on the same corpus — model size doesn't fix invocation discipline. The recovery-collection migration story (six weeks old) moves down to "What this fork has learned." The empirical claim is carried with a calibration paragraph that explicitly says the defensible findings are deltas under identical conditions and absolute percentages are decoration; methodology disclosure lives in docs/research/. 2. **Auto Dream framing rewritten as vindication.** Anthropic shipped Auto Dream in late April in two research-preview surfaces (Claude Code Auto Dream with manual `/dream` + 24h/5-sessions auto-trigger; Dreams API in Managed Agents with the `dreaming-2026-04-21` beta header on `claude-opus-4-7` and `claude-sonnet-4-6`). The Dreams API design — input read-only, output a separate store you review/attach/discard — ratifies the verbatim-vs-derivative axis the fork's been arguing for. Section closes affirmatively: "the verbatim layer doesn't need consolidation; it needs durability." This replaces the prior "neither layer has consolidation" framing which was wrong post-2026-04-21. 3. **Substrate section updated to in-flight status.** The earlier "exploring — not committed" framing is replaced by the actual state: PR #21 on this fork, cherry-pick of skuznetsov's upstream MemPalace#665 against the RFC 001 BaseBackend contract, live PG16+pgvector 0.8.2+AGE 1.6.0 test container on JP's homelab LAN (10.0.6.120:5433, internal-only), Plan-B trigger documented at 2026-06-08. New section: "Convergence with peer systems" — triangulation across Familiar, CampaignGenerator, Kent, adaptmem. Four agreements across the systems; divergence is where intelligence above retrieval lives (algorithm vs human-in-loop vs trained policy). Adds 6 research files in docs/research/ (compass_artifact_wf-*, adaptmem-orthogonal-layers, convergent-findings-kostadis-comparison, three-mempalace-consumers, three-patterns-for-agent-memory) and the pgvector composition decision doc as supporting material. All linked from the README's Sources section. Tactical fixes: test count ~1,500 → ~1,850; sync date 2026-04-27 → 2026-05-10; fork-ahead count ~16 → ~14; drawer count ~151K → ~160K; setup commands now lead with `uv sync --extra dev` (matching CLAUDE.md); PR table now shows 10 open jphein PRs (verified via gh). Doesn't change: ChromaDB stays the default backend, principles 1-3 stay the design test, the comparison table of peer memory systems keeps its shape. The substrate section explicitly says ChromaDB is the default; pgvector+AGE is "one specific implementation worth picking" against the RFC 001 seam, not a parallel reimplementation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR substantially rewrites the fork README to foreground a “four-layer” agent-memory model (storage/encoder/retrieval/consumption), updates positioning around Anthropic Auto Dream / Dreams API, and refreshes substrate/backends status. It also adds several research/internal documents that the README links to as supporting context.
Changes:
- Reframe README around the four-layer model and updated operational thesis, including new “Convergence with peer systems” and “Two memory layers” sections.
- Update substrate section to “in-flight” pgvector+AGE implementation status with Plan‑B trigger and links to a new internal decision record.
- Add new research notes under
docs/research/and an internal decision doc underdocs/internal/.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Major narrative restructure: four-layer model, Auto Dream framing, substrate status, updated setup commands, refreshed sources/claims. |
| docs/research/three-patterns-for-agent-memory.md | New research note on retrieval patterns and SME findings; referenced by README. |
| docs/research/three-mempalace-consumers.md | New comparison note across Familiar/CampaignGenerator/Kent consumption-layer approaches. |
| docs/research/convergent-findings-kostadis-comparison.md | New companion note summarizing Kostadis/CampaignGenerator findings and implications. |
| docs/research/compass_artifact_wf-ad108fcc-3960-4eab-ad5d-234bf365b2f4_text_markdown.md | New synthesis artifact covering four-layer framing + methodology distinctions. |
| docs/research/compass_artifact_wf-28bac4e8-71d9-4175-837a-d4ad563aec8d_text_markdown.md | New broader landscape synthesis doc; includes SME invocation claims. |
| docs/research/adaptmem-orthogonal-layers.md | New note positioning adaptmem as encoder-layer orthogonal lift. |
| docs/internal/pgvector-665-decision.md | New internal decision record documenting the MemPalace#665 “WAIT” composition stance and triggers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| --- | ||
|
|
||
| This fork tracks `upstream/develop` through the 2026-04-27 sync and runs in production on a 151,478-drawer palace behind [palace-daemon](https://github.com/jphein/palace-daemon) at `disks.jphe.in:8085`. It carries 16 fork-ahead changes that compose with — not replace — bensig's release direction; four landed upstream on 2026-04-26 (#1173, #1177, #1198, #1201). 1,500 tests pass on `main`. The new things here are *what we've learned*, not just what we've fixed. | ||
| This fork tracks `upstream/develop` through the 2026-05-10 sync (commit `a67be3f`, fork merge via `chore/sync-develop-2026-05-10`) and runs in production on a ~160K-drawer palace behind [palace-daemon](https://github.com/jphein/palace-daemon). It carries roughly 14 fork-ahead changes that compose with — not replace — bensig's release direction; the v3.3.5 release (2026-05-10) includes our co-authored `_get_collection` retry-once via upstream #1377. ~1850 tests pass on `main`. The new things here are *what we've learned*, not just what we've fixed. |
| 4. **Consumption.** What happens after retrieval. This is where the R@k → end-to-end QA gap lives — MemPalace's 96.6% R@5 → 82.6% E2E QA on Issue #39 reproduction; @terrizoaguimor's [Celiums](https://celiums.ai/) benchmarking shows 100% retrieval rate but only 62.3% QA with Opus 4.6. Three architectural bets compete: algorithm (Familiar's always-on pipeline), human-in-the-loop ([Kostadis's retrieve/render isolation](https://github.com/kostadis/CampaignGenerator/blob/main/docs/rlm_paper_comparison.md) CI invariant), trained policy ([Kent's APO recall games](https://github.com/kenchambers/kent)). The right answer probably depends on workload; the field doesn't yet know. | ||
|
|
||
| Both modes returned real content. The structural fix did the work the algorithmic patch (`kind=` filter + over-fetch) couldn't. Empirical detail at [`~/Projects/notebook/data/cat9-postmigrate/REPORT.md`](https://github.com/jphein/notebook/blob/main/data/cat9-postmigrate/REPORT.md); the long-form story behind it lives at [`notebook/essays/2026-04-25-mempalace-lessons.md`](https://github.com/jphein/notebook/blob/main/essays/2026-04-25-mempalace-lessons.md). | ||
| *Calibration on the SME numbers:* 30 questions, beta-level instrumentation, substring-on-filename scoring. The defensible findings are deltas under identical conditions; absolute percentages are decoration. Methodology disclosure and the broader four-layer synthesis live in [`docs/research/three-patterns-for-agent-memory.md`](docs/research/three-patterns-for-agent-memory.md) and the [compass artifact](docs/research/compass_artifact_wf-ad108fcc-3960-4eab-ad5d-234bf365b2f4_text_markdown.md). The shape of the result — invocation ceiling — generalizes; the exact numbers do not. |
| ## Substrate: Postgres + pgvector + Apache AGE | ||
|
|
||
| *Status: exploring — not committed.* | ||
| *Status: implementation in flight on `feat/pgvector-age-impl`, composing on upstream [#665](https://github.com/MemPalace/mempalace/pull/665) (skuznetsov's PostgreSQL backend on the RFC 001 `BaseBackend` contract). Live test container on JP's homelab LAN (`10.0.6.120:5433`, internal-only) running PG16 + pgvector 0.8.2 + AGE 1.6.0 via `apache/age:release_PG16_1.6.0` + `postgresql-16-pgvector`. Composition stance: WAIT for #665 to merge upstream, with a documented Plan-B trigger at 2026-06-08 — full rationale at [`docs/internal/pgvector-665-decision.md`](docs/internal/pgvector-665-decision.md).* |
|
|
||
| **Verbatim-on-write** (MemPalace): preservation costs are low — just store raw — so defer all interpretation. Agents get raw text plus good retrieval. The bet is that derived artifacts lose the exact command/error/snippet you actually need to recover. | ||
|
|
||
| The OSS landscape has heavily explored the compile-upstream side — Cognee, Letta, Mem0, Graphiti, LightRAG, GraphRAG — almost all of which transform on write. Verbatim-first storage with MCP exposure is rare. MemPalace is the most production-tested instance we know of, at 134K+ drawers across 60+ rooms. |
Comment on lines
+39
to
+42
| | Run | Mean recall | Tool-call distribution | | ||
| |---|---|---| | ||
| | `rlm` + Qwen 2.5 7B Q5_K_M | 46.67% | 25/30 zero-call, 2/30 used tool | | ||
| | `rlm` + Llama 3.3 70B | 46.67% | 22/30 zero-call, 8/30 used tool | |
Comment on lines
+45
to
+51
| - **rlm + Qwen 2.5 7B Q5_K_M**: 46.67% recall, 25/30 zero-call (no tool invocation), 2/30 used tool | ||
| - **rlm + Llama 3.3 70B**: 46.67% recall, 22/30 zero-call, 8/30 used tool | ||
| - **familiar v0.3.9** (deterministic pipeline: rerank, temporal decay, extractive compression): 78.33% recall | ||
|
|
||
| Critical calibrations: 30 questions, beta-level instrumentation, substring-on-filename scoring. Defensible findings are deltas under identical conditions; absolute numbers are decoration. Sample size limits generalization. | ||
|
|
||
| The pattern: both RLM configurations ceiling at the orchestrator's willingness to call retrieval tools. Model capability (7B versus 70B) shifts invocation frequency slightly (2/30 → 8/30) but doesn't break the ceiling—both land at identical 46.67% recall. The deterministic pipeline, which invokes retrieval unconditionally, achieves 32-point higher recall without any LLM orchestration intelligence. |
Comment on lines
+98
to
+100
| - [kostadis/mempalace](https://github.com/kostadis/mempalace) — verbatim memory palace fork, hierarchical retrieval branch | ||
| - `mempalace/tests/benchmarks/test_hierarchical_aaak_gate1.py` — Gate 1 (recall@10 = 1.0 at 19.82× cost reduction vs. flat) | ||
| - `CampaignGenerator/tests/benchmarks/test_rlm_benchmark_rpg_gate2.py` — Gate 2 (top-3 ≥ 90% on 15 RPG queries) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Substantial README rewrite (+137 / -100 net) reflecting what's changed since the last refresh (commit
a67be3f, 2026-05-10 sync):/dream+ Managed Agents Dreams API withdreaming-2026-04-21beta header). The Dreams API's input-read-only / output-separate-store design ratifies the verbatim-vs-derivative axis. Closes affirmatively: "the verbatim layer doesn't need consolidation; it needs durability."10.0.6.120:5433, internal-only), and the Plan-B trigger date 2026-06-08.New section
Convergence with peer systems — triangulation across Familiar, CampaignGenerator, Kent, adaptmem. Four agreements; divergence is where intelligence above retrieval lives.
Supporting research
Adds 6 research files in
docs/research/(compass_artifact_wf-*, adaptmem-orthogonal-layers, convergent-findings-kostadis-comparison, three-mempalace-consumers, three-patterns-for-agent-memory) anddocs/internal/pgvector-665-decision.md. All linked from the Sources section.Tactical fixes
a67be3f)uv sync --extra dev(matches updated CLAUDE.md)gh pr list --repo MemPalace/mempalace --author jphein --state open)tests/test_corpus_origin_integration.py::test_no_internal_coordination_jargon_in_source_or_testsenforcement)What doesn't change
ChromaDB stays the default backend. Principles 1–3 stay the design test. The peer-system comparison table keeps its shape. Substrate section explicitly says ChromaDB is default and pgvector+AGE is "one specific implementation worth picking" against the RFC 001 seam — not a parallel reimplementation.
Test plan
Phase N/Task N.X.Y) in README — passes the lint test/dream)gh pr listdocs/research/*.mdSources links resolve (files committed in this PR)docs/internal/pgvector-665-decision.mdlink resolves (cherry-picked from feat/pgvector-age-impl commitfbd8dbd)notebook/data/...if exact values challenged — current text hedges with the calibration paragraph and links to methodology disclosureRelationship to PR #21
Separate concerns: PR #21 is the substrate code (cherry-pick of upstream MemPalace#665, 1851 tests passing, live container). This PR is the README/positioning refresh. Independent merge order; README correctly hedges "in-flight on feat/pgvector-age-impl" so it reads honestly whether or not #21 has merged.
🤖 Generated with Claude Code