docs(plan): pgvector + AGE migration implementation plan - #20
Merged
Conversation
…2 steps) Implementation plan derived from the 2026-05-10 substrate-migration spec. Three implementable phases + operator cutover, structured TDD: Phase 0 (preflight) — verify Postgres + extensions on dev host, decide upstream MemPalace#665 composition stance (wait vs fork-port) in a recorded internal doc. Phase 1 (pgvector backend) — bifurcated by 0.2's decision: 1.A.* — compose with MemPalace#665, cherry-pick + verify against our v3.3.5 main 1.B.* — fork-port mempalace.backends.postgres ourselves Plus common tasks: HNSW + supporting indexes (fork rows 1/7/12), where- clause translation ($eq/$ne/$in), MempalaceConfig.backend + postgres_dsn properties, postgres CI workflow with pgvector/pgvector service image. Phase 2 (AGE-backed KG) — KnowledgeGraphAGE skeleton + graph bootstrap, add_triple via Cypher MERGE+CREATE with temporal validation mirroring upstream v3.3.5 KG hardening, as_of temporal filter, kg_backend config property + mcp_server routing. Phase 3 (migration tool) — mempalace migrate-to-postgres CLI: - phase 0: preflight (daemon-stopped check, extension availability) - phase 1: schema (extensions + tables + checkpoint helpers) - phase 2: batched drawer copy (ON CONFLICT DO NOTHING; idempotent) - phase 3-5: closets, indexes, KG -> AGE (Task 3.4 template) - phase 6: verify (count parity + sample round-trip) - phase 7: done (cleanup checkpoints, print cutover steps) Phase 4 (operator cutover) — dry-run on cp -al snapshot of canonical 160K-drawer palace, measure phase timings, update spec performance budget with measured numbers, then production cutover. Self-review verified: spec coverage complete, type consistency clean (phase_N_<name> + KnowledgeGraphAGE signatures + _set_checkpoint helpers all consistent across tasks), no placeholder failures (the 2 TBDs are open questions with documented fallback paths). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a detailed, phased implementation plan document for migrating MemPalace’s storage substrate from ChromaDB + sqlite KG to Postgres with pgvector (vectors) and Apache AGE (knowledge graph), including a restartable/idempotent migration CLI plan and task-by-task TDD steps.
Changes:
- Introduces a multi-phase execution plan (preflight → pgvector backend → AGE KG → migration tool → operator cutover).
- Specifies concrete TDD task breakdowns, including schema/indexing, where-clause translation, config/env routing, and migration phases/checkpointing.
- Includes a proposed CI workflow snippet for Postgres backend testing.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| **Architecture:** Three phases on top of upstream's `BaseCollection` seam (#413, merged). Phase 1 lands the pgvector backend (composing with upstream #665 or fork-porting). Phase 2 adds the AGE-backed KG layer. Phase 3 ships the `mempalace migrate-to-postgres` CLI tool. Each phase produces working, testable software on its own. | ||
|
|
||
| **Tech Stack:** Python 3.9+ (existing mempalace floor), psycopg[binary]>=3.1, pgvector-python>=0.3, apache-age-python (TBD on package availability — fallback is raw Cypher over a psycopg cursor), pytest-postgresql for unit tests, docker-compose Postgres for integration tests. |
Comment on lines
+736
to
+742
| steps: | ||
| - uses: actions/checkout@v5 | ||
| - uses: actions/setup-python@v6 | ||
| with: { python-version: "3.13" } | ||
| - run: pip install -e ".[dev,postgres]" | ||
| - run: pip install pgvector | ||
| - env: |
Comment on lines
+739
to
+745
| with: { python-version: "3.13" } | ||
| - run: pip install -e ".[dev,postgres]" | ||
| - run: pip install pgvector | ||
| - env: | ||
| TEST_POSTGRES_DSN: postgresql://palace:palace@localhost:5432/mempalace_test | ||
| run: python -m pytest tests/test_backends_postgres.py -v | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implementation plan derived from the substrate-migration spec merged via #19. Four phases, 92 bite-sized TDD steps, ready for execution under
superpowers:subagent-driven-developmentorsuperpowers:executing-plans.Structure
MEMPALACE_BACKEND=postgresopt-in; HNSW + supporting indexes; where-clause translation; CI workflowMEMPALACE_KG_BACKEND=age; Cypher MERGE+CREATE with temporal validation; as_of filtermempalace migrate-to-postgresCLI, 7 sub-phases, restartable, idempotentcp -alsnapshot, measure timings, production cutoverKey decisions encoded
mempalace.backends.postgresourselves. Plan supports both paths without forking the document._run_cypherso the executor can proceed even if no python AGE wrapper exists.Self-review
phase_N_<name>+KnowledgeGraphAGEsignatures + checkpoint helpers consistent across all tasksTest plan
superpowers:subagent-driven-development(recommended for 92-step plan with independent tasks)After this PR
Plan execution is the next step. Per the writing-plans skill, the choice is:
The upstream-coordination comments on MemPalace#665 and MemPalace#574 happen at the very end, after working code is on fork main (per
feedback_upstream_comment_timingfrom this session).🤖 Generated with Claude Code