feat(age-integration): 6-phase mempalace knowledge base ↔ AGE integration

[jphein]

What

Six-phase plan to fully integrate the mempalace knowledge base with Apache AGE so the metaphor of the AI walking into a palace and finding wings, rooms, and drawers becomes real Cypher traversal — not a narrative device.

Phase	What	Commit
1	KnowledgeGraphAGE API parity (add_entity, invalidate, query_entity, query_relationship, timeline, seed_from_entity_facts)	ff7187d
2	Write-through middleware on PostgresCollection.add/upsert — every drawer write extracts entities + creates :MENTIONS edges in AGE	3321d83
3	Palace structure as native AGE nodes (Wing → CONTAINS → Room → CONTAINS → Drawer, plus Wing -[SHARED_VIA]- Wing tunnels)	ff583c0
4	backfill_age — restartable, checkpointed one-shot to populate AGE from existing drawer table (~22h projected for production 274K palace)	b3f0206
6	mempalace_walk_palace MCP tool — agent-facing walk primitive (start_wing / start_room / start_entity × depth)	8022ecb
docs	FORK_CHANGELOG entries + .gemini/config.yaml	07f63ba, c35c74e

Phase 5 (read-side fusion in palace-daemon /search) lives on techempower-org/palace-daemon:feat/age-fused-search.

Why

The 2026-05-17 AGE write-through spike on n=200 git-derived probes showed graph signal adds +9pp R@5 over vector-only (graph_only beats vector by +5pp; RRF fusion adds another +4pp on top). The architecture before this PR couldn't realize that lift because:

1. KnowledgeGraphAGE was skeleton-only — 5 methods, no query surface (independently confirmed by nakata-app on MemPalace/mempalace/discussions/1384#discussioncomment-16951226)
2. AGE graph state was unpopulated — 2 placeholder vertices + 1 placeholder edge total
3. No write path connected drawer writes to entity-graph population
4. No read path consulted AGE during search

This PR fixes (1), (2), (3), and (6). Companion PR on techempower-org/palace-daemon fixes (4).

Unified graph schema

Wing  -[:CONTAINS]->  Room  -[:CONTAINS]->  Drawer  -[:MENTIONS]->  Entity
  ↑                                                                    ↑
  +-[:SHARED_VIA {via_room}]- Wing (tunnels)        (auto-populated on every write)

An agent now walks the palace via Cypher:

walk_palace(start_wing="memorypalace", depth=3)
# → [{wing, room, drawer, entity}] rows for every reachable leaf

AGE Cypher dialect gaps documented

Three real limitations in AGE 1.6.0 we worked around during the build:

1. No multi-column RETURN inside cypher(...) dollar-quoted form — workaround: return one column, query twice for additional projections
2. No list literals (RETURN [a, b] errors) — workaround: query columns one at a time
3. No MERGE ... ON CREATE SET, no SET on edge properties inline, no coalesce() in SET — workaround: truncate fresh + plain CREATE in three passes (entities → drawers → edges)

All three are encoded in mempalace/knowledge_graph_age.py workarounds.

Test plan

• Smoke-tested each phase on sme_lme_bench against the production postgres+pgvector+AGE stack
• backfill_age validated on sme_lme_bench (5344 drawers, 58948 entities created, 26 min, 0 errors → projects to ~22h for prod 274K palace)
• AGE write-through bench on n=200 git-probes: graph_only beats vector by +5pp R@5; fusion (RRF) adds +4pp on top
• mempalace_walk_palace returns navigable structured walks across all three modes (start_wing / start_room / start_entity)
• Full pytest suite — should pass (no changes to existing code paths beyond the additive write-through hook in PostgresCollection._insert_rows, which is gated by set_kg_writethrough(hook) registration; default is zero overhead)
• Gemini Code Assist review

Operational rollout

Backfill against mempalace_2026_05_13 (production palace) takes ~22h overnight. Phases 1-3 + 6 are safe to deploy without backfill — the write-through middleware adds new graph state as drawers come in, and walk_palace returns empty walks gracefully when graph isn't populated yet. Phase 4 backfill can run independently when ready.

Cross-references

• Companion PR: techempower-org/palace-daemon — /search/age-fused endpoint
• Spike result: techempower-org/multipass-structural-memory-eval@28ae3f1
• Independent cross-fork audit (nakata-app side): MemPalace/mempalace/discussions/1384#discussioncomment-16951226

🤖 Generated with Claude Code

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive integration of Apache AGE for graph-based knowledge management and a research feature for multi-encoder retrieval using Reciprocal Rank Fusion (RRF). Key additions include write-through middleware for real-time KG population, a backfill utility, and an MCP tool for palace traversal. Feedback highlights critical security vulnerabilities regarding SQL injection in Cypher and SQL queries. Additionally, high-priority improvements are needed for fragile parameter replacement logic, performance bottlenecks due to excessive database commits in loops, and the use of unstable hashing for cross-process identity mapping. Other suggestions focus on optimizing database connection reuse and removing redundant JSON parsing.

[gemini-code-assist]

The use of f-string interpolation to place cypher_inlined inside dollar-quoted tags ($$...$$) is vulnerable to SQL injection. If any parameter value contains the string $$, it will terminate the dollar quoting prematurely and allow execution of arbitrary SQL. Use standard parameter binding (%s) for the Cypher query string itself.

Suggested change

	cur.execute(
	f"SELECT * FROM cypher(%s, $${cypher_inlined}$$) AS (v agtype)",
	(self.GRAPH_NAME,),
	)
	cur.execute(
	"SELECT * FROM cypher(%s, %s) AS (v agtype)",
	(self.GRAPH_NAME, cypher_inlined),
	)

[gemini-code-assist]

The manual parameter inlining logic using replace is fragile and susceptible to recursive replacement bugs. If a parameter value contains a string that matches another parameter's placeholder (e.g., $id), it will be incorrectly replaced during the loop. This can lead to malformed Cypher queries or data corruption. Use a single-pass replacement strategy with a regular expression.

Suggested change

	for key in sorted(params.keys(), key=len, reverse=True):
	cypher_inlined = cypher_inlined.replace(f"${key}", _cypher_literal(params[key]))
	cypher_inlined = re.sub(
	r"\$(\w+)",
	lambda m: _cypher_literal(params.get(m.group(1))),
	cypher
	)

[gemini-code-assist]

Direct interpolation of table_name into the SQL string is vulnerable to SQL injection. Even with double quotes, a malicious table name could break out of the identifier. Use psycopg2.sql.Identifier to safely handle dynamic table names.

Suggested change

	sql_drawers = f"""
	SELECT id, document, wing, room FROM "{table_name}"
	WHERE document IS NOT NULL
	"""
	from psycopg2 import sql
	sql_drawers = sql.SQL("""
	SELECT id, document, wing, room FROM {}
	WHERE document IS NOT NULL
	""").format(sql.Identifier(table_name))

[gemini-code-assist]

Executing _run_cypher for every drawer in a loop is extremely inefficient because KnowledgeGraphAGE._run_cypher performs a database commit on every call. For a large palace (e.g., 274K drawers), this results in massive overhead and explains the projected 22-hour runtime. Consider implementing a batching mechanism or wrapping the entire loop in a single transaction by managing the connection state externally.

[gemini-code-assist]

Python's built-in hash() function is not stable across process restarts (due to hash randomization) or between different processes. This can cause identity mismatches in a multi-process environment where palace-daemon and the MCP server are separate processes. Use a stable cryptographic hash like hashlib.sha256 for consistent identity mapping.

Suggested change

	return f"doc:{hash(doc)}"
	import hashlib
	doc = payload.get("document") or ""
	return f"doc:{hashlib.sha256(doc.encode()).hexdigest()[:16]}"

[gemini-code-assist]

Directly instantiating KnowledgeGraphAGE bypasses the connection cache implemented in _get_kg(). This results in a new database connection being established for every call to tool_walk_palace. Use the existing _get_kg() helper to reuse connections.

Suggested change

	from .knowledge_graph_age import KnowledgeGraphAGE
	kg = KnowledgeGraphAGE(dsn)
	try:
	kg = _get_kg()

[gemini-code-assist]

Re-parsing metadata_json back into a dictionary is redundant and inefficient here, as the original metadata dictionaries were already available in the metadatas list before being serialized for the database insert. You should use the original metadata objects instead of re-parsing them from the serialized row data.

Suggested change

	hook = getattr(self, "_kg_writethrough", None)
	if hook is not None:
	for row in rows:
	doc_id, document, metadata_json = row[2], row[3], row[5]
	try:
	metadata = json.loads(metadata_json) if metadata_json else {}
	except (ValueError, TypeError):
	metadata = {}
	try:
	hook = getattr(self, "_kg_writethrough", None)
	if hook is not None:
	meta_map = {did: m for did, m in zip(ids, metadatas)} if metadatas else {}
	for row in rows:
	doc_id, document = row[2], row[3]
	metadata = meta_map.get(doc_id, {})
	try: