Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ A Claude Code plugin that gives your AI a persistent memory system. Mine project
### Claude Code Marketplace

```bash
claude plugin marketplace add milla-jovovich/mempalace
claude plugin marketplace add MemPalace/mempalace
claude plugin install --scope user mempalace
```

Expand Down
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,5 @@
"palace",
"search"
],
"repository": "https://github.com/milla-jovovich/mempalace"
"repository": "https://github.com/MemPalace/mempalace"
}
6 changes: 3 additions & 3 deletions .codex-plugin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ codex /init
1. Clone the MemPalace repository:

```bash
git clone https://github.com/milla-jovovich/mempalace.git
git clone https://github.com/MemPalace/mempalace.git
cd mempalace
```

Expand Down Expand Up @@ -71,5 +71,5 @@ Set the `MEMPAL_DIR` environment variable to a directory path to automatically r

## Support

- Repository: https://github.com/milla-jovovich/mempalace
- Issues: https://github.com/milla-jovovich/mempalace/issues
- Repository: https://github.com/MemPalace/mempalace
- Issues: https://github.com/MemPalace/mempalace/issues
10 changes: 5 additions & 5 deletions .codex-plugin/plugin.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
"author": {
"name": "milla-jovovich"
},
"homepage": "https://github.com/milla-jovovich/mempalace",
"repository": "https://github.com/milla-jovovich/mempalace",
"homepage": "https://github.com/MemPalace/mempalace",
"repository": "https://github.com/MemPalace/mempalace",
"license": "MIT",
"keywords": [
"memory",
Expand Down Expand Up @@ -39,9 +39,9 @@
"Read",
"Write"
],
"websiteURL": "https://github.com/milla-jovovich/mempalace",
"privacyPolicyURL": "https://github.com/milla-jovovich/mempalace",
"termsOfServiceURL": "https://github.com/milla-jovovich/mempalace",
"websiteURL": "https://github.com/MemPalace/mempalace",
"privacyPolicyURL": "https://github.com/MemPalace/mempalace",
"termsOfServiceURL": "https://github.com/MemPalace/mempalace",
"defaultPrompt": [
"Search my memories for recent decisions",
"Mine this project into my memory palace",
Expand Down
69 changes: 67 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,62 @@
# Changelog

All notable changes to [MemPalace](https://github.com/milla-jovovich/mempalace) are documented in this file.
All notable changes to [MemPalace](https://github.com/MemPalace/mempalace) are documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/).

---

## [Unreleased] — v3.3.0 (on develop)

### New Features
- Closet layer — a compact searchable index of pointers to verbatim drawers, enabling fast topical lookup without reading all content (#788)
- BM25 hybrid search — closets boost ranking, drawers remain the source of truth (#795, #829)
- Entity metadata on every drawer for filterable search (#829)
- Diary ingest — day-based rooms for conversation transcripts (#829)
- Cross-wing tunnels — explicit links between rooms in different wings for multi-project agents (#829)
- Drawer-grep — returns the best-matching chunk plus adjacent context drawers (#829)
- Offline fact checker against the entity registry and knowledge graph (#829)
- LLM-based closet regeneration — optional, bring-your-own endpoint, no mandatory API key (#793)
- Hall detection — routes drawer content to `emotions` / `technical` / `family` / `memory` / `identity` / `consciousness` / `creative` halls, enabling hall-based graph connectivity within wings (#835)

### Bug Fixes
- Set `hnsw:space=cosine` metadata on all collection creation sites — fixes broken similarity scoring under ChromaDB's default L2 distance (#807, #218)
- File-level locking prevents duplicate drawers when agents mine the same file concurrently (#784, #826)
- Hybrid closet+drawer retrieval — closets boost ranking, never gate results (#795)
- Stop hooks from making agents write in chat — saves tokens on every turn (#786)
- Strip system tags, hook output, and Claude UI chrome from drawers before filing (#785)
- Verbatim-safe `strip_noise` scoped to Claude Code JSONL only (#785)
- Prevent diary entry ID collisions via microsecond timestamp and full content hash (#819)
- Auto-rebuild stale drawers via `NORMALIZE_VERSION` schema gate
- Enforce atomic topics in closets and extract richer pointers
- Sync `version.py` to match `pyproject.toml` (#820)
- Remove unused `main` import from `mempalace/__init__.py` (#827)
- README audit — fix 7 stale claims (tool count, version badge, wake-up token cost, `dialect.py` lossless disclaimer, `pyproject.toml` version) with 42 regression-guard tests (#835)

### Improvements
- Optimize entity detection with regex caching and pre-compilation (#828)
- Extract locked filing block into helper to keep `mine_convos` under C901 complexity

### Documentation
- Add `docs/CLOSETS.md` — closet layer overview
- Fix stale `milla-jovovich/*` org URLs in website and plugin manifests (#787)
- Fix remaining stale org URLs in contributor docs (#808)

### Internal
- Add test coverage for `mine_lock`, closets, entity metadata, BM25, and diary
- Verify `mine_lock` via disjoint critical-section intervals
- Serialize `mine_lock` concurrency test with multiprocessing
- Make diary state path assertion platform-neutral
- Add `TestTunnels` coverage for cross-wing tunnel operations
- Ruff format with CI-pinned version (0.4.x); format `mempalace/palace.py`

---

## [Unreleased] — v3.2.0 (on develop)
## [3.2.0] — 2026-04-12

### Packaging
- Remove `chromadb<0.7` upper bound — unblocks installs against chromadb 1.x palaces (#690)
- Bump version to 3.2.0 across `pyproject.toml`, `mempalace/version.py`, README badge, and OpenClaw SKILL (#761)

### Security
- Harden palace deletion, WAL redaction, and MCP search input handling (#739)
Expand All @@ -13,6 +65,11 @@ All notable changes to [MemPalace](https://github.com/milla-jovovich/mempalace)
- Remove global SSL verification bypass in convomem_bench (#176)

### Bug Fixes
- Parse Claude.ai privacy export with `messages` key and sender field (#685, #677)
- Detect mtime changes in `_get_client` to prevent stale HNSW index (#757)
- Hash full content in `tool_add_drawer` drawer ID — stable re-mines (#716)
- Remove 10k drawer cap from status display (#707, #603)
- Correct typo in entity_detector interactive classification prompt (#755)
- Prevent convo_miner from re-processing 0-chunk files on every run (#732, #654)
- Remove silent 8-line AI response truncation in convo_miner (#708, #692)
- Store full AI response in convo_miner exchange chunking (#695)
Expand Down Expand Up @@ -55,6 +112,7 @@ All notable changes to [MemPalace](https://github.com/milla-jovovich/mempalace)
- Add VitePress documentation site (#439)
- Add warning about fake MemPalace websites (#598)
- Fix stale org URLs and PR branch target in contributor docs (#679)
- Fix misaligned architecture diagram (#734, #733)
- Add ROADMAP.md — v3.1.1 stability patch and v4.0.0-alpha plan

### Internal
Expand Down Expand Up @@ -144,3 +202,10 @@ Initial public release.
- CLI: `init`, `mine`, `search`, `status`, `compress`, `repair`, `split`
- Benchmark suite with recall and scale tests
- README with MCP flow, local model flow, and specialist agent documentation

---

[Unreleased]: https://github.com/MemPalace/mempalace/compare/v3.2.0...HEAD
[3.2.0]: https://github.com/MemPalace/mempalace/compare/v3.1.0...v3.2.0
[3.1.0]: https://github.com/MemPalace/mempalace/compare/v3.0.0...v3.1.0
[3.0.0]: https://github.com/MemPalace/mempalace/releases/tag/v3.0.0
5 changes: 4 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,11 @@ Thanks for wanting to help. MemPalace is open source and we welcome contribution
## Getting Started

```bash
git clone https://github.com/MemPalace/mempalace.git
# Fork the repo on GitHub first, then clone your fork
git clone https://github.com/<your-username>/mempalace.git
cd mempalace
git remote add upstream https://github.com/MemPalace/mempalace.git

pip install -e ".[dev]" # installs with dev dependencies (pytest, build, twine)
```

Expand Down
42 changes: 31 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ Restart Claude Code, then type `/skills` to verify "mempalace" appears.
claude mcp add mempalace -- python -m mempalace.mcp_server
```

Now your AI has 19 tools available through MCP. Ask it anything:
Now your AI has 29 tools available through MCP. Ask it anything:

> *"What did we decide about auth last month?"*

Expand All @@ -161,7 +161,7 @@ mempalace wake-up > context.txt
# Paste context.txt into your local model's system prompt
```

This gives your local model ~170 tokens of critical facts (in AAAK if you prefer) before you ask a single question.
This gives your local model ~600-900 tokens of critical facts (in AAAK if you prefer) before you ask a single question.

**2. CLI search** — query on demand, feed results into your prompt:

Expand Down Expand Up @@ -192,10 +192,10 @@ Decisions happen in conversations now. Not in docs. Not in Jira. In conversation
|----------|--------------|-------------|
| Paste everything | 19.5M — doesn't fit any context window | Impossible |
| LLM summaries | ~650K | ~$507/yr |
| **MemPalace wake-up** | **~170 tokens** | **~$0.70/yr** |
| **MemPalace wake-up** | **~600-900 tokens** | **~$0.70/yr** |
| **MemPalace + 5 searches** | **~13,500 tokens** | **~$10/yr** |

MemPalace loads 170 tokens of critical facts on wake-up — your team, your projects, your preferences. Then searches only when needed. $10/year to remember everything vs $507/year for summaries that lose context.
MemPalace loads ~600-900 tokens of critical facts on wake-up — your team, your projects, your preferences. Then searches only when needed. $10/year to remember everything vs $507/year for summaries that lose context.

---

Expand Down Expand Up @@ -293,7 +293,7 @@ Wings and rooms aren't cosmetic. They're a **34% retrieval improvement**. The pa
| **L2** | Room recall — recent sessions, current project | On demand | When topic comes up |
| **L3** | Deep search — semantic query across all closets | On demand | When explicitly asked |

Your AI wakes up with L0 + L1 (~170 tokens) and knows your world. Searches only fire when needed.
Your AI wakes up with L0 + L1 (~600-900 tokens) and knows your world. Searches only fire when needed.

### AAAK Dialect (experimental)

Expand Down Expand Up @@ -470,7 +470,7 @@ claude plugin install --scope user mempalace
claude mcp add mempalace -- python -m mempalace.mcp_server
```

### 19 Tools
### 29 Tools

**Palace (read)**

Expand Down Expand Up @@ -508,6 +508,18 @@ claude mcp add mempalace -- python -m mempalace.mcp_server
| `mempalace_traverse` | Walk the graph from a room across wings |
| `mempalace_find_tunnels` | Find rooms bridging two wings |
| `mempalace_graph_stats` | Graph connectivity overview |
| `mempalace_create_tunnel` | Create explicit cross-wing link between two rooms |
| `mempalace_list_tunnels` | List all explicit tunnels, filter by wing |
| `mempalace_delete_tunnel` | Remove a tunnel by ID |
| `mempalace_follow_tunnels` | Follow tunnels from a room to connected rooms in other wings |

**Drawer Management**

| Tool | What |
|------|------|
| `mempalace_get_drawer` | Fetch a single drawer by ID |
| `mempalace_list_drawers` | Paginated drawer listing |
| `mempalace_update_drawer` | Update drawer content or metadata |

**Agent Diary**

Expand All @@ -516,6 +528,14 @@ claude mcp add mempalace -- python -m mempalace.mcp_server
| `mempalace_diary_write` | Write AAAK diary entry |
| `mempalace_diary_read` | Read recent diary entries |

**System**

| Tool | What |
|------|------|
| `mempalace_hook_settings` | Get/set hook behavior (silent save, toast) |
| `mempalace_memories_filed_away` | Check if recent checkpoint was saved |
| `mempalace_reconnect` | Force DB reconnect after external writes |

The AI learns AAAK and the memory protocol automatically from the `mempalace_status` response. No manual configuration.

---
Expand Down Expand Up @@ -645,12 +665,12 @@ Plain text. Becomes Layer 0 — loaded every session.
| `cli.py` | CLI entry point |
| `config.py` | Configuration loading and defaults |
| `normalize.py` | Converts 5 chat formats to standard transcript |
| `mcp_server.py` | MCP server — 19 tools, AAAK auto-teach, memory protocol |
| `mcp_server.py` | MCP server — 29 tools, AAAK auto-teach, memory protocol |
| `miner.py` | Project file ingest |
| `convo_miner.py` | Conversation ingest — chunks by exchange pair |
| `searcher.py` | Semantic search via ChromaDB |
| `layers.py` | 4-layer memory stack |
| `dialect.py` | AAAK compression — 30x lossless |
| `dialect.py` | AAAK index format for closet pointers |
| `knowledge_graph.py` | Temporal entity-relationship graph (SQLite) |
| `palace_graph.py` | Room-based navigation graph |
| `onboarding.py` | Guided setup — generates AAAK bootstrap + wing config |
Expand All @@ -669,7 +689,7 @@ mempalace/
├── README.md ← you are here
├── mempalace/ ← core package (README)
│ ├── cli.py ← CLI entry point
│ ├── mcp_server.py ← MCP server (19 tools)
│ ├── mcp_server.py ← MCP server (29 tools)
│ ├── knowledge_graph.py ← temporal entity graph
│ ├── palace_graph.py ← room navigation graph
│ ├── dialect.py ← AAAK compression
Expand All @@ -694,7 +714,7 @@ mempalace/
│ └── mcp_setup.md
├── tests/ ← test suite (README)
├── assets/ ← logo + brand assets
└── pyproject.toml ← package config (v3.0.0)
└── pyproject.toml ← package config (v3.3.0)
```

---
Expand Down Expand Up @@ -722,7 +742,7 @@ PRs welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for setup and guidelines.
MIT — see [LICENSE](LICENSE).

<!-- Link Definitions -->
[version-shield]: https://img.shields.io/badge/version-3.1.0-4dc9f6?style=flat-square&labelColor=0a0e14
[version-shield]: https://img.shields.io/badge/version-3.3.0-4dc9f6?style=flat-square&labelColor=0a0e14
[release-link]: https://github.com/milla-jovovich/mempalace/releases
[python-shield]: https://img.shields.io/badge/python-3.9+-7dd8f8?style=flat-square&labelColor=0a0e14&logo=python&logoColor=7dd8f8
[python-link]: https://www.python.org/
Expand Down
88 changes: 88 additions & 0 deletions docs/CLOSETS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Closets — The Searchable Index Layer

## What closets are

Drawers hold your verbatim content. Closets are the index — compact pointers that tell the searcher which drawers to open.

```
CLOSET: "built auth system|Ben;Igor|→drawer_api_auth_a1b2c3"
↑ topic ↑ entities ↑ points to this drawer
```

An agent searching "who built the auth?" hits the closet first (fast scan of short text), then opens the referenced drawer to get the full verbatim content.

## Lifecycle

### When are closets created?

Closets are created during `mempalace mine`. For each file mined:
1. Content is chunked into drawers (verbatim, ~800 chars each)
2. Topics, entities, and quotes are extracted from the content
3. A closet is created with pointer lines to those drawers

### What's inside a closet?

Each line is one atomic topic pointer:
```
topic description|entity1;entity2|→drawer_id_1,drawer_id_2
"verbatim quote from the content"|entity1|→drawer_id_3
```

Topics are never split across closets. If adding a topic would exceed 1,500 characters, a new closet is created.

### When do closets update?

When a file is re-mined (content changed, or `NORMALIZE_VERSION` was bumped), the miner first deletes every closet for that source file (`purge_file_closets`) and then writes a fresh set. Stale topics from the prior mine are gone — closets are always a snapshot of the current content, never an accumulation across runs.

### What about stale topics?

There are no stale topics: each re-mine is a clean rebuild for that source file. If a file gets larger and produces fewer or more closets than last time, the leftover numbered closets from the larger run are still purged because the delete is done by `source_file`, not by ID.

### Do closets survive palace rebuilds?

Closets are stored in the `mempalace_closets` ChromaDB collection alongside `mempalace_drawers`. If you delete and rebuild the palace, closets are recreated during the next `mempalace mine`.

## How search uses closets

```
Query → search mempalace_closets (fast, small documents)
↓
top closet hits → parse `→drawer_id_a,drawer_id_b` pointers
↓
fetch exactly those drawers from mempalace_drawers (verbatim content)
↓
apply max_distance filter
↓
return chunk-level results (same shape as direct search)
```

Hits carry `matched_via: "closet"` (or `"drawer"` for the fallback path) plus a `closet_preview` field showing the line that surfaced them.

If no closets exist (palace created before this feature) — or all closet hits get filtered out by `max_distance` — search falls back to direct drawer search. Closets are created on next mine.

> **BM25 hybrid re-rank** is on the roadmap (deferred to a follow-up PR alongside generic `LLM_*` env-var support); the current closet search ranks purely by ChromaDB cosine distance against the closet text.

## Limits

| Setting | Value | Reason |
|---------|-------|--------|
| Max closet size | 1,500 chars (`CLOSET_CHAR_LIMIT`) | Leaves buffer under ChromaDB's working limit |
| Source content scanned | 5,000 chars (`CLOSET_EXTRACT_WINDOW`) | Caps regex extraction cost on long files; back-of-file content is currently invisible to closet extraction (tracked for follow-up) |
| Max topics per file | 12 | Keeps closets focused |
| Max quotes per file | 3 | Most relevant only |
| Max entities per pointer | 5 | Top names by frequency, after stoplist filtering |

## For developers

Closet functions live in `mempalace/palace.py`:
- `get_closets_collection()` — get the closets ChromaDB collection
- `build_closet_lines()` — extract topics/entities/quotes into pointer lines
- `upsert_closet_lines()` — write lines to closets respecting the char limit (overwrites existing IDs; does not append — call `purge_file_closets` first when re-mining)
- `purge_file_closets()` — delete every closet for a given source file before rebuild
- `CLOSET_CHAR_LIMIT` / `CLOSET_EXTRACT_WINDOW` — size constants

The closet-first search path lives in `mempalace/searcher.py`:
- `_extract_drawer_ids_from_closet()` — parse `→drawer_a,drawer_b` pointers out of a closet document
- `_closet_first_hits()` — query closets, parse pointers, hydrate matching drawers, return chunk-level hits or `None` to fall back

Note: only the project miner (`miner.py::process_file`) builds closets today. Conversation-mined wings (Claude Code JSONL, ChatGPT export, etc.) will keep using direct drawer search via the searcher fallback until the convo-closet PR lands.
6 changes: 5 additions & 1 deletion hooks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,10 @@ Example output:
[14:40:01] Session abc123: 18 exchanges, 3 since last save
```

## Known Limitations

**Hooks require session restart after install.** Claude Code loads hooks from `settings.json` at session start only. If you run `mempalace init` or manually edit hook config mid-session, the hooks won't fire until you restart Claude Code. This is a Claude Code limitation.

## Cost

**Zero extra tokens.** The hooks are bash scripts that run locally. They don't call any API. The only "cost" is the AI spending a few seconds organizing memories at each checkpoint — and it's doing that with context it already has loaded.
**Zero extra tokens.** The hooks notify the AI that saves happened in the background — the AI doesn't need to write anything in the chat. All filing is handled automatically. Previous versions asked the AI to write diary entries and drawer content in the chat window, which cost ~$1/session in retransmitted tokens.
Loading
Loading