feat: batch writes, concurrent mining, MCP tools, hooks, export, search improvements

.claude-plugin/hooks/mempal-precompact-hook.sh

@@ -1,5 +1,20 @@
 #!/bin/bash
 # MemPalace PreCompact Hook — thin wrapper calling Python CLI
 # All logic lives in mempalace.hooks_cli for cross-harness extensibility
+#
+# Python resolution order:
+#   1. MEMPALACE_PYTHON env var (user override)
+#   2. Plugin root's venv (development installs)
+#   3. System python3 (pip install --user / pipx)
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+PLUGIN_ROOT="$(dirname "$SCRIPT_DIR")"
+
+if [ -n "$MEMPALACE_PYTHON" ] && [ -x "$MEMPALACE_PYTHON" ]; then
+    PYTHON="$MEMPALACE_PYTHON"
+elif [ -x "$PLUGIN_ROOT/venv/bin/python3" ]; then
+    PYTHON="$PLUGIN_ROOT/venv/bin/python3"
+else
+    PYTHON="python3"
+fi
 INPUT=$(cat)
-echo "$INPUT" | python3 -m mempalace hook run --hook precompact --harness claude-code
+echo "$INPUT" | "$PYTHON" -m mempalace hook run --hook precompact --harness claude-code

.claude-plugin/hooks/mempal-stop-hook.sh

@@ -1,5 +1,20 @@
 #!/bin/bash
 # MemPalace Stop Hook — thin wrapper calling Python CLI
 # All logic lives in mempalace.hooks_cli for cross-harness extensibility
+#
+# Python resolution order:
+#   1. MEMPALACE_PYTHON env var (user override)
+#   2. Plugin root's venv (development installs)
+#   3. System python3 (pip install --user / pipx)
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+PLUGIN_ROOT="$(dirname "$SCRIPT_DIR")"
+
+if [ -n "$MEMPALACE_PYTHON" ] && [ -x "$MEMPALACE_PYTHON" ]; then
+    PYTHON="$MEMPALACE_PYTHON"
+elif [ -x "$PLUGIN_ROOT/venv/bin/python3" ]; then
+    PYTHON="$PLUGIN_ROOT/venv/bin/python3"
+else
+    PYTHON="python3"
+fi
 INPUT=$(cat)
-echo "$INPUT" | python3 -m mempalace hook run --hook stop --harness claude-code
+echo "$INPUT" | "$PYTHON" -m mempalace hook run --hook stop --harness claude-code

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "mempalace",
-  "version": "3.0.14",
+  "version": "3.1.0",
   "description": "Give your AI a memory — mine projects and conversations into a searchable palace. 19 MCP tools, auto-save hooks, and guided setup.",
   "author": {
     "name": "milla-jovovich"

.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "mempalace",
-  "version": "3.0.14",
+  "version": "3.1.0",
   "description": "Give your AI a memory — mine projects and conversations into a searchable palace. 19 MCP tools, auto-save hooks, and guided setup.",
   "author": {
     "name": "milla-jovovich"

AGENTS.md

@@ -76,3 +76,5 @@ Knowledge Graph:
 - **Modifying mining**: `mempalace/miner.py` (project files) or `mempalace/convo_miner.py` (transcripts)
 - **Input validation**: `mempalace/config.py` — `sanitize_name()` / `sanitize_content()`
 - **Tests**: mirror source structure in `tests/test_<module>.py`
+- **Changing transcript normalization**: `mempalace/normalize.py` — format detection, tool_use/tool_result extraction
+- **Hook auto-mining**: `hooks/mempal_save_hook.sh` and `hooks/mempal_precompact_hook.sh` — inline Python for transcript mining

CLAUDE.md

@@ -0,0 +1,62 @@
+# CLAUDE.md — memorypalace
+
+## What This Is
+
+JP's fork of [milla-jovovich/mempalace](https://github.com/milla-jovovich/mempalace) — a local AI memory system using ChromaDB for verbatim storage and semantic search.
+
+- **Fork**: `jphein/mempalace` (origin) / `milla-jovovich/mempalace` (upstream)
+- **Version**: 3.1.0 + local fixes
+- **Python**: venv at `./venv/`, editable install with dev deps
+- **Palace data**: `~/.mempalace/palace` (ChromaDB) + `~/.mempalace/config.json`
+
+## Key Files
+
+- `~/Projects/mempalace.yaml` — **do not delete**. Mining config with wing/room definitions. Regenerate with `mempalace init ~/Projects --yes` if lost.
+- `~/.mempalace/config.json` — topic wings and hall keywords, customized for JP's domains (infrastructure, development, tools, creative, projects, system).
+- `~/.mempalace/palace/` — ChromaDB vector store. The actual data.
+- `~/.mempalace/hook_state/` — stop hook session tracking.
+
+## Development
+
+```bash
+source venv/bin/activate
+python -m pytest tests/ -x -q           # run tests (648 expected)
+mempalace status                         # check palace state
+mempalace search "query"                 # test search
+python -m mempalace.mcp_server           # run MCP server standalone
+```
+
+Ruff for linting (`ruff check`), line length 100, target Python 3.9.
+
+## Fork Changes (ahead of upstream)
+
+1. **fix: epsilon mtime comparison** — `palace.py` uses `abs() < 0.01` instead of `==` for float mtime dedup
+2. **feat: bulk_check_mined()** — paginated pre-fetch of all source_file/mtime pairs
+3. **fix: MCP server** — search limit capped [1,100], status/taxonomy tools paginated past 10K, duplicate cache decls removed
+4. **perf: batch ChromaDB writes** — one upsert per file instead of per chunk in both miners
+5. **fix: entity detector STOPWORDS** — 73 technical terms added (Handler, Node, Service, etc.)
+6. **feat: similarity threshold** — `max_distance` parameter in search (renamed from `min_similarity`), default 1.5 L2 distance in MCP
+7. **feat: hooks_cli** — stop hook saves directly via Python API with systemMessage notification, precompact blocks for AI-driven save, auto-ingest transcripts
+8. **feat: --version flag** — CLI supports `mempalace --version` (from upstream PR #559 pattern)
+9. **feat: tool output mining** — normalize.py captures tool_use/tool_result blocks from Claude Code JSONL with per-tool formatting strategies (Bash head+tail, Read/Edit/Write path-only, Grep/Glob capped)
+10. **fix: dry-run room=None crash** — miner.py handles None room from unreadable files (#586)
+11. **fix: precompact hook SESSION_ID sanitization** — applies same safe() regex as save hook (#589)
+12. **feat: chromadb >=1.5.4** — upgraded from 0.6.x pin, auto-migrates existing databases (#581)
+13. **feat: hooks auto-mine transcript** — both hooks now auto-mine JSONL transcript into palace (captures raw tool output), updated reason messages to request verbatim tool output, MP_PYTHON auto-detection
+
+## Upstream PRs
+
+- milla-jovovich/mempalace#483 — mtime dedup fix
+- milla-jovovich/mempalace#484 — search limit + pagination + cache fix
+- milla-jovovich/mempalace#562 — tool output mining, bug fixes, chromadb upgrade (18 upstream issues addressed)
+
+## Integration
+
+- **Claude Code plugin**: installed at user scope via marketplace
+- **MCP server**: global user scope — available in all projects
+- **Stop hook**: fires every 15 messages, auto-mines transcript for tool output, blocks AI to save with verbatim tool output instructions
+- **PreCompact hook**: emergency save before context compaction, auto-mines transcript, finds transcript by session_id fallback
+
+## Testing
+
+Always run `python -m pytest tests/ -x -q` after changes. 648 tests expected to pass. Benchmark and stress tests are excluded by default (use `-m benchmark` or `-m stress` to include).

README.md

@@ -512,9 +512,9 @@ The AI learns AAAK and the memory protocol automatically from the `mempalace_sta
 
 Two hooks for Claude Code that automatically save memories during work:
 
-**Save Hook** — every 15 messages, triggers a structured save. Topics, decisions, quotes, code changes. Also regenerates the critical facts layer.
+**Save Hook** — Every 15 messages, auto-mines the JSONL transcript directly into the palace (capturing raw tool output — Bash results, search findings, build errors), then triggers a structured AI save with explicit instructions to preserve tool output verbatim.
 
-**PreCompact Hook** — fires before context compression. Emergency save before the window shrinks.
+**PreCompact Hook** — Fires before context compression. Auto-mines the transcript, then forces emergency save with verbatim tool output instructions.
 
 ```json
 {
@@ -527,6 +527,8 @@ Two hooks for Claude Code that automatically save memories during work:
 
 **Optional auto-ingest:** Set the `MEMPAL_DIR` environment variable to a directory path and the hooks will automatically run `mempalace mine` on that directory during each save trigger (background on stop, synchronous on precompact).
 
+Set `MEMPAL_PYTHON` to your Python interpreter with mempalace + chromadb installed. Auto-detects repo venv if not set.
+
 ---
 
 ## Benchmarks
@@ -632,7 +634,7 @@ Plain text. Becomes Layer 0 — loaded every session.
 |------|------|
 | `cli.py` | CLI entry point |
 | `config.py` | Configuration loading and defaults |
-| `normalize.py` | Converts 5 chat formats to standard transcript |
+| `normalize.py` | Converts 6 chat formats to standard transcript, captures tool_use/tool_result blocks from Claude Code JSONL |
 | `mcp_server.py` | MCP server — 19 tools, AAAK auto-teach, memory protocol |
 | `miner.py` | Project file ingest |
 | `convo_miner.py` | Conversation ingest — chunks by exchange pair |
@@ -645,8 +647,8 @@ Plain text. Becomes Layer 0 — loaded every session.
 | `entity_registry.py` | Entity code registry |
 | `entity_detector.py` | Auto-detect people and projects from content |
 | `split_mega_files.py` | Split concatenated transcripts into per-session files |
-| `hooks/mempal_save_hook.sh` | Auto-save every N messages |
-| `hooks/mempal_precompact_hook.sh` | Emergency save before compaction |
+| `hooks/mempal_save_hook.sh` | auto-mine + save every N messages |
+| `hooks/mempal_precompact_hook.sh` | auto-mine + emergency save |
 
 ---
 
@@ -674,23 +676,23 @@ mempalace/
 │   └── membench_bench.py      ← MemBench runner
 ├── hooks/                     ← Claude Code auto-save hooks
 │   ├── README.md              ← hook setup guide
-│   ├── mempal_save_hook.sh    ← save every N messages
-│   └── mempal_precompact_hook.sh ← emergency save
+│   ├── mempal_save_hook.sh    ← auto-mine + save every N messages
+│   └── mempal_precompact_hook.sh ← auto-mine + emergency save
 ├── examples/                  ← usage examples
 │   ├── basic_mining.py
 │   ├── convo_import.py
 │   └── mcp_setup.md
 ├── tests/                     ← test suite (README)
 ├── assets/                    ← logo + brand assets
-└── pyproject.toml             ← package config (v3.0.0)
+└── pyproject.toml             ← package config (v3.1.0)
 ```
 
 ---
 
 ## Requirements
 
 - Python 3.9+
-- `chromadb>=0.4.0`
+- `chromadb>=1.5.4`
 - `pyyaml>=6.0`
 
 No API key. No internet after install. Everything local.