feat: two-layer hook capture, auto-mine transcripts, hook settings

.claude-plugin/hooks/mempal-precompact-hook.sh

@@ -1,5 +1,20 @@
 #!/bin/bash
 # MemPalace PreCompact Hook — thin wrapper calling Python CLI
 # All logic lives in mempalace.hooks_cli for cross-harness extensibility
+#
+# Python resolution order:
+#   1. MEMPALACE_PYTHON env var (user override)
+#   2. Plugin root's venv (development installs)
+#   3. System python3 (pip install --user / pipx)
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+PLUGIN_ROOT="$(dirname "$SCRIPT_DIR")"
+
+if [ -n "$MEMPALACE_PYTHON" ] && [ -x "$MEMPALACE_PYTHON" ]; then
+    PYTHON="$MEMPALACE_PYTHON"
+elif [ -x "$PLUGIN_ROOT/venv/bin/python3" ]; then
+    PYTHON="$PLUGIN_ROOT/venv/bin/python3"
+else
+    PYTHON="python3"
+fi
 INPUT=$(cat)
-echo "$INPUT" | python3 -m mempalace hook run --hook precompact --harness claude-code
+echo "$INPUT" | "$PYTHON" -m mempalace hook run --hook precompact --harness claude-code

.claude-plugin/hooks/mempal-stop-hook.sh

@@ -1,5 +1,20 @@
 #!/bin/bash
 # MemPalace Stop Hook — thin wrapper calling Python CLI
 # All logic lives in mempalace.hooks_cli for cross-harness extensibility
+#
+# Python resolution order:
+#   1. MEMPALACE_PYTHON env var (user override)
+#   2. Plugin root's venv (development installs)
+#   3. System python3 (pip install --user / pipx)
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+PLUGIN_ROOT="$(dirname "$SCRIPT_DIR")"
+
+if [ -n "$MEMPALACE_PYTHON" ] && [ -x "$MEMPALACE_PYTHON" ]; then
+    PYTHON="$MEMPALACE_PYTHON"
+elif [ -x "$PLUGIN_ROOT/venv/bin/python3" ]; then
+    PYTHON="$PLUGIN_ROOT/venv/bin/python3"
+else
+    PYTHON="python3"
+fi
 INPUT=$(cat)
-echo "$INPUT" | python3 -m mempalace hook run --hook stop --harness claude-code
+echo "$INPUT" | "$PYTHON" -m mempalace hook run --hook stop --harness claude-code

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "mempalace",
-  "version": "3.0.14",
+  "version": "3.1.0",
   "description": "Give your AI a memory — mine projects and conversations into a searchable palace. 19 MCP tools, auto-save hooks, and guided setup.",
   "author": {
     "name": "milla-jovovich"

.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "mempalace",
-  "version": "3.0.14",
+  "version": "3.1.0",
   "description": "Give your AI a memory — mine projects and conversations into a searchable palace. 19 MCP tools, auto-save hooks, and guided setup.",
   "author": {
     "name": "milla-jovovich"

examples/HOOKS_TUTORIAL.md

@@ -25,4 +25,27 @@ Add this to your configuration file to enable automatic background saving:
       }
     ]
   }
-}
+}
+```
+
+### 3. What changed (v3.1.0+)
+
+Both hooks now have **two-layer capture**:
+
+1. **Auto-mine**: Before blocking the AI, the hook runs the normalizer on the JSONL transcript and upserts chunks directly into the palace. This captures raw tool output (Bash results, search findings, build errors) that the AI would otherwise summarize away.
+
+2. **Updated reason messages**: The block reason now explicitly tells the AI to save tool output verbatim — not just topics and decisions.
+
+### 4. Backfill past conversations (one-time)
+
+The hooks capture conversations going forward, but you probably have months of past sessions. Run this once to mine them all:
+
+```bash
+mempalace mine ~/.claude/projects/ --mode convos
+```
+
+### 5. Configuration
+
+- **`SAVE_INTERVAL=15`** — How many human messages between saves
+- **`MEMPALACE_PYTHON`** — Python interpreter with mempalace + chromadb. Auto-detects: env var → repo venv → system python3
+- **`MEMPAL_DIR`** — Optional directory for auto-ingest via `mempalace mine`

hooks/README.md

@@ -6,10 +6,10 @@ These hook scripts make MemPalace save automatically. No manual "save" commands
 
 | Hook | When It Fires | What Happens |
 |------|--------------|-------------|
-| **Save Hook** | Every 15 human messages | Blocks the AI, tells it to save key topics/decisions/quotes to the palace |
-| **PreCompact Hook** | Right before context compaction | Emergency save — forces the AI to save EVERYTHING before losing context |
+| **Save Hook** | Every 15 human messages | Auto-mines transcript (tool output included), then blocks the AI to save topics/decisions/quotes |
+| **PreCompact Hook** | Right before context compaction | Auto-mines transcript, then emergency save — forces the AI to save EVERYTHING before losing context |
 
-The AI does the actual filing — it knows the conversation context, so it classifies memories into the right wings/halls/closets. The hooks just tell it WHEN to save.
+**Two-layer capture:** Hooks auto-mine the JSONL transcript directly into the palace (capturing raw tool output — Bash results, search findings, build errors). They also block the AI with a reason message telling it to save verbatim tool output and key context. Belt and suspenders — tool output gets stored even if the AI summarizes instead of quoting.
 
 ## Install — Claude Code
 
@@ -68,6 +68,7 @@ Edit `mempal_save_hook.sh` to change:
 - **`SAVE_INTERVAL=15`** — How many human messages between saves. Lower = more frequent saves, higher = less interruption.
 - **`STATE_DIR`** — Where hook state is stored (defaults to `~/.mempalace/hook_state/`)
 - **`MEMPAL_DIR`** — Optional. Set to a conversations directory to auto-run `mempalace mine <dir>` on each save trigger. Leave blank (default) to let the AI handle saving via the block reason message.
+- **`MEMPALACE_PYTHON`** — Optional env var. Python interpreter with mempalace + chromadb installed. Auto-detects: `MEMPALACE_PYTHON` env var → repo `venv/bin/python3` → system `python3`. Set this if your venv is in a non-standard location.
 
 ### mempalace CLI
 
@@ -91,15 +92,19 @@ User sends message → AI responds → Claude Code fires Stop hook
                                             ↓
                               ┌─── < 15 since last save ──→ echo "{}" (let AI stop)
                               │
-                              └─── ≥ 15 since last save ──→ {"decision": "block", "reason": "save..."}
-                                                                    ↓
-                                                            AI saves to palace
-                                                                    ↓
-                                                            AI tries to stop again
-                                                                    ↓
-                                                            stop_hook_active = true
-                                                                    ↓
-                                                            Hook sees flag → echo "{}" (let it through)
+                              └─── ≥ 15 since last save
+                                            ↓
+                                    Auto-mine transcript → palace (tool output captured)
+                                            ↓
+                                    {"decision": "block", "reason": "save tool output verbatim..."}
+                                            ↓
+                                    AI saves to palace (topics, decisions, quotes)
+                                            ↓
+                                    AI tries to stop again
+                                            ↓
+                                    stop_hook_active = true
+                                            ↓
+                                    Hook sees flag → echo "{}" (let it through)
 ```
 
 The `stop_hook_active` flag prevents infinite loops: block once → AI saves → tries to stop → flag is true → we let it through.
@@ -109,14 +114,18 @@ The `stop_hook_active` flag prevents infinite loops: block once → AI saves →
 ```
 Context window getting full → Claude Code fires PreCompact
                                         ↓
-                                Hook ALWAYS blocks
+                                Find transcript (from input or session_id lookup)
+                                        ↓
+                                Auto-mine transcript → palace (tool output captured)
+                                        ↓
+                                {"decision": "block", "reason": "save tool output verbatim..."}
                                         ↓
                                 AI saves everything
                                         ↓
                                 Compaction proceeds
 ```
 
-No counting needed — compaction always warrants a save.
+No counting needed — compaction always warrants a save. The auto-mine captures raw tool output before the AI gets a chance to summarize it away.
 
 ## Debugging
 
@@ -133,6 +142,26 @@ Example output:
 [14:40:01] Session abc123: 18 exchanges, 3 since last save
 ```
 
+## Backfill Past Conversations
+
+The hooks only capture conversations going forward. To mine **past** Claude Code sessions into your palace, run a one-time backfill:
+
+```bash
+mempalace mine ~/.claude/projects/ --mode convos
+```
+
+This scans all JSONL transcripts from previous sessions and files them into the `conversations` wing. On a typical developer machine with months of history, this can yield 50K–200K drawers.
+
+For Codex CLI sessions:
+```bash
+mempalace mine ~/.codex/sessions/ --mode convos
+```
+
+This only needs to be done once — after that, the hooks auto-mine each session as you go.
+
 ## Cost
 
-**Zero extra tokens.** The hooks are bash scripts that run locally. They don't call any API. The only "cost" is the AI spending a few seconds organizing memories at each checkpoint — and it's doing that with context it already has loaded.
+**Zero extra API tokens.** The hooks are bash scripts that run locally. They don't call any API. The auto-mining uses the local ChromaDB instance. The only "cost" is:
+- ~1-13 seconds for transcript mining (depending on session length)
+- The AI spending a few seconds organizing memories at each checkpoint — with context it already has loaded
+- ChromaDB disk space for the mined chunks (~1KB per exchange pair)

hooks/mempal_precompact_hook.sh

@@ -54,24 +54,107 @@ mkdir -p "$STATE_DIR"
 # Leave empty to skip auto-ingest (AI handles saving via the block reason).
 MEMPAL_DIR=""
 
+# Python interpreter with mempalace + chromadb installed.
+# Auto-detects: MEMPALACE_PYTHON env var → repo venv → system python3
+if [ -n "$MEMPALACE_PYTHON" ]; then
+    MP_PYTHON="$MEMPALACE_PYTHON"
+elif [ -f "$(dirname "$(dirname "${BASH_SOURCE[0]}")")/venv/bin/python3" ]; then
+    MP_PYTHON="$(dirname "$(dirname "${BASH_SOURCE[0]}")")/venv/bin/python3"
+else
+    MP_PYTHON="python3"
+fi
+
 # Read JSON input from stdin
 INPUT=$(cat)
 
-SESSION_ID=$(echo "$INPUT" | python3 -c "import sys,json; print(json.load(sys.stdin).get('session_id','unknown'))" 2>/dev/null)
+SESSION_ID=$(echo "$INPUT" | python3 -c "
+import sys, json, re
+data = json.load(sys.stdin)
+sid = data.get('session_id', 'unknown')
+safe = lambda s: re.sub(r'[^a-zA-Z0-9_/.\-~]', '', str(s))
+print(safe(sid))
+" 2>/dev/null)
 
 echo "[$(date '+%H:%M:%S')] PRE-COMPACT triggered for session $SESSION_ID" >> "$STATE_DIR/hook.log"
 
+# Also parse transcript_path if present in the input
+TRANSCRIPT_PATH=$(echo "$INPUT" | python3 -c "
+import sys, json, re
+data = json.load(sys.stdin)
+tp = data.get('transcript_path', '')
+safe = lambda s: re.sub(r'[^a-zA-Z0-9_/.\-~]', '', str(s))
+print(safe(tp))
+" 2>/dev/null)
+TRANSCRIPT_PATH="${TRANSCRIPT_PATH/#\~/$HOME}"
+
+# If no transcript_path in input, find it by session_id
+if [ -z "$TRANSCRIPT_PATH" ] || [ ! -f "$TRANSCRIPT_PATH" ]; then
+    if [ -n "$SESSION_ID" ] && [ "$SESSION_ID" != "unknown" ]; then
+        FOUND=$(find "$HOME/.claude/projects" -name "${SESSION_ID}.jsonl" -type f 2>/dev/null | head -1)
+        if [ -n "$FOUND" ]; then
+            TRANSCRIPT_PATH="$FOUND"
+        fi
+    fi
+fi
+
+# Auto-mine the transcript — captures tool output before compaction loses it
+if [ -f "$TRANSCRIPT_PATH" ]; then
+    echo "[$(date '+%H:%M:%S')] Mining transcript: $TRANSCRIPT_PATH" >> "$STATE_DIR/hook.log"
+    "$MP_PYTHON" - "$TRANSCRIPT_PATH" <<'PYMINE'
+import sys
+try:
+    import hashlib
+    from datetime import datetime
+    from mempalace.normalize import normalize
+    from mempalace.convo_miner import chunk_exchanges, detect_convo_room
+    from mempalace.palace import get_collection
+    from mempalace.config import MempalaceConfig
+    palace = MempalaceConfig().palace_path
+    content = normalize(sys.argv[1])
+    if content and len(content.strip()) >= 50:
+        collection = get_collection(palace)
+        source = sys.argv[1]
+        # No file_already_mined check — transcript grows during session.
+        # upsert is idempotent: same chunk_index → same ID → overwrite.
+        chunks = chunk_exchanges(content)
+        if chunks:
+            room = detect_convo_room(content) or "session"
+            wing = "conversations"
+            docs, ids, metas = [], [], []
+            for chunk in chunks:
+                cid = hashlib.sha256(
+                    (source + str(chunk["chunk_index"])).encode()
+                ).hexdigest()[:24]
+                docs.append(chunk["content"])
+                ids.append(f"drawer_{wing}_{room}_{cid}")
+                metas.append({
+                    "wing": wing, "room": room, "source_file": source,
+                    "chunk_index": chunk["chunk_index"],
+                    "added_by": "hook", "filed_at": datetime.now().isoformat(),
+                    "ingest_mode": "convos", "extract_mode": "exchange",
+                })
+            for i in range(0, len(docs), 100):
+                collection.upsert(
+                    documents=docs[i:i+100], ids=ids[i:i+100],
+                    metadatas=metas[i:i+100],
+                )
+except Exception:
+    pass  # Hook must never crash the AI
+PYMINE
+    >> "$STATE_DIR/hook.log" 2>&1
+fi
+
 # Optional: run mempalace ingest synchronously so memories land before compaction
 if [ -n "$MEMPAL_DIR" ] && [ -d "$MEMPAL_DIR" ]; then
     SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
     REPO_DIR="$(dirname "$SCRIPT_DIR")"
-    python3 -m mempalace mine "$MEMPAL_DIR" >> "$STATE_DIR/hook.log" 2>&1
+    "$MP_PYTHON" -m mempalace mine "$MEMPAL_DIR" >> "$STATE_DIR/hook.log" 2>&1
 fi
 
 # Always block — compaction = save everything
 cat << 'HOOKJSON'
 {
   "decision": "block",
-  "reason": "COMPACTION IMMINENT. Save ALL topics, decisions, quotes, code, and important context from this session to your memory system. Be thorough — after compaction, detailed context will be lost. Organize into appropriate categories. Use verbatim quotes where possible. Save everything, then allow compaction to proceed."
+  "reason": "COMPACTION IMMINENT — MEMPALACE SAVE REQUIRED. Use the mempalace MCP tools (mempalace_add_drawer, mempalace_diary_write) to save EVERYTHING to the memory palace. Do NOT save to your internal Claude memory (~/.claude/projects/.../memory/) — save to the MEMPALACE via MCP tools only. CRITICAL: Save tool output VERBATIM — Bash command results, probe findings, search results, build output, error messages. These are lost on compaction and exist nowhere else. Also save all topics, decisions, quotes, code, and important context. Be thorough — after compaction, detailed context will be lost. Organize into appropriate wings/rooms. Save everything, then allow compaction to proceed."
 }
 HOOKJSON

hooks/mempal_save_hook.sh

@@ -61,6 +61,16 @@ mkdir -p "$STATE_DIR"
 # Leave empty to skip auto-ingest (AI handles saving via the block reason).
 MEMPAL_DIR=""
 
+# Python interpreter with mempalace + chromadb installed.
+# Auto-detects: MEMPALACE_PYTHON env var → repo venv → system python3
+if [ -n "$MEMPALACE_PYTHON" ]; then
+    MP_PYTHON="$MEMPALACE_PYTHON"
+elif [ -f "$(dirname "$(dirname "${BASH_SOURCE[0]}")")/venv/bin/python3" ]; then
+    MP_PYTHON="$(dirname "$(dirname "${BASH_SOURCE[0]}")")/venv/bin/python3"
+else
+    MP_PYTHON="python3"
+fi
+
 # Read JSON input from stdin
 INPUT=$(cat)
 
@@ -137,15 +147,61 @@ if [ "$SINCE_LAST" -ge "$SAVE_INTERVAL" ] && [ "$EXCHANGE_COUNT" -gt 0 ]; then
     if [ -n "$MEMPAL_DIR" ] && [ -d "$MEMPAL_DIR" ]; then
         SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
         REPO_DIR="$(dirname "$SCRIPT_DIR")"
-        python3 -m mempalace mine "$MEMPAL_DIR" >> "$STATE_DIR/hook.log" 2>&1 &
+        "$MP_PYTHON" -m mempalace mine "$MEMPAL_DIR" >> "$STATE_DIR/hook.log" 2>&1 &
+    fi
+
+    # Auto-mine the transcript — captures tool output that the AI would summarize away
+    if [ -f "$TRANSCRIPT_PATH" ]; then
+        "$MP_PYTHON" - "$TRANSCRIPT_PATH" <<'PYMINE'
+import sys
+try:
+    import hashlib
+    from datetime import datetime
+    from mempalace.normalize import normalize
+    from mempalace.convo_miner import chunk_exchanges, detect_convo_room
+    from mempalace.palace import get_collection
+    from mempalace.config import MempalaceConfig
+    palace = MempalaceConfig().palace_path
+    content = normalize(sys.argv[1])
+    if content and len(content.strip()) >= 50:
+        collection = get_collection(palace)
+        source = sys.argv[1]
+        # No file_already_mined check — transcript grows during session.
+        # upsert is idempotent: same chunk_index → same ID → overwrite.
+        chunks = chunk_exchanges(content)
+        if chunks:
+            room = detect_convo_room(content) or "session"
+            wing = "conversations"
+            docs, ids, metas = [], [], []
+            for chunk in chunks:
+                cid = hashlib.sha256(
+                    (source + str(chunk["chunk_index"])).encode()
+                ).hexdigest()[:24]
+                docs.append(chunk["content"])
+                ids.append(f"drawer_{wing}_{room}_{cid}")
+                metas.append({
+                    "wing": wing, "room": room, "source_file": source,
+                    "chunk_index": chunk["chunk_index"],
+                    "added_by": "hook", "filed_at": datetime.now().isoformat(),
+                    "ingest_mode": "convos", "extract_mode": "exchange",
+                })
+            for i in range(0, len(docs), 100):
+                collection.upsert(
+                    documents=docs[i:i+100], ids=ids[i:i+100],
+                    metadatas=metas[i:i+100],
+                )
+except Exception:
+    pass  # Hook must never crash the AI
+PYMINE
+        >> "$STATE_DIR/hook.log" 2>&1
     fi
 
     # Block the AI and tell it to save
     # The "reason" becomes a system message the AI sees and acts on
     cat << 'HOOKJSON'
 {
   "decision": "block",
-  "reason": "AUTO-SAVE checkpoint. Save key topics, decisions, quotes, and code from this session to your memory system. Organize into appropriate categories. Use verbatim quotes where possible. Continue conversation after saving."
+  "reason": "MEMPALACE AUTO-SAVE checkpoint. Use the mempalace MCP tools (mempalace_add_drawer, mempalace_diary_write) to save to the memory palace. Do NOT save to your internal Claude memory (~/.claude/projects/.../memory/) — save to the MEMPALACE via MCP tools only. IMPORTANT: Save tool output VERBATIM — Bash command results, probe findings, search results, build output, error messages. These are lost on compaction and exist nowhere else. Also save key topics, decisions, and quotes. Organize into appropriate wings/rooms. Continue conversation after saving."
 }
 HOOKJSON
 else