fix(hooks): auto-mine transcripts in convos mode with stable wing

[kasparas-anusauskas]

Problem

The plugin's Stop/PreCompact hooks auto-mine Claude Code (and Codex) session transcripts on every save trigger, but the invocation is hardcoded as mempalace mine <dir> with no --mode flag. The CLI default is projects mode, which treats the JSONL as a source file.

The primary defect is chunk quality. Projects mode runs the project-file chunker over the JSONL, producing 800-char sliding windows over raw {"role": "…", "content": "…"} records. Drawers cut mid-JSON and mid-sentence:

}, {"role": "user", "content": "Can you help me understand how\nthe system works?"}, {"role": "assistant", "content": "S

These drawers are BM25-findable but their embeddings represent structural boilerplate, and when they surface in search results the returned text is unreadable — users can't reconstruct what was actually said. That defeats the point of storing the transcript.

Convos mode chunks via chunk_exchanges after normalizing the JSONL into >-prefixed turn markers, emitting one semantically whole Q+A exchange per drawer. That's the chunk-level quality difference this PR delivers.

Secondary cleanup also fixed:

• Wing name. With no mempalace.yaml in the transcript folder, projects mode falls back to the encoded-path directory name — producing wings like -Users-you-Projects-my_app. This PR defaults to the stable conversations wing.
• Room classification. Projects mode without a yaml puts every drawer in room:general. Convos mode's detect_convo_room picks a topical room per session from {technical, architecture, planning, decisions, problems}. Note on room-per-session semantics: detect_convo_room is called once per JSONL file (convo_miner.py:442-446), so all drawers from a single session share one room by design. Per-chunk room routing requires --extract general (exposed here via the MEMPAL_EXTRACT env var).

Evidence (from a user's ~/.mempalace/hook_state/hook.log on current develop @ 32ec74d)

[19:28:59] TRIGGERING SAVE at exchange 80
  No mempalace.yaml found in /Users/…/.claude/projects/-Users-…-Projects-mempalace
  — using auto-detected defaults (wing='-Users-…-Projects-mempalace').

  MemPalace Mine
  Wing:    -Users-kasparasanusauskas-Projects-mempalace
  Rooms:   general
  Drawers filed: 1322
  By room: general 3 files

Four independent signals confirm projects-mode fired: the mempalace.yaml not found warning only fires in projects mode, Rooms: general is the projects-mode single-room fallback, the source files are .jsonl session transcripts, and the drawer count matches the orphan wing exactly.

Prior art

PR #633 ("feat: two-layer hook capture, auto-mine transcripts", @jphein) proposed the same fix, but for the pre-migration standalone bash hooks (hooks/mempal_save_hook.sh). It was closed on 2026-04-12 as "superseded by recently merged PRs to develop" — but the supersession was architectural only. PRs #840 and #863 moved hook logic from bash into mempalace/hooks_cli.py and carried the infrastructure forward but not the correctness fixes:

• No --mode convos flag on the auto-mine command anywhere in the plugin codepath on develop.
• No stable-wing default ("conversations" in feat: two-layer hook capture, auto-mine transcripts, hook settings #633) was carried forward.
• grep "wing.*conversations\|chunk_exchanges\|PYMINE" hooks/mempal_save_hook.sh on develop returns zero matches.

This PR completes that migration — brings the correctness payload onto the hooks_cli.py codepath that the installed plugin actually uses.

Changes

mempalace/hooks_cli.py

• _get_mine_target(transcript_path) returns (mine_dir, source) where source is "mempal_dir", "transcript", or "". Callers now know whether the mine dir came from a user-configured MEMPAL_DIR or was derived from the harness's transcript path.

• _get_mine_dir() kept as a thin wrapper over _get_mine_target for backward compatibility (tests and external callers unaffected).

• _build_mine_cmd(mine_dir, source) builds the mempalace mine argv, driven by three new optional env vars:

  Env var	Values	Default	Effect
  MEMPAL_MODE	projects / convos / auto	auto	Auto picks convos for transcript-sourced mines, projects for MEMPAL_DIR-sourced.
  MEMPAL_WING	any string	conversations (transcript source only)	Explicit wing override.
  MEMPAL_EXTRACT	exchange / general	miner default (exchange)	exchange → one room per file via detect_convo_room; general → per-chunk rooms via general_extractor.memory_type.

  Auto-detect rule (default):

  • source=transcript → --mode convos --wing conversations (exchange-pair chunking, stable wing)
  • source=mempal_dir → --mode projects (wing left to the miner, preserving existing behavior)

• Unknown values for MEMPAL_MODE / MEMPAL_EXTRACT fall back to auto-detect / miner-default with a warning in hook.log. They are never silently dropped — silent-dropping --mode would let the CLI's projects default reassert and reintroduce the original bug.

• Both auto-mine spawn sites (_maybe_auto_ingest, _mine_sync) now use _get_mine_target + _build_mine_cmd.

mempalace/cli.py

• Shared constants MINE_MODES and MINE_EXTRACTS added at module level as the single source of truth. add_argument now uses choices=list(MINE_MODES) / list(MINE_EXTRACTS). hooks_cli._build_mine_cmd imports the same tuples. Adding a new mode to the CLI propagates automatically to the hook validator.

tests/test_hooks_cli.py

• 12 new tests covering _get_mine_target (all four branches) and _build_mine_cmd (auto-detect for each source, every env-var override, invalid-value fallback + log-warning emission, extract-only-in-convos).
• One existing test (test_precompact_mines_transcript_dir) updated: it asserted mine_dir was the last argv entry, which no longer holds now that the command grows flags. It now locates mine_dir by position after "mine".

Compatibility

• MEMPAL_DIR behavior unchanged by default. Users who point MEMPAL_DIR at a project directory keep projects-mode and existing wing-derivation rules.
• _get_mine_dir() signature preserved, so PRs in flight that import it (e.g. fix: harden hooks, MCP server, and config (security audit) #893) continue to compile.
• Transcript auto-mines on upgrade: the first post-upgrade mine of a live session's .jsonl will delete-and-reinsert its drawers under the new conversations wing via the existing file_already_mined + mtime-rebuild path (convo_miner.py:320-333). Pre-existing drawers under ugly wings on static/old sessions remain in place — documented behavior of the purge-on-change pattern, not new to this PR.

Relationship to other PRs

• Complementary with feat: deterministic hook saves — zero data loss via silent Python API #673 ("deterministic hook saves", open) — that PR adds a parallel _ingest_transcript Python-API path and does not modify _maybe_auto_ingest / _spawn_mine. This PR fixes the existing subprocess path; the two can land in either order without conflict.
• Does not conflict with fix: harden hooks, MCP server, and config (security audit) #893 (hook path-traversal hardening) — that PR operates on _is_safe_transcript_path and leaves the spawn sites alone.
• Follows feat(hooks): tune stop-hook cadence via env vars (MEMPAL_SAVE_INTERVAL / MEMPAL_STOP_HOOK_DISABLE) #985's pattern — env-var-driven hook configuration.

Verification

Primary benefit — chunk quality

Before (projects mode on JSONL, 800-char windows over structural noise):

}, {"role": "user", "content": "Can you help me understand how\nthe system works?"}, {"role": "assistant", "content": "S

After (convos mode with exchange-pair chunking):

> Can you help me understand how the system works?

Sure, here's how it works: [full coherent response preserved in one drawer]

Secondary — wing and room

Before (current develop):

Wing:    -Users-kasparasanusauskas-Projects-mempalace
Rooms:   general

After (this branch, default --extract exchange):

Wing:    conversations
Room:    one of { technical | architecture | planning | decisions | problems }
         — detect_convo_room is called once per JSONL (convo_miner.py:442-446),
           so all drawers from a single session share one room. This is by
           design for the default extractor. Set MEMPAL_EXTRACT=general for
           per-chunk rooms (decision / preference / milestone / problem / emotional).
Chunks:  exchange pairs (one > turn + AI response per drawer) instead of raw
         800-char windows over JSONL structural noise.

Test + lint

Tests: pytest tests/ --ignore=tests/benchmarks -q → 1047 passed.
Lint: ruff format --check + ruff check → clean.

Credit

Approach (convos mode + stable conversations wing default) follows @jphein's design in #633. This PR brings it onto the current hooks_cli.py architecture that #840 / #863 established.

[igorls]

Hi, thanks for the contribution.

This PR has merge conflicts with develop, and the branch has not been updated in over 7 days, which puts it before our most recent release. The conflicts are likely against work that landed in that release.

Could you rebase onto develop so we can take another look?

If this change is no longer relevant, feel free to close the PR.

(This message is part of a periodic backlog pass, sent to all open PRs that match this state.)

[jphein]

Operator follow-up — we landed an architecturally adjacent change in our techempower-org/palace-daemon fork yesterday (2026-05-14) that supersedes the hook side of what this PR is fixing. Sharing in case the design point is useful.

What we did

Same problem class — hooks were producing low-quality drawers because of how they wrote — but we took it a step further by collapsing the two parallel write paths into one.

Before:

• Hook calls mempalace_diary_write MCP tool → 1 summary drawer per session
• Hook ALSO calls mempalace mine <dir> (no --mode) → N chunked drawers per file with projects-mode chunking

Both paths emit drawers, with different wing/room/ID conventions, different metadata fields, different chunking behavior.

After (our Phase 1D refactor):

• Hook is trigger-only — it computes cadence, picks a wing, POSTs /mine to palace-daemon with --mode convos
• Miner is the sole writer — all drawer-shape decisions live there
• mempalace_diary_write becomes a deprecated legacy shim calling miner internally

So all your fixes here (--mode convos, stable wing, room classification) become invariants enforced by the single write path rather than per-hook configuration. Hook hosts that haven't been updated automatically get the right behavior because they don't decide anything — the miner does.

Wing derivation specifically

We also tightened wing derivation. Our hook.py _project_wing does:

1. Explicit --wing flag
2. mempalace.yaml in project dir
3. Claude Code transcript path: strip ~/.claude/projects/, strip leading -, replace - → /, take final basename → slug-normalize. This handles the hyphenated-project-name case from fix(hooks): correct wing extraction for hyphenated project names (#1410) #1424 generally rather than as a special case.
4. cwd basename → slug-normalize
5. fallback: "unknown" (not conversations — conversations collides with real wings on multi-machine setups)

Trade-offs

The "single write path" approach is more invasive than just adding --mode convos to the hook. We accepted that cost because we were also adding canonical-room taxonomy enforcement at the same time, which similarly needs a single point of validation. If your design is "hook + miner stay separate", the change here is the right shape for your codebase.

Happy to share the full refactor diff if useful. Operator details in our changelog at familiar.realm.watch CHANGELOG.md.