Skip to content

feat(hooks): add hooks.transcript_mining config knob - #1940

Draft
Lcstyle wants to merge 1 commit into
MemPalace:developfrom
Lcstyle:feat/hooks-transcript-mining-knob
Draft

feat(hooks): add hooks.transcript_mining config knob#1940
Lcstyle wants to merge 1 commit into
MemPalace:developfrom
Lcstyle:feat/hooks-transcript-mining-knob

Conversation

@Lcstyle

@Lcstyle Lcstyle commented Jul 6, 2026

Copy link
Copy Markdown

Summary

Adds a hooks.transcript_mining config knob (env: MEMPALACE_HOOKS_TRANSCRIPT_MINING) that lets operators disable the in-hook transcript-mining subprocess while keeping the save-reminder / silent-diary-checkpoint behavior. Zero-breaking-change default (mining stays on).

Motivation

Multi-agent deployments (multiple concurrent Claude Code / Codex sessions filing through the same palace) currently have to choose between:

  1. Full hooks on — every hook fire can spawn a transcript-mine subprocess. Under 8+ concurrent sessions, mines pile up in the daemon queue (Question: should MemPalace recommend a single-writer/gateway pattern for multi-agent setups? #1497) and interactive MCP writes time out waiting behind them. Or, without a daemon, hooks contend directly for the palace storage lock.
  2. hooks.auto_save: false — turns off both the reminder AND the mine. Operator loses the assistant-nudge that drives interactive saves.

The gap: users want reminders without the mine. Downstream has been carrying out-of-tree patches for months:

This PR upstreams the pattern as a first-class config knob so no fork is required.

Design

  • New property MempalaceConfig.hooks_transcript_mining (defaults to True → matches pre-PR behavior).
  • Env override MEMPALACE_HOOKS_TRANSCRIPT_MINING={false,0,no,true,1,yes}.
  • Master gate hooks.auto_save=false still masks it (backwards compat: existing auto_save: false deployments stay silent).
  • Env override wins over the auto_save mask — mirrors how MEMPALACE_HOOKS_AUTO_SAVE already behaves.

Belt-and-suspenders guards short-circuit at the top of each mining entry point rather than at each call site — one source of truth, no risk of a future caller forgetting the outer gate:

  • _maybe_auto_ingest (async background MEMPAL_DIR mine)
  • _mine_sync (synchronous precompact mine)
  • _ingest_transcript (transcript → convo drawer ingest)

Composes with #1826's daemon

With mining out of the hook path, the daemon queue only holds ~200ms interactive writes — the daemon becomes viable for multi-session deployments without the mine-starvation problem that motivated the workarounds above.

Recommended multi-agent config unlocked by this PR:

{
  "hooks": {
    "auto_save": true,
    "silent_save": false,
    "transcript_mining": false,
    "daemon": true
  }
}
  • Stop/PreCompact still fire and emit save reminders
  • Assistant responds by calling mempalace_diary_write / mempalace_add_drawer
  • Interactive writes serialize through the daemon (safe under N concurrent sessions)
  • No mining subprocess is ever spawned by a hook fire
  • Bulk mining runs out-of-band via a scheduled mempalace mine --daemon when no session is active

Tests

  • Config truth table (8 tests in tests/test_config.py): default / from-config / four env overrides / auto_save-mask / env-wins-over-mask.
  • Guard tests (4 tests in tests/test_hooks_cli.py): each entry point verified as no-op when transcript_mining=false, plus a regression test that mining still fires when transcript_mining=true (default).
  • All 275 existing tests in test_hooks_cli.py + test_config.py continue to pass.
275 passed, 1 skipped in 8.80s

Test plan

  • uv run pytest tests/test_config.py tests/test_hooks_cli.py -v
  • uv run ruff check . + uv run ruff format --check . clean
  • Manual: set hooks.transcript_mining: false on a live deployment, confirm hooks fire but no _spawn_mine subprocess appears; confirm assistant still gets Stop reminder and calls MCP tools; confirm scheduled mempalace mine still populates the palace at 04:00.

Deliberately out of scope

  • PRECOMPACT_BLOCK_REASON wiring — the constant is defined at hooks_cli.py:110 but never emitted (hook_precompact at line ~1353 always does _output({})). Wiring it up would revisit Compaction Blocking #955 / PreCompact hook unconditionally blocks /compact, making it unusable #1172 (PreCompact-block-caused-deadlock) which were closed with a specific "don't block on PreCompact" fix. That's a separate design decision better handled in a follow-up PR that specifically considers those closed issues.

Related: #1497 (multi-writer safety discussion), #1826 (write daemon), #1828 (daemon hardening).

Adds `hooks.transcript_mining` (env: MEMPALACE_HOOKS_TRANSCRIPT_MINING)
to let operators disable the in-hook transcript-mining subprocess without
losing the save-reminder / silent-diary-checkpoint behavior.

Motivation
----------
In multi-agent deployments (multiple concurrent Claude Code / Codex
sessions filing through the same palace) an in-hook transcript mine can
either starve interactive MCP writes queued behind it in the daemon
(MemPalace#1497), or contend with concurrent processes for the palace storage
lock. Downstream users have been carrying out-of-tree "mine off" patches
(@anastasiiaanfimova's hooks_cli patch, @jphein's palace-daemon at the
gateway layer) for months. This upstreams the pattern as a first-class
config knob so no fork is required.

Design
------
- New property `MempalaceConfig.hooks_transcript_mining` defaults to True
  (matches pre-PR behavior).
- Env override `MEMPALACE_HOOKS_TRANSCRIPT_MINING={false,0,no,true,1,yes}`.
- Master `hooks.auto_save=false` still masks it (back-compat).
- Env override wins over the auto_save mask (mirrors how the existing
  MEMPALACE_HOOKS_AUTO_SAVE env override works).

Belt-and-suspenders guards short-circuit at the top of each mining
entry point rather than at each call site — one source of truth, no risk
of a future caller forgetting the outer gate:

- `_maybe_auto_ingest` (async background MEMPAL_DIR mine)
- `_mine_sync` (synchronous precompact mine)
- `_ingest_transcript` (transcript-into-convo-drawer ingest)

Composes with MemPalace#1826's daemon: with mining out of the hook path, the
daemon queue only holds ~200ms interactive writes and the daemon becomes
a viable single-writer for multi-session deployments without the
mine-starvation problem that motivated Choice 1 pure workarounds.

Recommended multi-agent config (unlocked by this PR)
----------------------------------------------------
    {
      "hooks": {
        "auto_save": true,
        "silent_save": false,
        "transcript_mining": false,
        "daemon": true
      }
    }

- Stop/PreCompact still fire and emit save reminders
- Assistant responds by calling `mempalace_diary_write` / `mempalace_add_drawer`
- Interactive writes serialize through the daemon (safe under N sessions)
- No mining subprocess is ever spawned by a hook fire
- Bulk mining runs out-of-band via a scheduled `mempalace mine --daemon`
  when no session is active

Tests
-----
- Config truth table (8 tests): default/from-config/env-overrides/
  auto_save-mask/env-wins-over-mask
- Guard tests (4 tests): each entry point verified as no-op when
  transcript_mining=false, plus a regression test that mining still
  fires when transcript_mining=true (default)
- All 275 existing tests in the touched files continue to pass.
@Lcstyle

Lcstyle commented Aug 22, 2026

Copy link
Copy Markdown
Author

Reporting dogfood experience with this knob, in case it helps the PR land.

We have been running this change as a local patch on several hosts since it was
opened. The operational need it addresses is real: on hosts running many
concurrent agent sessions, hook-triggered mining is the dominant write source, and
without a switch there is no way to separate "record the session" from "mine the
transcript" once the latter becomes expensive.

No regressions observed in that time. The knob defaults to existing behaviour, so
adopting it is a no-op for anyone who does not set it.

Would be glad to see this merged, and can provide before/after mine timings from a
large palace if that would help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants