fix: persistent daemon to prevent zombie processes and enable multi-session access (closes #1229)

[Vasanth19]

Problem

Closes #1229.

The current mempalace-mcp / mempalace.mcp_server model runs one Python process per agent session. In practice, developers run Claude Code, Codex, Gemini CLI, and GG simultaneously. This exposes two compounding failure modes.

Failure 1 — Zombie processes after SIGKILL

MCP hosts sometimes force-quit sessions with SIGKILL. Python's atexit and signal.signal(SIGTERM) cleanup never fires on SIGKILL, so the process exits without releasing the PID file. The next session sees a stale PID file, decides another instance is running, and refuses to start. Every MCP tool call returns "Connection closed". Manual PID-file removal is the only recovery — until it happens again.

Failure 2 — Concurrent ChromaDB writers corrupt HNSW

When multiple sessions each hold an open PersistentClient against the same chroma.sqlite3 and simultaneously call upsert(), the writes interleave at the mmap level. The in-memory HNSW tree and on-disk sqlite metadata diverge — exactly the divergence that #1222 detects but cannot prevent. The only safe fix is a single process owning the ChromaDB connection.

Solution — Daemon + Bridge Architecture

macOS LaunchAgent
  └── mempalace-daemon.py   (one process, holds ChromaDB, listens on ~/.mempalace/mcp.sock)
        ├── Claude Code  ←→  mempalace-bridge.py  ←→  socket
        ├── Codex        ←→  mempalace-bridge.py  ←→  socket
        ├── Gemini CLI   ←→  mempalace-bridge.py  ←→  socket
        └── GG           ←→  mempalace-bridge.py  ←→  socket

• No zombie problem. If a session is SIGKILL'd, only the bridge dies. The daemon keeps running; the next session connects within milliseconds.
• No concurrent writer corruption. All tools/call requests are serialized through a threading.Lock inside the daemon. Protocol messages (initialize, tools/list, ping) remain lock-free.
• Auto-start on first use. The bridge detects a missing socket and starts the daemon automatically.
• LaunchAgent keeps it alive. If the daemon crashes, launchd restarts it within ThrottleInterval seconds (default: 5).

Changes

Four files added — no changes to the core mempalace package:

File	Description
examples/mempalace-daemon.py	Persistent Unix socket MCP server; the LaunchAgent target
examples/mempalace-bridge.py	~60-line stdio↔socket relay; this is the MCP command each session uses
examples/com.mempalace.daemon.plist	macOS LaunchAgent template (edit paths, then launchctl load)
docs/multi-session-daemon.md	Full explanation of the problem, architecture, install steps, and MCP config examples for Claude Code, Codex, Gemini CLI, and any stdio client

Test Plan

• Confirm mempalace-daemon.py starts and creates ~/.mempalace/mcp.sock
• Confirm mempalace-bridge.py connects and relays tools/list / mempalace_status correctly
• SIGKILL a bridge process; confirm daemon keeps running and next session connects cleanly
• Run two bridge sessions concurrently; confirm both mempalace_add_drawer calls succeed without HNSW divergence
• Unload LaunchAgent; confirm daemon stops; confirm bridge auto-restarts it on next connection attempt
• Verify no changes to existing mcp_server.py behavior for users not using the daemon

Tested on

• macOS 14 Sonoma / macOS 15 Sequoia
• Python 3.11, 3.12
• MemPalace 3.3.x (ChromaDB 0.6.x)
• Concurrent sessions: Claude Code + Codex + Gemini CLI + GG

🤖 Generated with Claude Code

[igorls]

Moved to v3.4 milestone — this is a feature-scale change (new long-running daemon process model + multi-session access pattern), not a maintenance fix. v3.3.5 is scoped to stability/bug fixes only. Companion issue #1229 moved with it.

Continuing review here is welcome; we'll target landing this in the v3.4 cycle once v3.3.5 ships.

[marcoaperez]

Real-world testimony — running these 4 files unmodified on macOS (M-series, bash 3.2) since 2026-05-05 (9 days) alongside mempalace==3.3.4 from pipx. Setup: com.mempalace.daemon.plist keeps the daemon up; 2 concurrent Claude Code sessions plus their MCP servers hit it through the bridge.

~/.mempalace/daemon.log in that window:

• 15 clean daemon restarts, zero hung shutdowns
• 13 flush-lag, not corruption events correctly classified by the daemon's heuristic (segment metadata intact → no quarantine) — the exact false-positive class that would have triggered unnecessary repair pre-fix: persistent daemon to prevent zombie processes and enable multi-session access (closes #1229) #1270
• 0 HNSW corruption, 0 SIGSEGV, 0 SQLite lock errors
• 15,544 drawers across 8 productive wings, growing daily via Stop/PreCompact hooks

The multi-session bridge is the load-bearing piece — without it, the two concurrent MCP server processes race exactly as #1229 describes.

Not asking to bump the milestone, just adding a field signal that the design works end-to-end on a non-trivial palace. Happy to share logs / plist / anything else if it helps review.

[marcoaperez]

Heads-up for anyone following this PR: I've been running this exact daemon design in production (it's a great fix for the concurrent-writer HNSW corruption). But the peer-writer guard merged in 3.5.0 (#1818 / #1823) holds mine_palace_lock for the whole process lifetime, which collides with this topology — it locks out hook/manual mempalace mine runs and its re-entrancy is per-thread while a daemon serves clients on many threads. Details + suggestions in #1888. Would be great to land #1270 and the guard in a way that lets them coexist.

[fatkobra]

Hi @Vasanth19 — I’ve reworked #1976 so it complements this PR rather than introducing a competing standalone daemon.

The revised implementation keeps #1270’s daemon + bridge architecture, but uses the existing package-level mempalace.daemon queue/server as the owner:

• mempalace-mcp becomes a stdio bridge into the existing daemon;
• the daemon exposes a token-authenticated /mcp endpoint;
• MCP tools/call requests and existing daemon queue jobs share one in-process writer gate;
• the raw stdio server remains available as mempalace-mcp-stdio;
• the low-level mine_palace_lock() fail-fast behavior remains unchanged;
• focused tests cover bridge forwarding, request serialization, identity mismatch handling, and daemon client behavior.

I removed the separate mempalace.mcp_daemon implementation, so this is now intended as package integration and automated-test coverage for the direction proposed here.

There is still rollout work before the full Tier 3 goal is complete: hook and CLI workflows need to consistently use daemon-backed execution rather than direct writer paths.

I’d appreciate your feedback on whether this division makes sense and whether any parts should be moved into, or coordinated more closely with, #1270.

[fatkobra]

Follow-up to my earlier comment: the package-integration and writer-routing work discussed there has now been implemented as a staged PR stack.

Current pieces:

• fix(mcp): route stdio clients through single-writer daemon #1976 — package-level MCP stdio bridge into the existing mempalace.daemon, including shared MCP/daemon writer serialization, authoritative backend/collection identity, and installed-command E2E coverage;
• feat(routing): add shared daemon write-routing policy #2027 — shared direct / prefer / require write-routing policy;
• feat(hooks): apply shared daemon write-routing policy #2030 — applies that policy to hook-triggered writes;
• feat(cli): apply shared daemon write-routing policy #2033 — applies it to routine CLI writes such as mine, sweep, sync, and the optional post-init mine.

The routing PRs intentionally preserve the current direct default. They provide an explicit require mode that never silently falls back to a direct ChromaDB writer.

The stack is currently:

#2027
  -> #2030
      -> #2033

All of these are intended to complement the daemon + bridge direction proposed in #1270, not to erase its design history or compete with it.

I’m leaving the branches unchanged while maintainers decide which architectural direction and merge sequence they prefer.