fix(mcp): route stdio clients through single-writer daemon

[fatkobra]

What does this PR do?

Contributes to #1963.

This reworks #1976 into a package-level complement to the daemon + bridge direction in #1270.

Instead of introducing a second MCP-specific daemon, this PR reuses the existing mempalace.daemon queue/server as the local owner and adds MCP stdio bridging into that daemon.

The previous default mempalace-mcp entry point started a full MCP server per client session. Each process could open its own local ChromaDB PersistentClient, which is one source of the concurrent-writer HNSW divergence cluster tracked in #1963.

This PR changes the MCP stdio topology:

• mempalace-mcp becomes a lightweight stdio bridge.
• The bridge auto-starts/reuses the existing per-palace MemPalace daemon.
• The bridge forwards JSON-RPC requests to a new daemon /mcp endpoint.
• The daemon imports mempalace.mcp_server.handle_request() lazily.
• MCP tools/call requests are serialized through the daemon writer gate.
• Existing daemon queue jobs and MCP tools/call requests share one in-process writer lock.
• Protocol requests such as initialize, ping, and notifications stay outside the writer lock.
• The old direct stdio server remains available as mempalace-mcp-stdio.
• MEMPALACE_MCP_DISABLE_DAEMON=1 provides an emergency rollback path.

Important scope note

This PR is one implementation slice within #1963.

#1963 is a tracking epic for the broader single-writer architecture. This PR makes the MCP stdio path use the existing daemon owner and ties MCP writes into the same daemon serialization point as daemon jobs.

The full Tier 3 rollout still needs installer/config follow-through so hook and CLI workflows consistently use daemon-backed paths rather than direct ChromaDB writers.

Why this complements #1270

#1270 proposed the durable daemon + bridge architecture: one long-lived owner and thin clients.

This PR adds production pieces around that direction:

• package-level bridge module and console entry points;
• reuse of the existing mempalace.daemon instead of a second daemon implementation;
• token-authenticated cross-platform loopback transport already used by the daemon;
• daemon-side /mcp endpoint;
• shared writer serialization between queued daemon jobs and MCP tools;
• identity checks for palace/backend/read-only/collection behavior;
• focused tests for bridge forwarding, daemon MCP handling, serialization, and mismatch refusal.

Why this does not touch mine_palace_lock()

The low-level palace lock remains fail-fast for direct external writers.

Earlier attempts to make the lock wait by default caused existing lock tests to fail or hang because direct Chroma/MCP writers are expected to raise MineAlreadyRunning when another process holds the palace.

This PR keeps that safety contract intact and moves queueing/serialization into the daemon path instead.

Compatibility

Existing MCP configs that call:

mempalace-mcp

continue to work, but now go through the daemon bridge.

Raw stdio fallback:

mempalace-mcp-stdio

Emergency rollback:

MEMPALACE_MCP_DISABLE_DAEMON=1 mempalace-mcp

How to test

Focused tests:

    python3 -m pytest tests/test_mcp_daemon_bridge.py -q

Related safety tests:

    python3 -m pytest tests/test_daemon.py::test_daemon_http_lifecycle_executes_job tests/test_daemon.py::test_submit_job_uses_client_and_waits tests/test_daemon.py::test_run_mcp_tool_dispatches_write_tool tests/test_chroma_collection_lock.py::test_writer_blocks_during_mine tests/test_chroma_collection_lock.py::test_concurrent_writers_serialize tests/test_convo_miner.py::test_mine_convos_refuses_concurrent_run_against_same_palace tests/test_palace_locks.py::test_same_palace_serializes_across_processes tests/test_palace_locks.py::test_reentrant_same_thread_passes_through -q

Full CI shape:

    python3 -m ruff format --check .
    python3 -m ruff check .
    python3 -m pytest tests/ -v --ignore=tests/benchmarks --cov=mempalace --cov-report=term-missing --cov-fail-under=80 --durations=10

References

Contributes to the Tier 3 daemon/bridge direction in #1963.
Complements the daemon + bridge architecture from #1270.
Related to #1229, #1888, and #1966.

Checklist

• Tests pass (python -m pytest tests/ -v)
• No hardcoded paths
• Linter passes (ruff check .)

[fatkobra]

/gemini review

[mvalentsev]

@fatkobra on Windows this breaks the default entry point: CPython never got socket.AF_UNIX there, so the bridge can't connect, the auto-started daemon dies at startup in serve_unix_socket, and mempalace-mcp polls for 15s and exits 1. CI stays green because the socket paths are all pragma: no cover and there's no end-to-end test of the default command. Windows users would need MEMPALACE_MCP_DISABLE_DAEMON=1 just to get a working server, so the rollback path becomes the default path for a whole platform.

The daemon lifetime also changes the writer story for everything that isn't an MCP client: there's no idle shutdown, and mcp_server takes its server-lifetime writer lease, so once a daemon exists, hook-spawned mempalace mine gets MineAlreadyRunning permanently instead of only while a session is alive. CLI and hook mines don't route through the daemon, so MCP clients consolidate while every other writer is starved harder than before. The socket is also keyed by palace path only, so a second client's --read-only/--backend flags are silently ignored when a daemon is already up with different settings.

And honestly, the framing here isn't ok: Closes #1963 would close a tracking epic that exists to stay open across a multi-PR pipeline and give the maintainers one decision point, and #1270 is an open PR implementing this same daemon direction. Re-implementing an open PR and closing the epic in one shot isn't the way to land this; that needs coordination with the #1270 author and a maintainer call on the direction first.

To be clear, I'm not hunting your PRs. I'd dug deep into #1924 and #1920 myself earlier, so I had the context loaded when yours showed up, and after those two I read this one too and couldn't walk past the Windows thing. Nothing personal.

[fatkobra]

Thanks for the detailed review @mvalentsev . I agree these are serious design concerns, not just CI/test issues.

The PR currently makes mempalace-mcp default to a Unix-socket bridge, but I did not add an end-to-end default-command test on Windows, and the transport story is not cross-platform enough to make this the default safely.

I also agree the daemon lifetime/writer-lease interaction needs a maintainer decision. This PR consolidates MCP stdio clients, but CLI mines and hook-spawned mines do not route through the daemon, so it does not make the single-writer story true end-to-end. In some setups it may actually starve those external writers harder.

I also agree that Closes #1963 is too broad. #1963 is a tracking epic and decision point, not something this PR should close unilaterally, especially while #1270 already exists for the daemon + bridge direction.

I’ll step back from this implementation and, if maintainers want help, I’m happy to coordinate on #1270 or on a smaller opt-in/cross-platform slice with explicit tests.

[fatkobra]

Pushed a rework.

The PR no longer introduces a separate MCP-specific daemon. It now reuses the existing mempalace.daemon queue/server and adds MCP stdio bridging into that daemon.

Main changes:

• removed the standalone mempalace.mcp_daemon implementation;
• made mempalace-mcp bridge into the existing daemon;
• added a daemon /mcp endpoint for JSON-RPC requests;
• made daemon queued jobs and MCP tools/call requests share one in-process writer lock;
• preserved the existing fail-fast mine_palace_lock() contract for direct external writers;
• kept mempalace-mcp-stdio as the raw fallback;
• added focused tests in tests/test_mcp_daemon_bridge.py;
• updated docs in docs/mcp-daemon-bridge.md;
• changed the PR framing from a standalone “MCP daemon” to a package-level complement to fix: persistent daemon to prevent zombie processes and enable multi-session access (closes #1229) #1270.

This still contributes to #1963 rather than closing it. Full closure still needs hook/CLI rollout so all writer paths consistently use the daemon-backed route.

[fatkobra]

@mvalentsev thanks again for the review. I reworked the PR around your concerns.

The new version no longer starts a separate MCP-specific Unix-socket daemon. It reuses the existing mempalace.daemon loopback/token-auth daemon and adds a /mcp endpoint there.

That changes the shape materially:

• Windows no longer depends on Unix sockets for the default MCP bridge path.
• The daemon is the existing package daemon, not a second daemon process model.
• MCP tools/call and daemon queued jobs share one in-process writer lock.
• The low-level mine_palace_lock() fail-fast contract is preserved.
• The PR no longer claims to close Epic: concurrent-writer HNSW corruption — unified tracking for the recurring-divergence cluster (357, 1092, 1581, 1799, 1845, 1888, chroma#6975, …) #1963.
• The PR is now framed as a production/package complement to fix: persistent daemon to prevent zombie processes and enable multi-session access (closes #1229) #1270.

It still does not claim to complete the full Tier 3 rollout by itself; hook/CLI configuration still needs to consistently route through daemon-backed paths for the whole #1963 cluster to close.

[Timofa]

I tested the current head (125aaec) against a real loopback daemon/client and found two blockers in the reworked bridge path.

1. POST /mcp currently returns 404. DaemonClient.mcp_request() sends a POST, but the /mcp route is inside _handle_get(); do_POST() only dispatches /jobs. The existing bridge test mocks the HTTP client, so it does not exercise this route mismatch.

2. The advertised default backend inheritance is rejected. A bare mempalace-mcp sends backend="". If an existing hook/CLI daemon is running with backend="chroma", _check_mcp_identity() compares those literally and returns daemon='chroma' bridge=''. An explicit non-empty mismatch should still be rejected; only omission should inherit the daemon's effective backend.

I prepared and independently reviewed a narrow patch based directly on this head:

• move the /mcp block from _handle_get() to do_POST();
• canonicalize an omitted incoming backend to expected["backend"] before pinning/comparing the MCP identity;
• add a real loopback HTTP server/client regression test;
• add coverage that blank inherits chroma while explicit qdrant is still refused.

The core normalization is:

received_backend = str(identity.get("backend") or "").strip().lower()
if not received_backend:
    received_backend = expected["backend"]

Validation on the PR head:

• before the fix: exactly two new regressions reproduced (HTTP 404 and daemon='chroma' bridge='');
• after the fix: 48 passed across test_daemon.py and test_mcp_daemon_bridge.py;
• ruff check ., ruff format --check ., pre-commit, and git diff --check all pass;
• an independent subprocess E2E bridge -> daemon -> ping passes with a blank bridge backend and daemon chroma;
• diff scope: 3 files, +87/-12.

I have the clean tested patch ready and can provide it as a cherry-pick/stacked branch if useful. One separate non-blocking follow-up is env-only backend selection (for example MEMPALACE_BACKEND=qdrant with no CLI flag), which deserves its own packaged E2E.

[fatkobra]

Thanks, @Timofa — this is excellent validation, and I appreciate you taking the time to reproduce both failures against the real daemon/client path, prepare the narrow fix, and independently verify it.

I confirmed both blockers in the current head:

• the bridge sends POST /mcp, while the route currently lives under the GET dispatcher;
• a blank bridge backend is treated as conflicting with the existing daemon’s effective backend instead of inheriting it.

Please share the commit hash or stacked branch. I’m happy to cherry-pick your tested patch, including the real loopback HTTP regression test, so your authorship is preserved in the commit history.

I also agree that environment-only backend selection, such as MEMPALACE_BACKEND=qdrant without a CLI flag, should be handled separately.

Thank you again for finding, reproducing, and preparing fixes for these two blockers.

[Timofa]

Thanks for confirming both issues.

I've pushed the patch as a single commit stacked directly on the current PR head (125aaeced3e7fbe046986f1a4f4333cd0da87a60):

• commit: 43a1f9d0
• branch: Timofa:codex/fix-1976-mcp-bridge

To cherry-pick:

git fetch https://github.com/Timofa/mempalace.git codex/fix-1976-mcp-bridge
git cherry-pick 43a1f9d0ba01fddea9f8200c7c09dd415d7185d0

The commit is limited to the POST /mcp routing fix, omitted-backend inheritance while preserving explicit mismatch rejection, and the corresponding real-loopback and identity regression tests.

Validation on this commit:

• 48 passed across tests/test_daemon.py and tests/test_mcp_daemon_bridge.py;
• ruff check ., ruff format --check ., pre-commit, and git diff --check pass;
• the subprocess bridge -> daemon -> ping E2E passes with a blank bridge backend inheriting the daemon's chroma backend.

The environment-only MEMPALACE_BACKEND case is intentionally not included in this patch.

[fatkobra]

Cherry-picked @Timofa’s tested fix for the /mcp POST routing and omitted-backend inheritance regressions.

Credit to @Timofa for reproducing both failures against a real loopback daemon/client path, preparing the focused patch, and adding the integration and identity regression coverage. The cherry-picked commit preserves @Timofa as the author.

[fatkobra]

Pushed a scope cleanup removing the obsolete standalone-daemon documentation.

I initially attempted to remove the earlier formatting-only change to tests/test_mcp_server.py, but the repository’s ruff-format pre-commit hook correctly reapplied it. I therefore retained that required formatting change so both pre-commit and repo-wide formatter checks remain green.

[fatkobra]

Pushed the effective-identity and installed-command E2E follow-up.

Effective identity changes:

• the daemon now independently resolves its effective backend using the normal MemPalace backend resolver;
• the daemon resolves and pins its drawer collection at startup;
• /health and endpoint.json report the effective backend and collection;
• the lazy MCP import is pinned to those same resolved values;
• omitted backend/collection values inherit the running daemon’s identity;
• explicit backend or collection mismatches are still refused;
• MEMPALACE_COLLECTION_NAME is now the supported process-level collection override.

E2E coverage:

• added a subprocess test that invokes the actual installed mempalace-mcp command;
• starts/reuses a real local daemon;
• selects qdrant through MEMPALACE_BACKEND with no --backend flag;
• sends a real JSON-RPC ping;
• verifies the daemon endpoint, health response, and pinned MCP identity;
• shuts the daemon down after the test.

Because the normal CI jobs install the editable package and run the full test suite on Linux, macOS, and Windows, this installed-command E2E is exercised across all three platforms.

[henderlabs]

I tested the current pull request head against the two-client topology discussed in #1963.

Tested commit:

1b024772e980e87311dc0b6fc249f6221a4abd59

Topology tested

• Fresh clone of MemPalace/mempalace
• Pull request fix(mcp): route stdio clients through single-writer daemon #1976 checked out directly
• Isolated virtual environment
• Scratch palace under /tmp
• MEMPALACE_PALACE_PATH pointed at the scratch palace
• MEMPALACE_DAEMON_STATE_ROOT pointed at scratch daemon state
• Production palace was not opened
• Two separate mempalace-mcp standard input/output bridge client processes
• Simultaneous mutating calls released through a barrier

Result

Check	Result
Daemon starts from this pull request	Pass
Two separate bridge clients initialize against one scratch palace	Pass
Simultaneous mutating calls from both clients	Pass
Both writes return distinct drawer IDs	Pass
No held by PID or peer-writer lock error	Pass
Final drawer count before restart	Pass: 2
Daemon restart with a cold ChromaDB client	Pass
Drawer count after restart	Pass: 2
Scratch cleanup	Pass

Neither client emitted standard error, and both exited successfully.

One useful migration detail observed during the test: the entry point mapping now makes mempalace-mcp call mempalace.mcp_bridge:main, while the old direct server path is available as mempalace-mcp-stdio. That means existing Claude Code and Codex Model Context Protocol configurations that call mempalace-mcp should become bridge clients on upgrade without configuration changes.

One scope caveat: this validates the bridge, daemon writer gate, concurrent two-client coordination path, and basic restart durability for a small write set. It does not exercise the ChromaDB sync_threshold data-loss mechanism described in #1963, because the test wrote 2 drawers rather than crossing the 1000-record threshold. A larger threshold-crossing test would be needed before claiming that this pull request alone proves out the full HNSW purge/corruption scenario.

Within that scope, this pull request looks aligned with the desired shared-agent topology: one daemon owner process, one ChromaDB handle, standard input/output clients bridged into that owner, and serialized writes without direct peer-writer conflicts.

[fatkobra]

Thank you for running this independent two-client integration test and documenting the topology and results so clearly.

This validates the core behavior #1976 is intended to provide:

• two separate mempalace-mcp bridge processes sharing one daemon owner;
• simultaneous mutating calls serialized without peer-writer conflicts;
• both writes retained with distinct drawer IDs;
• successful cold-daemon restart with the drawer count preserved;
• existing MCP configurations migrating to the bridge through the unchanged mempalace-mcp command.

I agree with the scope caveat. A two-drawer test validates the gateway topology and basic restart durability, but it does not reproduce the sync_threshold-crossing failure described in #1963. This PR is intentionally framed as one implementation slice within that epic rather than proof that it alone closes the entire corruption cluster.

A larger threshold-crossing scenario would be useful as a separate stress/integration validation if maintainers want it, but I will not broaden the claims of this PR based on the current test.

Thanks @henderlabs again for the careful verification.

[igorls]

Thanks for this contribution, and apologies for the slow turnaround.

develop has moved a fair way since this was opened and the branch no longer merges cleanly. If you're still interested in landing it, could you rebase onto current develop? Once it merges cleanly and CI is green I'll get it reviewed for the 3.8.0 cycle.

If you'd rather not pick it back up, no problem at all — just say so and I'll close it out, and thanks either way for taking the time to send it.

[fatkobra]

@igorls I will work on it and rebase it and address it and make it ready.

[fatkobra]

@igorls rebased this branch onto current develop and resolved the daemon
and test conflicts.

The rebase preserves current upstream behavior, including:

• process-lifetime local-backend writer ownership;
• retention of the writer lease while a timed-out worker remains active;
• lock-deferral handling for queued jobs;
• current daemon lifecycle, request-safety, and environment-restoration
  behavior;
• current package metadata, dependency pins, and source-adapter entry
  points.

It also retains the PR-specific MCP bridge behavior:

• mempalace-mcp routes through the existing daemon;
• POST /mcp is handled by the daemon;
• daemon jobs and MCP mutations share the daemon writer gate;
• omitted backend and collection values inherit the daemon's effective
  identity;
• explicit identity mismatches remain rejected;
• the direct stdio fallback remains available as mempalace-mcp-stdio.

@Timofa Timofa's /mcp routing and backend-inheritance fix was replayed with Timofa
preserved as the commit author.

Git skipped the old formatting-only 007284f commit because those changes
were already present in current develop.

I also extracted writer-lease acquisition into a small helper so the merged
run_server() remains within the repository's Ruff complexity limit
without suppressing C901.

The rebased history was pushed with --force-with-lease, and GitHub CI is
green.

[stefano-scarpone-bluvacanze]

Disclosure: produced by Claude Code on my machine — it ran the measurements and drafted this, I reviewed and I'm posting it.

Strong +1 on this direction, from a Windows install that is currently in exactly the state the PR describes. Plus one measurement that I think is worth considering for the writer gate.

Why this PR matters for a normal desktop setup

mempalace 3.7.1 / chromadb 1.5.9, chroma backend, 44,657 drawers, Windows 11. One mempalace.daemon supervised at logon by a Task Scheduler task — which is what docs/hook-write-routing.md recommends for require — plus N Claude Code sessions, each spawning its own mempalace-mcp.

Result: the daemon takes mine_palace_lock in run_server() and holds it for its whole lifetime, so every session gets

MCP error -32001: Peer MCP writer active; this server is read-only for mutating tools

for the entire life of that session. Following the recommended supervised configuration is precisely what makes every agent read-only, and since mcp_server.py has no DaemonClient, a refused server cannot enqueue onto the owner — it can only wait for it to exit. The workaround users land on is killing the daemon, which reintroduces the interrupted-write risk that the single-writer enforcement exists to prevent.

That is the loop this PR breaks: the bridge makes the supervised daemon the correct configuration rather than the thing that locks everyone out, and it keeps the "no external service by default" property from AGENTS.md.

Measurement: serializing reads behind writes has a visible cost

I couldn't test the stdio bridge against my production palace, so I measured the closest available shape of "one owner, N clients": mempalace serve (3.7.1, stock) on a throwaway palace, driven by two independent JSON-RPC clients.

Concurrent mutations — works, as the earlier two-client test in this thread also found:

client A -> mempalace_add_drawer -> success, drawer_test_technical_3e35261c...
client B -> mempalace_add_drawer -> success, drawer_test_technical_a8ab6ba6...

Long write vs. concurrent read:

operation	elapsed
mempalace_mine mode=convos on a 3.0 MB transcript	10.0 s
mempalace_status issued by a second client 3 s into that mine	7.0 s blocked

In the HTTP transport this is _HTTP_REQUEST_LOCK wrapping dispatch, with _HTTP_LOCK_FREE_TOOLS exempting only the logstream/artifact tools. The PR description says protocol requests (initialize, ping, notifications) stay outside the writer lock — my question is whether read tools (mempalace_search, mempalace_status, mempalace_get_drawer, …) go through the same gate in the daemon /mcp endpoint.

If they do, one agent's transcript ingest stalls every other agent's recall for the duration of the mine. On my machine transcripts run up to 5.3 MB, so worst case is tens of seconds of blocked reads across all sessions — and recall latency is the thing users notice immediately. Under the pre-bridge topology those reads came from separate read-only processes and were never blocked by a mine, so this would be a visible regression for a setup that otherwise gets strictly better.

Two shapes that would avoid it, both with precedent in the codebase:

1. an explicit read-tool exemption list, mirroring _HTTP_LOCK_FREE_TOOLS — there seems to be precedent for the split already, see the note on classify_tool at the end of this comment;
2. a readers-writer lock at the gate: concurrent reads, exclusive writes, which matches what the underlying constraint actually requires (one writer, not one caller).

Not a blocker for landing it — just flagging it before the topology becomes the default, since after that every session's recall latency depends on the answer.

The read/write split you need already exists in service.py

While working around the read-only state on my own palace I found service.execute_job already dispatches a generic mcp_tool job kind, which runs any write-classified MCP tool inside the daemon that owns the lease. (Quotes below are from the released 3.7.1 package, at service.py:409 there — develop has moved since, so the line numbers won't match; the function names should.)

if kind == "mcp_tool":
    return run_mcp_tool(payload)

classification = classify_tool(name) if name else "unknown"
if classification != "write":
    return {"success": False,
            "error": f"daemon mcp_tool only accepts write tools; {name!r} is {classification}",
            "exit_code": 2}
from .mcp_server import TOOLS
result = TOOLS[name]["handler"](**arguments)

with the docstring noting "No internal caller currently uses mcp_tool; this allowlist bounds the blast radius of the generic escape hatch."

Two things follow that seem useful for this PR:

1. It corroborates the premise on a real palace. I used it today to file a drawer (5 chunks), five kg_add facts and a tunnel into a 44,657-drawer palace while every stdio MCP session was read-only, with no lease contention and no daemon restart:

submit_job("mcp_tool", {"name": "mempalace_add_drawer", "arguments": {...}},
           palace_path=..., wait=True)
# -> state: succeeded, drawer_id: drawer_wing_mempalace_decisions_232ab45a..., chunks: 5

So the daemon can already execute MCP tool handlers in-process under the lease. What's missing is the transport — which is exactly what this PR adds.

2. It may bear on my question above about the writer gate — flagged as an inference, not a review comment: I have not read this PR's diff closely enough to know how the daemon /mcp endpoint classifies incoming tools/call requests, so please discount this accordingly. What I can see in the released package is that classify_tool / service.WRITE_TOOLS already split read from write at the granularity such a gate would need, so exempting read tools from the writer lock may not require new classification machinery. The reasoning in that docstring seems to point the same way: reads are deliberately kept out of the durable queue path (verbatim palace content would land in the queue DB and in /jobs results), which would make routing them straight through handle_request() outside the writer lock consistent with the design already there. If the bridge already does this, ignore the whole point.

As a side note, mcp_tool is also a usable stopgap for anyone hitting -32001 today — worth a line in the docs even if the bridge lands, since it needs no config change and no process restart.

Worth checking before this merges: KG writes executed inside the daemon may target a different knowledge_graph.sqlite3

Using that stopgap, my kg_add calls returned state: succeeded, success: true and a valid triple_id — and the facts were not in the graph my MCP session reads. No error, no warning. They had gone into a second KG file created on the spot.

In 3.7.1:

# mcp_server.py
_palace_flag_given: bool = bool(_args.palace)

def _resolve_kg_path() -> str:
    if _palace_flag_given:
        return os.path.join(_config.palace_path, "knowledge_graph.sqlite3")
    return DEFAULT_KG_PATH          # ~/.mempalace/knowledge_graph.sqlite3

_palace_flag_given is decided at import time from the argv of whatever process imported the module. The stdio server the plugin launches has no --palace, so it uses DEFAULT_KG_PATH. The daemon is spawned as python -m mempalace.daemon serve --palace <path>, so when service.run_mcp_tool does from .mcp_server import TOOLS, the module parses the daemon's argv, _palace_flag_given is True, and the KG path becomes <palace>/knowledge_graph.sqlite3 — a new, empty database inside the palace directory.

On my host that produced a clean split:

~/.mempalace/knowledge_graph.sqlite3          365 triples, full history (MCP, hooks, CLI)
~/.mempalace/palace/knowledge_graph.sqlite3     6 triples, created today, only what the daemon wrote

Drawers are unaffected — add_drawer through the same path lands in the right collection, confirmed with get_drawer and by repair-status counts.

Two notes on scope:

• This is not caused by this PR. It reproduces for anyone whose writer process was started with --palace while their reader wasn't; mempalace-mcp --palace <p> and mempalace-mcp disagree about the KG for the same palace. It may deserve its own issue.
• But this PR makes it systemic, which is why I'm raising it here rather than separately: once every tools/call is executed inside the daemon, mempalace_kg_add / kg_supersede / kg_invalidate from every agent would resolve the KG the daemon's way, while hooks and CLI keep writing the other file — silently, since the tools report success. I hit this through the mcp_tool job kind, not through this PR's /mcp endpoint, so the bridge may already resolve it differently. Worth a check, and if it does apply, a KG round-trip in the integration test would catch it.

Happy to re-run the same measurement against the bridge on a copy of my palace (44k drawers, Windows) if that would be useful evidence for the 3.8.0 review.

[fatkobra]

@stefano-scarpone-bluvacanze thank you for the detailed testing and for separating direct observations from inferences.

I checked both points against the current PR head.

Read serialization

Your read-latency concern applies to the current implementation.

DaemonRuntime.handle_mcp_request() currently treats every tools/call request as requiring the shared writer gate, so read tools such as mempalace_search, mempalace_status, and mempalace_get_drawer can wait behind a long-running mine or write.

The existing service.classify_tool() split could support a narrower gate, but current develop’s HTTP transport also deliberately serializes almost every Chroma/KG tool request. I do not want to exempt reads without a focused concurrent read/write test proving that the shared client, reconnect state, and caches remain safe.

I will treat this as a performance follow-up unless maintainers want read concurrency included in this PR’s scope.

Knowledge-graph path

The KG concern also applies.

mcp_server._resolve_kg_path() currently chooses between the legacy global KG and <palace>/knowledge_graph.sqlite3 based on whether --palace appeared in the importing process’s argv. A daemon or hub process started with --palace can therefore write to a different KG file from an ordinary stdio MCP process started without that flag.

That behavior exists on current develop; it was not introduced by #1976. However, making daemon-side MCP execution the default would make the inconsistency systematic, so I agree it needs a separate fix and a KG round-trip integration test before the gateway becomes the default.

I do not think the internal mcp_tool job should be documented as a general workaround while this path mismatch exists.

Current architecture overlap

There is one additional development since this branch was last rebased: current develop now maps mempalace-mcp to the new mempalace.mcp_proxy, which forwards stdio sessions to a live mempalace serve hub. #1976 still maps the same command to mempalace.mcp_bridge, which auto-starts/reuses mempalace.daemon.

The branch now conflicts because these are overlapping gateway implementations, not just because of incidental code drift.

@igorls before I rebase again, could you confirm the intended 3.8 direction?

• should fix(mcp): route stdio clients through single-writer daemon #1976 be considered superseded by the new mcp_proxy / mempalace serve path;
• should it be narrowed to pieces that are still unique;
• or should its automatic daemon-start behavior be integrated into the new proxy/hub architecture?

I do not want to resolve the entry-point conflict by silently replacing the newer upstream design.

[igorls]

Thanks for the careful review and for surfacing the lifecycle and identity concerns.

We’ve made a directional decision since this PR was opened: the long-term architecture is a central, long-lived mempalace serve hub for each palace on a machine. MCP clients should be thin callers that connect to that hub, rather than each project or agent opening an independent local server and database handle.

This is especially important now that the replicated palace and logstream are becoming core infrastructure. We need one visible, well-configured, stable service owning the palace locally, with clear health and identity information.

Because of that, we do not want to rebase or merge the daemon bridge and lifecycle implementation in this PR. That path is superseded by the hub/proxy architecture.

However, the identity-validation concern here is still important. A caller must not silently connect to a hub that does not match its effective palace and configuration. We would welcome a focused follow-up PR against current develop that adds explicit hub identity validation and clear mismatch errors.

The next lifecycle step on our side is safe “connect or start” behavior: if the correct hub is absent, one caller may start it under a cross-process lock, while other callers wait for readiness and connect. The running hub should remain visible and inspectable to the user.

We will handle the database-path consistency and existing split-store migration as a separate correctness fix before making this topology the default.

Thank you again—your review helped identify the safeguards we need to preserve as we move to the unified hub model.

[fatkobra]

Thanks @igorls — understood.

I will not rebase or further update #1976. I accept the architectural decision that the long-term owner is one visible, long-lived mempalace serve hub per palace, with stdio MCP sessions acting as thin proxy clients. I’ll close this PR rather than resolve its new conflicts against the hub/proxy implementation.

I’m happy to extract the still-relevant identity work into a focused PR against current develop, limited to:

• the hub publishing its effective canonical palace and configuration identity;
• the proxy validating that identity before forwarding;
• clear palace/backend/collection/read-only mismatch errors;
• focused matching and mismatch regression coverage.

I will leave the connect-or-start lifecycle work to the maintainer-side implementation you described.