fix(cli): add repair rebuild-index alias - #1670
Merged
Merged
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a new rebuild-index action to the repair command in the CLI, which serves as an alias for rebuilding the palace from SQLite with the current embedding model. It updates the Chroma backend error message to suggest this exact command with properly quoted paths using shlex.quote, and adds corresponding unit tests to verify the CLI dispatching and backend error message formatting. I have no feedback to provide as there are no review comments.
Contributor
|
This command is already documented as if it exists, but it was never added to the CLI. This PR is what introduces it. The references describe
But the subcommand doesn't exist on develop, so following any of them lands on exit 2: So a user who switches (Routing looks right too: mapping to |
jsiu93
force-pushed
the
fix/repair-rebuild-index-alias
branch
from
June 7, 2026 03:05
6120d73 to
24ef6c0
Compare
jphein
added a commit
to techempower-org/mempalace
that referenced
this pull request
Jun 26, 2026
* feat(miner): add C# and .NET file extensions to READABLE_EXTENSIONS
Adds .cs, .csproj, .sln, .razor, and .cshtml so C#/.NET projects
are indexed by the project miner. .razor/.cshtml are analogous to
the already-supported .jsx/.tsx.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(miner): add support for Swift and Kotlin file extensions
- Updated READABLE_EXTENSIONS in miner.py to include ".swift", ".kt", and ".kts".
- Added tests in test_miner.py to ensure scanning includes Swift and Kotlin files.
* feat: add Pi agent JSONL session normalizer
Add _try_pi_jsonl parser for Pi agent session files stored at
~/.config/pi/agent/sessions/{encoded-cwd}/{timestamp}_{uuid}.jsonl.
Uses type "message" entries with role "user"/"assistant". Skips
toolResult messages, model_change, thinking_level_change, and other
operational events. Requires session header (type "session" with
"version" key) to avoid false positives.
Format documented at github.com/badlogic/pi-mono session.md and
verified via Context7. Sample data provided by tunnckoCore in #59.
Refs: #59
* feat: add Gemini CLI / AI Studio JSON session import support
Adds _try_gemini_json parser to normalize.py for three layouts:
1. Gemini API contents format (~/.gemini/sessions/*.json):
{"contents": [{"role": "user", "parts": [{"text": "..."}]}, ...]}
2. Messages-wrapper variant:
{"messages": [{"role": "user", ...}, {"role": "model", ...}]}
3. Flat top-level list with role="model".
This complements the existing _try_gemini_jsonl parser (which handles
~/.gemini/tmp/<hash>/chats/session-*.jsonl with session_metadata
sentinel) — JSONL covers Gemini CLI runtime sessions, JSON covers
exported / Studio-saved transcripts.
## Review feedback addressed (PR #204)
bgauryy review:
- #1 Parser-precedence bug: _try_gemini_json runs *before*
_try_claude_ai_json so the {"messages":[..., role=model, ...]}
layout is no longer silently claimed by the Claude parser. The
Gemini parser's has_model_role guard prevents false-positives
against Claude / ChatGPT data.
- #2 Layout 2a coverage: TestGeminiJson.test_messages_wrapper_format
+ test_messages_wrapper_does_not_get_claimed_by_claude pin the
fix in place.
- #3 Test conflicts with current main: rebased onto develop;
tests restructured into TestGeminiJson class.
- #4 tempfile/os.unlink → pytest tmp_path everywhere.
- #5 elif not text → else (the elif branch was dead).
- #6 Module docstring updated to mention Google AI Studio.
Tests: 9 new cases in TestGeminiJson covering all three layouts,
multi-part text joining, non-text part skipping, has_model_role
disambiguation, dispatch-chain regression for review #1.
* feat(normalize): add Continue.dev session parser
Add _try_continue_json() normalizer for Continue.dev AI assistant
sessions (~/.continue/sessions/*.json). Parses history array with
role/content pairs, handles tool calls, system messages, and metadata.
Closes #59 (partial — adds Continue.dev format support)
Includes comprehensive test coverage for valid sessions, edge cases,
malformed input, and unicode content.
* fix: preserve collection name on MCP search retry
* feat: add Cursor IDE support (hooks, plugin, skill, docs, tests)
Adds first-class Cursor IDE integration alongside the existing Claude
Code and Codex hook flows, so Cursor users get the same automatic
diary saves, pre-compaction transcript capture, and session-start
memory recall — without changing any default behaviour for existing
users.
What's included
---------------
Cursor hook scripts (hooks/cursor/):
- mempal_save_hook_cursor.sh — Stop event, counter +
loop_count guard, pending-save marker consumption, background
mempalace mine, followup_message emission.
- mempal_precompact_hook_cursor.sh — synchronous mine before
compaction, drops a pending_save marker, returns user_message.
- mempal_wake_hook_cursor.sh — sessionStart event,
wing-scoped recall guidance via additional_context.
- lib/common.sh — shared parsing + state helpers
(bash 3.2 safe, no heredoc-in-subshell traps).
- install.sh — idempotent installer with
--scope, --variant, --dry-run, --uninstall. Recognises existing
entries by basename so re-installs across paths work.
- STDIN_SHAPE.md, README.md — payload schemas + quick
reference.
Cursor plugin (.cursor-plugin/ + repo-root components):
- plugin.json, marketplace.json, README.md.
- skills/mempalace/SKILL.md — model-invocable skill mirroring the
Claude plugin's skill surface.
- commands/mempalace-{help,init,mine,search,status}.md — slash
commands for marketplace-published installs (filename = slug).
- mcp.json — auto-registers the mempalace MCP
server, wrapped under the documented mcpServers key.
Examples + docs:
- examples/cursor/hooks.json, hooks.minimal.json + README.
- website/guide/cursor-hooks.md + sidebar entry.
- README.md and CHANGELOG.md updates.
Tests (129 new, all green):
- tests/test_cursor_hooks_shell.py — 75 behavioural tests for
the three hook scripts: kill switches, input parsing, counter
logic, loop prevention, pending markers, wing inference, logging.
- tests/test_cursor_hooks_install.py — 19 contract tests for the
installer: dry-run, idempotent merge, basename-matched uninstall,
refusal to overwrite malformed JSON.
- tests/test_cursor_plugin_manifest.py — 35 contract tests for the
plugin: manifest validity, version sync with mempalace.version,
mcp.json shape, skill/command frontmatter, default-discovery
layout invariants.
Design notes
------------
- Local-first and zero-API by default; hooks never call external
services. Same privacy model as the existing Claude Code hooks.
- Fail-open: hook scripts deliberately do not use set -e so a broken
hook can never block the user's conversation.
- Cursor preCompact cannot block + return a followup, so we
synchronously mine the transcript and drop a pending_save marker
that the next stop hook consumes — guarantees verbatim capture
before context window compression.
- Cursor's default plugin discovery requires real commands/, skills/,
and mcp.json at the plugin root (verified against the cached
cloudflare plugin); .cursor-plugin/{commands,skills} are convenience
symlinks back to those canonical locations.
- bash 3.2 compatibility throughout: avoids heredoc-in-command-
substitution parser bugs; uses python -c for JSON parsing;
basename-matched entry recognition in install.sh.
- All changes are additive. No existing files are removed, no
existing hooks change behaviour, and no new runtime dependencies
are introduced.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(cursor): address gemini-code-assist review on PR #1632
Five fixes from the Gemini Code Assist review on
https://github.com/MemPalace/mempalace/pull/1632 — three real bugs,
two cleanups, all consistent with the bash-3.2-compatibility
contract documented in the original commit.
Bug fixes (high)
----------------
1. hooks/cursor/lib/common.sh — config.json kill-switch check used
a `python3 - <<'PYEOF' ... PYEOF` heredoc inside a `$(...)`
command substitution. The heredoc body contains parens which
trips the macOS bash 3.2.57 parser bug. Replaced with a
`python -c '...'` call passing the config path as argv[1]. Matches
the pattern already used in mempal_parse_stdin in the same file.
2. hooks/cursor/install.sh — a relative `--install-dir` was written
verbatim into hooks.json. Cursor invokes hook commands from its
own working directory (typically the project root), so a relative
command path would silently fail to launch the hook. Now resolved
to an absolute path against `$PWD` before being baked in.
3. hooks/cursor/mempal_save_hook_cursor.sh — `MEMPAL_SAVE_INTERVAL=0`
would crash bash on `$((NEXT % 0))` (division by zero). Extended
the existing sanitiser case to coerce 0 to the default interval
alongside empty / non-numeric values.
Cleanups (medium)
-----------------
4. hooks/cursor/install.sh — the EMPTY_CHECK_PY temp file is now
inlined as `python -c '...'`. Removes a small leak window
(tmpfile would linger if the script were interrupted between
mktemp and rm -f) and shortens the script.
5. hooks/cursor/install.sh — `mktemp -t prefix` has subtly different
semantics on BSD (macOS) vs GNU mktemp. Switched to the
portable absolute-template form `mktemp "${TMPDIR:-/tmp}/...XXXXXX"`
which behaves identically on both.
Regression tests
----------------
- tests/test_cursor_hooks_shell.py
test_save_interval_zero_is_coerced_to_default — guards fix #3.
- tests/test_cursor_hooks_install.py — new TestInstallDirAbsolutePath
class:
test_relative_install_dir_is_absolutized_in_hooks_json — guards
fix #2 against regression.
test_absolute_install_dir_is_preserved_verbatim — guards that
the relative-to-absolute resolution does not mangle paths
that were already absolute.
Verification
------------
- bash -n on all three edited scripts: clean.
- uv run pytest tests/test_cursor_hooks_*.py tests/test_cursor_plugin_manifest.py: 132 passed (was 129; +3 regression tests).
- uv run pytest tests/ --ignore=tests/benchmarks: 2399 passed,
3 skipped (pre-existing).
- uv run ruff check . / ruff format --check .: clean.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(cursor): address igorls review on PR #1632
Resolves the maintainer review on the Cursor IDE support PR. Cursor-only
scope; cross-IDE items (wing-naming convention, shared-file merge order)
are coordinated on the separate Antigravity branch.
followup_message default (the one "decide before merge" item):
- Keep the stop-hook followup ON by default. Cursor's transcript format
is undocumented and mempalace/normalize.py has no Cursor parser, so the
background `mempalace mine --mode convos` is best-effort only and does
not yet yield clean verbatim drawers. The followup is therefore the
load-bearing verbatim-capture path; defaulting it off would leave a
default Cursor install capturing nothing.
- Add an opt-out (MEMPAL_CURSOR_SILENT=1, or MEMPAL_VERBOSE=false) for
users who want the Claude-style "zero tokens in chat" behaviour. The
hook still mines and keeps its counters/markers when silenced.
- Correct the misleading "background mine captures it" comments in the
save and precompact hooks; update hooks/cursor/README.md and the guide.
Hygiene fixes:
- Drop the hardcoded "version" field from .cursor-plugin/plugin.json and
marketplace.json (mempalace/version.py is the single source of truth);
tests now assert the field stays absent.
- Remove the committed .cursor-plugin/{commands,skills} symlinks (they
break on Windows clones with core.symlinks=false and were redundant
with the real repo-root components that `source: "."` already serves);
add a guard test that no symlinks exist under .cursor-plugin/.
- Document the preCompact synchronous-mine timeout tradeoff and that an
incremental/append-only mine is recoverable if killed (no corruption).
- Add a Cursor-namespaced, daily-throttled TTL sweep (MEMPAL_STATE_TTL_DAYS,
default 30) to lib/common.sh that GCs stale cursor_*.count/.pending only,
after the kill-switch check; shared logs and antigravity_* are untouched.
Verification: full suite green (2424 passed, 3 skipped), ruff check +
format clean, bash -n clean on all cursor scripts. +30 Cursor tests
(followup opt-out, state GC, TTL validation, no-symlink/version guards).
Co-authored-by: Cursor <cursoragent@cursor.com>
* style(tests): apply ruff 0.4.x format to test_normalize
Fixes lint CI: ruff format --check flagged blank-line and long-dict
wrapping in the Continue.dev parser tests.
* fix(searcher): scope neighbor expansion by parent_drawer_id (#1580)
* fix(mcp): drop top-level anyOf from diary_write schema
The mempalace_diary_write tool declared a top-level anyOf in its
input schema to require either entry or content. Anthropic's Messages
API rejects any tool schema with a top-level anyOf/oneOf/allOf and
returns a 400 for the entire tools array, so every MCP session failed
to start.
The entry/content constraint is already enforced at dispatch: content
is remapped to entry before the handler runs, and a missing value
returns -32602. Removing the combinator restores compatibility without
weakening validation.
Closes #1711
* fix: detect Java project manifests
* fix: handle rootless Java subprojects
* fix(mcp): fail closed when add_drawer idempotency pre-check fails
* feat: add mempalace-recall skill and optional Cursor recall rule
Ports the OpenClaw "search before answering" protocol to the Cursor and
Claude plugin surfaces so the agent reads the palace before answering
about past work, people, projects, or prior decisions instead of
guessing from model memory.
- integrations/shared/recall-protocol.md: single source of truth for the
recall protocol, referenced by the skill and the rule so they cannot
drift.
- skills/mempalace-recall/SKILL.md: recall-only skill (the mempalace
skill keeps setup/mine/status); cross-linked from the ops skill.
- rules/mempalace-recall.mdc: plugin recall rule, alwaysApply: false so
it only fires on recall-relevant turns and never adds MCP latency to
greenfield work.
- examples/cursor/rules/: opt-in copies for non-plugin users, including
an aggressive alwaysApply: true variant documented with its latency
tradeoff.
- .claude-plugin/skills/mempalace-recall/SKILL.md: Claude plugin parity.
- tests: assert the recall skill and rules/ discovery layout; the
shipped rule must be alwaysApply: false.
- docs: .cursor-plugin/README.md and the cursor-hooks guide now describe
the three layers of recall (hook + skill + rule).
The Antigravity plugin mirror lands as a follow-up on the antigravity
branch, where .antigravity-plugin/ exists.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(repair): run post-rebuild FTS5 cleanup on legacy cmd_repair path (#1747)
A clean `mempalace repair --yes` (legacy path) finished without
_vacuum_and_rebuild_fts5: the bulk delete_collection + re-upsert cycle
leaves the FTS5 inverted index inconsistent, so the next repair aborts
at the sqlite integrity preflight. rebuild_index() got this cleanup
when #1517 was fixed; cmd_repair never did.
Extract the shared epilogue _post_rebuild_cleanup() (close chroma
handles, then VACUUM + rebuild FTS5) and call it from both full-rebuild
paths so they cannot drift apart again. Cleanup runs on the legacy
success path only; failure/restore paths are unchanged.
Closes #1747
Co-Authored-By: nord- <3777600+nord-@users.noreply.github.com>
* test(backends): live-substrate conformance module for pgvector
Mirrors the portable fake-client arms of test_pgvector_backend.py
against a real PostgreSQL+pgvector server and adds live-only arms the
in-memory fake cannot exercise: real <=> operator ground truth, JSONB
pushdown vs local-fallback equivalence, cross-namespace isolation on
real tables, 8-connection concurrent writers, and the advisory-lock
serialization of run_maintenance('reindex') under a 2-connection race.
Gated on MEMPALACE_PGVECTOR_LIVE_DSN (same pattern as the qdrant live
gate); skips cleanly when unset. First run: 15/15 pass on PostgreSQL
16.10 + pgvector 0.8.2 (+AGE 1.6.0 in the same server), psycopg 3.3.4.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review(gemini): marker-race stub, created-list lock, exact-order + exactly-one-ran asserts
- Stub _write_marker on the 8 concurrent writer backends: upsert()
rewrites the marker on every call with a plain open('w'), so backends
sharing one local_path race on the same file (sharing violations on
Windows) — a test-design artifact, not the contract under test
- Guard the fixture's created list with a lock for the threaded tests
- Assert exact distance-ordered ids in the query/filter arms
- Reindex race: exactly one 'ran' (index absent beforehand, so the
advisory-lock winner must build)
Re-run live after changes: 15/15 pass (PG 16.10, pgvector 0.8.2).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(embedding): chunk EmbeddinggemmaONNX batches to bound ONNX memory (#1770)
One session.run over a repair-scale batch (5000 docs) allocates
attention buffers far beyond available RAM and the kernel kills the
process. Mirror chromadb's ONNXMiniLM_L6_V2 and embed in sub-batches
of 32; per-chunk padding also stops one long doc inflating the whole
batch.
Co-Authored-By: mojie5 <262519016+mojie5@users.noreply.github.com>
* fix(embedding): lock EF cache and lazy load, guard inputs (#1770)
Two threads sharing a cold EmbeddinggemmaONNX via _EF_CACHE could each
build a full model session, and two factory callers could each keep a
private instance. The load is now double-check locked with the session
published last, and the factory cache has an atomic check-then-construct
behind a lock-free fast path. __call__ wraps a bare string, returns []
for None and empty input before the lazy download, and its annotation
matches the accepted types.
* fix(hallways): scope hallway-file path to MempalaceConfig.palace_path (#1778)
Pre-3.4 the hallway store was hardcoded at ~/.mempalace/hallways.json
regardless of the configured palace_path, so two palaces on one host
silently shared one file. Mining into palace-A leaked records into
palace-B's hallway code paths.
Apply the 3.3.6 tunnel-file migration pattern:
* MempalaceConfig.hallway_file resolves to <dirname(palace_path)>/hallways.json
* hallways._get_hallway_file(config) reads through MempalaceConfig
* hallways._legacy_hallway_file() exposes the pre-migration hardcoded path
for one-time orphan detection; _load_hallways logs a one-line warning
when the legacy file exists but the configured one doesn't, matching
palace_graph._load_tunnels behavior. No auto-migration — silent merging
risks clobbering newer data.
Atomic-write + 0600 semantics unchanged. Module-level _HALLWAY_FILE
constant kept and honored when monkey-patched directly, so the three
existing test sites that patch it (test_hallways.py, test_hallways_pagination.py,
test_mcp_server.py) keep working without modification.
New coverage in tests/test_hallways_palace_scoped.py mirrors the analogous
tunnel tests: resolver default + custom palace_path + env-var redirect,
orphaned-legacy warning + no-warning when paths match, and an end-to-end
multi-palace isolation regression guard.
Closes #1778
* fixup(hallways): drop _HALLWAY_FILE back-compat shim, migrate existing tests to resolver
Replaces the back-compat shim in _load_hallways/_save_hallways (which
honored direct monkey-patches of the _HALLWAY_FILE module constant) with
a clean single-source-of-truth resolver, matching the palace_graph
tunnel-file migration in 3.3.6.
The three existing test sites (tests/test_hallways.py,
tests/test_hallways_pagination.py, tests/test_mcp_server.py) now
monkey-patch _get_hallway_file and _legacy_hallway_file directly,
exactly mirroring the helper in tests/test_palace_graph_tunnels.py.
Production code now has one branch through the path resolution instead
of two. No behavior change. 269/269 hallway + tunnel + mcp-server
tests pass on Python 3.11 and 3.12, ruff clean.
* fixup(hallways): address gemini-code-assist review on PR #1780
Two catches on tests/test_hallways_palace_scoped.py
TestMultiPalaceIsolation.test_save_then_load_under_different_palace_returns_empty:
1. Stale comment referencing the removed _HALLWAY_FILE back-compat shim
(deleted in the prior fixup commit). Removed.
2. _legacy_hallway_file was not monkey-patched, so the test isolation gap
let _load_hallways check the host's real ~/.mempalace/hallways.json
when evaluating the legacy-warning branch. Now patched to a tmp_path
sibling, matching the helper pattern used in test_palace_graph_tunnels.
* fix(ids): use length-prefixed recipe v3
* fix(mcp): treat chunked drawers as logical drawers
* fix(mcp): avoid mutating drawer metadata
* fix(ids): simplify v3 length-prefixed hashing
* chore(deps): bump docker/metadata-action from 5 to 6
Bumps [docker/metadata-action](https://github.com/docker/metadata-action) from 5 to 6.
- [Release notes](https://github.com/docker/metadata-action/releases)
- [Commits](https://github.com/docker/metadata-action/compare/v5...v6)
---
updated-dependencies:
- dependency-name: docker/metadata-action
dependency-version: '6'
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): bump docker/build-push-action from 6 to 7
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6 to 7.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v6...v7)
---
updated-dependencies:
- dependency-name: docker/build-push-action
dependency-version: '7'
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): bump docker/login-action from 3 to 4
Bumps [docker/login-action](https://github.com/docker/login-action) from 3 to 4.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...v4)
---
updated-dependencies:
- dependency-name: docker/login-action
dependency-version: '4'
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix(hooks): normalize Windows transcript paths in shell hooks
* fix(backends): serialize first connect in sqlite_exact and pgvector (#1774, #1775)
Co-Authored-By: jphein <19301265+jphein@users.noreply.github.com>
* fix(hooks): preserve fail-loud parse diagnostics
* fix(hooks): typo regression addressed
* fix(tests): run fact_checker __main__ via subprocess to clear runpy warning
`tests/test_fact_checker.py` imports symbols from `mempalace.fact_checker` at
module top (putting it in sys.modules), then `TestCLI.test_exits_nonzero_when_
issues_found` re-executed the same module as __main__ via
`runpy.run_module("mempalace.fact_checker", run_name="__main__")`. runpy warns
because it re-runs an already-imported module against a half-initialized state:
RuntimeWarning: 'mempalace.fact_checker' found in sys.modules after import of
package 'mempalace', but prior to execution of 'mempalace.fact_checker'
Run the CLI in a fresh process via `subprocess.run([sys.executable, "-m",
"mempalace.fact_checker", ...])` instead — no sys.modules collision, and it
exercises the real `python -m` entry point. Assertions are preserved
(SystemExit code 1 → returncode 1; captured stdout substring → result.stdout).
The child's entity registry (`~/.mempalace/known_entities.json`, resolved via
expanduser at import) is redirected by overriding both HOME and USERPROFILE in
the subprocess env so it works on POSIX and Windows.
Verified: `pytest tests/test_fact_checker.py -W error::RuntimeWarning` passes
(26) with the warning promoted to error — proving it no longer fires.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(mcp): avoid Chroma open when cached DB disappears
* test: stabilize release validation on develop
* fix(palace): clean source mine locks safely
* fix: close blob seq sqlite migration connection
* fix: address mine lock review feedback
* test: stabilize closet boost fixture on Windows
* chore(release): 3.4.1
Bump version across all sources (version.py, pyproject.toml, both
Claude plugin manifests, Codex plugin manifest, README badge, uv.lock)
and promote the Unreleased changelog to 3.4.1.
Shipping: Cursor IDE plugin + hooks, first-class Antigravity IDE
support (with zero-config interpreter resolution), embeddinggemma
bulk re-embed OOM fix, and backup-retention pruning.
Also rebuilds the CHANGELOG compare-link block, which had been left
at v3.2.0: adds the full 3.3.0-3.4.1 chain plus the previously
undocumented 3.4.0, and points Unreleased at v3.4.1...HEAD. Every
version header now resolves to a compare link.
* fix(hooks): portable mtime in macOS hook throttles; doc cleanup
Address review feedback surfaced on the 3.4.1 release promotion (#1810).
Bug fix — `date -r FILE` is GNU-only. On BSD/macOS `date -r` expects
epoch seconds, not a path, so the staleness/throttle checks in the new
Cursor and Antigravity hooks silently failed on macOS: the state GC
swept on every fire and the pending-save guard was skipped. Replace
with a portable `os.path.getmtime` one-liner via the already-resolved
$MEMPAL_PYTHON_BIN (cursor/lib, antigravity/lib, antigravity save hook).
This restores the "bash 3.2.57 / macOS default" compatibility the
Antigravity changelog claims.
Docs:
- Correct the MCP tool count to 33 (was 19/29/31 in 21 places across
plugin manifests, READMEs, and website docs — all drifted from the
TOOLS dict / mcp-tools.md reference, which both have 33).
- Fix broken CHANGELOG link to the Cursor skill (skills/, not
.cursor-plugin/skills/).
- Fix one-too-many `../` in skills/mempalace/SKILL.md's cursor-hooks
link (resolved above the repo root).
- Add the required `mcpServers` wrapper to the mcp.json example in
.cursor-plugin/README.md so copy-paste yields a valid Cursor config.
Left intentionally unchanged: the os.dup2 fd-1 redirect in
mcp_server.py is deliberate (#225 keeps JSON-RPC off fd 1).
* style(hooks): single-quote the static python -c mtime snippet
The snippet has no shell interpolation — the path arrives via argv, not
string interpolation — so single quotes are correct and make it
unambiguous that nothing is shell-expanded. Behavior is identical:
`sys.argv[1]` contains no `$`, so it was never expanded (verified
empirically). Matches the single-quoted `python -c` blocks already in
hooks/cursor/lib/common.sh. No functional change.
* test(migrate): cover swap-failure rollback
Adds end-to-end regression coverage for the migration swap path where os.replace hits EXDEV, the shutil.move fallback fails, and the original palace must be restored from the rename-aside copy.
* feat(mcp): add mempalace_delete_by_source bulk-cleanup tool (#1722)
Adds an MCP tool to remove every drawer mined from a given source_file
exact match, for cleaning up benchmark/test data accidentally mined into
a user wing (ShareGPT dumps, results_mempal_*.jsonl, language config
JSON) that drowns out real memories in semantic search.
Matching is pushed to the backend via delete(where={"source_file": ...})
the same idiom the miner and diary-ingest paths already use so it is not
subject to the SQLite variable limit regardless of how many drawers share
the source. Defaults to a dry run reporting match count and a sample;
dry_run=false commits. Absent source is an idempotent no-op, not an error.
* fix(mcp): harden delete_by_source per review — strip surrogates + type guard
Address Gemini review on #1729:
- normalize source_file with strip_lone_surrogates so exact matching hits
rows mined from non-ASCII paths via cp1252 stdin (#1488), mirroring
tool_add_drawer's ingestion-side normalization
- isinstance(str) guard so a non-string source_file returns a clean error
instead of AttributeError
- default missing wing/room to "" in the dry-run sample, consistent with
the rest of the file
- add tests: non-string rejection + surrogate-normalization match
* feat(miner): add PHP ecosystem file extensions
* fix(claude-plugin): run final mine on SessionEnd
* fix reviewer feedback: To prevent a KeyError and provide a clear, actionable assertion failure message if PreCompact is ever missing
* fix(chroma): route stale hnsw divergence to sqlite fallback
* fix reviewer feedback for chroma and tests
* fix(mcp): refuse second writer for same palace
* fix(mcp): cache writer lock setup failures
* feat: add opt-in local daemon for queued MemPalace writes
- New mempalace/daemon.py: long-lived localhost HTTP server (127.0.0.1) with a
SQLite WAL job queue, single worker thread, bearer-token auth, and owner-only
file perms (0600/0700) on queue DB, token, endpoint, and log.
- New mempalace/service.py: transport-neutral job execution surface shared by the
daemon, with per-job env isolation so one job's backend/palace switch cannot
leak into the next. mcp_tool is allowlisted to write-classified tools only.
- Crash recovery re-queues jobs left 'running' by a killed daemon; jobs that
already exhausted MAX_ATTEMPTS are dead-lettered to 'failed' instead of being
retried (non-idempotent diary_write would otherwise duplicate verbatim
content on every restart).
- Bounded retention prunes terminal jobs older than 7 days
(MEMPALACE_DAEMON_RETENTION_DAYS); queued/running jobs are never touched so a
crash mid-prune cannot drop in-flight work.
- CLI: --daemon/--background on mine/sync submit to the queue; new
`mempalace daemon {start,stop,status,jobs,wait}` subcommand. Strictly opt-in:
no flag, env, or config means no daemon and no behavior change.
- Hooks opt in via MEMPALACE_HOOKS_DAEMON or config hooks.daemon; when the daemon
is not already running, hooks fall back to the existing direct/spawn path so
the 500ms hook budget is preserved (hooks never auto-start the daemon).
- service.run_sync renders the same operator-facing report shape as the direct
CLI sync path (no_source, out_of_scope, by_source, Re-run/Removed hints) and
drops the old KeyError-prone 'deleted' read.
* fix(mcp): gate startup on sqlite integrity failures
* chore(deps-dev): bump ruff from 0.15.15 to 0.15.18
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.15 to 0.15.18.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/0.15.15...0.15.18)
---
updated-dependencies:
- dependency-name: ruff
dependency-version: 0.15.18
dependency-type: direct:development
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix(mcp): applied 3 of the 4 reviewer suggestions
* fix(mcp): guard remaining None palace_path in _mcp_sqlite_integrity_refusal. Added one regression test calling the function directly with palace_path=None.
* fix: unblock daemon PR CI + address review comments
CI was red on all three platforms for the daemon-mode draft PR. Root causes
and fixes:
- Linux 3.9 collection error: `_submit_daemon_job`'s `dedupe_key: str | None`
parameter annotation is evaluated at def time, and hooks_cli.py has no
`from __future__ import annotations` — `str | None` raises TypeError on 3.9.
Reverted to `dedupe_key: str = None` (the original, 3.9-safe). The other
`int | None` in the file is a function-local annotation, which is never
evaluated, so it was never the problem.
- macOS/Windows daemon lifecycle flakes: the 3 HTTP-lifecycle tests failed at
the 10s readiness deadline on contended CI runners (localhost bind is
sub-second locally but took ~5s when it passed on the macOS fleet, >10s when
it didn't), and because the server thread never shuts down on timeout,
run_server's `os.environ["MEMPALACE_PALACE_PATH"]` + `os.umask(0o077)`
mutations leaked into the rest of the suite — poisoning every later test that
reads MempalaceConfig().palace_path (the 60+ test_mcp_server cascade on macOS;
the at-exit socket hang → SIGINT on Windows). Bumped the readiness deadline
to 30s and added a module-scoped snapshot + autouse fixture in test_daemon.py
that force-restores the env + umask to the pre-suite baseline after every
daemon test, so a leaked server thread can't poison other test files.
Gemini review comments (fixed in code, no thread replies per convention):
- daemon.py `_connect()` was a bare `sqlite3.connect` whose `with`-block only
managed the transaction, not the connection — an unbounded FD leak in a
long-lived daemon running thousands of jobs (also the source of the Windows
"unclosed database" ResourceWarning noise). Converted to a closing
@contextlib.contextmanager.
- `QueueStore.finish()` gained `only_if_running`; `_safe_finish` passes it so a
late worker finish can't overwrite a shutdown-cancelled job back to
succeeded/failed — removes the reliance on process-exit timing.
- `DaemonClient.request` wraps the final `json.loads` in try/except
JSONDecodeError → DaemonError, so a non-JSON 2xx response surfaces as a
structured error instead of a bare JSONDecodeError.
- test_sync.py: removed the module-level `import mempalace.mcp_server` and moved
the stdout-rebinding side effect into an autouse fixture scoped to
TestServiceRunSyncReport, so the embedder/Chroma import chain is no longer
forced at collection time for the existing sync tests.
Coverage: added focused happy-path tests for service.run_sync early-returns,
run_mine backend application + invalid mode, execute_job kind dispatch,
run_diary_write arg forwarding, run_mcp_tool write-tool dispatch, and
print_job_result — lifts service.py from 57% to 85% so the new files
(service 85%, daemon 80%) don't drag the total below the 80% CI gate now that
the daemon tests complete and the gate is actually evaluated.
* fix: daemon client bypasses proxy discovery; tests force-shutdown server thread
DaemonClient.request now uses a no-proxy opener (build_opener(ProxyHandler({})))
instead of urllib.urlopen. The daemon is always on 127.0.0.1, so a request must
never go through an HTTP proxy — this is the correct production choice. It also
bypasses urllib's proxy discovery (macOS _scproxy via SystemConfiguration), which
runs on the first request to any host and is NOT bounded by the per-request
timeout: on a CI runner with no network it hangs for tens of seconds, which looked
exactly like the daemon never came up (test_daemon_http_lifecycle_executes_job
timed out at 30.18s). With the no-proxy opener the lifecycle runs in 0.78s and no
server thread is leaked — which also removes the timing skew that made the
sqlite_exact concurrent-connection test flake on macOS CI.
The leaked server thread was also the Windows exit-hang root cause: a slow/failed
client.shutdown() POST left serve_forever running, and the interpreter blocked on
the open listening socket at process exit. Tests now capture the httpd run_server
creates (by subclassing daemon.ThreadingHTTPServer) and force httpd.shutdown() +
server_close() from the test thread if the normal shutdown path leaves the thread
alive, asserting the thread died so a leak becomes a visible failure instead of
a silent exit hang.
* test: win32-only diagnostic for daemon process-exit hang
The Windows CI run passes all 666 tests then hangs at interpreter shutdown
(KeyboardInterrupt at socket.py:723) until the runner kills it. All daemon
lifecycle tests assert their server threads died, so the hang is a different
non-daemon thread blocked on a socket — not the daemon server thread. CI
round-trips can't show which thread it is.
Add a win32-only session fixture that:
- arms faulthandler.dump_traceback_later(130s) to print every thread's stack
to stderr once the hang has run a while, and
- prints every live thread (name + daemon flag) at session teardown — a
non-daemon thread present there is the shutdown blocker.
Gated to sys.platform == 'win32' so Linux/macOS CI see no extra output. Remove
once the Windows hang is fixed.
* fix(daemon): Windows-safe pid liveness probe; finalize cross-platform daemon tests
_pid_alive used os.kill(pid, 0) as an existence check. On Windows signal 0
is signal.CTRL_C_EVENT, so Python routes it to GenerateConsoleCtrlEvent and
sends a console Ctrl-C to the target's process group rather than probing the
pid. DaemonClient polls a same-process endpoint during startup, so on a CI
runner with an attached console that Ctrl-C was delivered back to the
interpreter as a spurious KeyboardInterrupt — the Windows CI hang that
interrupted the suite at the first daemon HTTP-lifecycle test (socket.py
recv). Probe via the Win32 OpenProcess/WaitForSingleObject handle API instead,
which has no signalling side effects. This is also a real Windows production
bug, not just a test artifact.
Tests:
- Skip the two owner-only (0600) permission tests on Windows: os.chmod cannot
represent POSIX mode bits there (files report 0o666); the daemon relies on
user-profile ACLs on Windows.
- _start_server now captures and re-surfaces a run_server thread crash instead
of spinning for 30s and failing with a bare assert (diagnoses the macOS
startup flake).
- Add a regression test asserting _pid_alive is correct and emits no console
control event when hammered like the poll loop.
- Remove the temporary win32 exit-hang diagnostic fixture from conftest now
that the root cause is fixed.
* fix(daemon): skip reverse-DNS in server_bind so startup can't block ~30s
HTTPServer.server_bind() resolves server_name via socket.getfqdn(host). For the
daemon's 127.0.0.1 bind that lookup is pointless, and on a host with slow or
absent reverse DNS it blocks startup until the resolver times out (~30s) — which
looks exactly like the daemon never coming up. This is why the first daemon
HTTP-lifecycle test timed out on the macOS CI runner (httpd_bound=False after
30s) while every later one bound in seconds once the OS had cached the negative
lookup. Bind via TCPServer directly and set server_name from the literal host.
* feat(hooks): add a budget-safe SessionEnd save hook for clean exits (#1341)
Short sessions that exit cleanly below SAVE_INTERVAL and without a PreCompact
were never saved. Add a SessionEnd hook that takes one final flush.
Claude Code budgets SessionEnd hooks at 1.5s and a plugin-provided timeout
cannot raise it, and a cold mempalace start exceeds that, so the wrapper
backgrounds the work and returns immediately; the detached child completes the
transcript ingest, project mine, and diary checkpoint after the session exits.
The handler validates transcript_path through _validate_transcript_path before
any ingest or diary write, so a traversal or wrong-suffix path is rejected while
the independent project mine still runs.
Adds hook_session_end, both shell wrappers, the plugin hooks.json entry, the
session-end CLI choice, and focused tests.
(cherry picked from commit 10e1450e04fc7cec72984ab7442d3b4fca1490e8)
* fix(claude-plugin): resolve SessionEnd merge semantics
* fix(claude-plugin):reviewer feedback for _validate_transcript_path function calls Path.resolve(), which can raise an OSError
* fix(daemon): address post-merge review feedback on #1826
Five fixes from the Copilot review of the merged daemon PR:
1. Privacy: the queue DB's SQLite WAL/SHM sidecars hold un-checkpointed
verbatim payloads but were created with the caller's umask. Set the
owner-only umask in run_server BEFORE DaemonRuntime builds the QueueStore
(not only once the HTTP server starts), and harden any existing sidecars in
QueueStore._init_db as defense-in-depth.
2. DoS guard: reject a negative Content-Length in the request reader.
rfile.read(-1) would block until the client disconnects and bypass the
MAX_BODY_BYTES cap.
3. Side effects: extract _wal_log (+ _ensure_wal, _WAL_FILE, _WAL_REDACT_KEYS)
into a new side-effect-free mempalace/wal.py. The CLI sync path and the
daemon service layer obtained _wal_log via `from .mcp_server import _wal_log`,
which runs mcp_server's import-time stdio protection (os.dup2(2, 1);
sys.stdout = sys.stderr) in a non-MCP process and misroutes operator output.
mcp_server/cli/service now import from mempalace.wal.
4. Correctness: run_mcp_tool treated any dict as success. Write tools that
return a bare {"error": ...} (e.g. tool_create_tunnel/tool_delete_tunnel
validation) were recorded as succeeded; now the "error" key infers failure.
5. Hook budget: get_client_if_running()/health() take an explicit timeout, and
the hook "is the daemon up?" precheck uses a short HOOK_PROBE_TIMEOUT (0.5s)
so a wedged daemon can't stall the hook for the default 5s.
Adds tests/test_wal.py (import isolation + redaction) and daemon tests for the
umask ordering, negative Content-Length, run_mcp_tool error inference, and the
short probe timeout.
* fix(backends): single-scroll bulk metadata fetch for Qdrant; bump scroll page size (#1796)
* fix: apply suggested reviewer suggestions
* updated tests/test_qdrant_bulk_metadata_scroll.py because of CI failure after push the 2nd commit
* perf(embedding): cap ORT intra-op threads so a background mine doesn't pin every core (#1068)
ChromaDB's ONNX embedder builds its InferenceSession without a thread cap, so
ORT's intra-op pool defaults to the physical core count. OMP_NUM_THREADS is
inert against it (ORT owns its own pool), so a background `mempalace mine`
pins 4-5 cores and stacked Stop-hook fires turn the machine into a thermal
event.
Add an `embedding_threads` config knob (env MEMPALACE_EMBEDDING_THREADS or
config.json). Unset/"auto" caps the intra-op pool at half the logical CPUs so
a fresh install stays usable out of the box; a positive integer sets an exact
count; 0/negative leaves ORT uncapped for users who want max throughput.
The cap is applied via SessionOptions at session construction:
- `_MempalaceONNX` (default minilm) overrides the `model` cached_property to
rebuild the session the same way upstream does plus the cap, falling back to
upstream's uncapped build if chromadb internals shift.
- `EmbeddinggemmaONNX` builds its session through the shared
`_intra_op_session_options()` helper.
* perf(mcp): answer overview tools from the sqlite aggregate to fix large-palace timeouts (#1748, #1379)
tool_status / list_wings / list_rooms / get_taxonomy paged the entire
collection metadata through the chroma client (`_fetch_all_metadata`, a
1000-row offset loop), which cold-loads the HNSW index and materializes
hundreds of MB of dicts. On six-figure palaces these exceed the MCP host
tool-call limit (180k drawers ~3-4 min; 349k times out at 120-240s). The 5s
metadata cache only dedups repeat calls — it does not stop the cold-call
timeout.
A correct single-query SQL cross-tab already exists
(`backends.chroma._sqlite_wing_room_counts`) and is already the CLI default
(`miner.status`), but the MCP tools never used it — and the MCP-side sqlite
reader only ran behind the `vector_disabled` recovery path.
Add `_sqlite_taxonomy()` (guards on `_is_chroma_backend()`, returns None to
fall back) and wire it as the default path into all four overview tools. They
now answer from one GROUP BY without touching HNSW. Non-chroma backends
(qdrant, sqlite_exact) and unbootstrapped/legacy layouts fall back to the
existing client path unchanged.
graph_stats (also named in #1379) builds an in-memory graph via build_graph()
and needs its own treatment — tracked separately.
* fix(pgvector): strip NUL bytes so a transcript NUL no longer aborts the mine (#1829)
PostgreSQL cannot store NUL (0x00) in text or jsonb. On the pgvector write path
a NUL in `document` is rejected by psycopg ("PostgreSQL text fields cannot
contain NUL (0x00) bytes") and a NUL in `metadata` becomes a JSON unicode escape
the jsonb cast rejects ("unsupported Unicode escape sequence"). `_execute`
re-wraps either as BackendError and `_mine_impl` re-raises, so the whole mine
exits non-zero and every file after the offending one is left unmined. ChromaDB,
SQLite, and Qdrant store the byte verbatim, so only pgvector hard-fails.
Add a recursive `_strip_nul` helper and apply it to id, document, and metadata
in `_PgVectorClient.upsert_rows`, mirroring the backend-layer sanitization
`_sanitize_documents_for_chromadb` already does for lone surrogates on the same
bulk-ingest paths. ids are SHA-256 hashes and metadata keys are fixed field
names, so the id and key passes are no-ops in practice; only transcript-derived
values change.
Co-authored-by: hrabbach <181709360+hrabbach@users.noreply.github.com>
* fix: address PR review feedback (preserve "unknown" label; use super().model)
#1748: normalize the sqlite fast path's "?" COALESCE placeholder (and None)
back to "unknown" inside _sqlite_taxonomy, so drawers missing wing/room
metadata keep the client path's output contract — no observable API change
for MCP clients on legacy/partial drawers.
#1068: invoke the parent embedder build via super().model instead of reaching
into cached_property's .func attribute, so the uncapped/fallback path survives
chromadb changing `model` to a plain @property or other descriptor.
* perf(mcp): sqlite fast path for graph_stats to fix large-palace timeouts (#1379)
tool_graph_stats built the whole palace graph via build_graph(), which pages
every metadata row (col.get limit/offset) and cold-loads the HNSW index — the
remaining overview-tool timeout from #1379 (#1836 fixed status / list_wings /
list_rooms / get_taxonomy but deliberately left graph_stats out, as it builds
an in-memory graph rather than a flat tally).
Add _sqlite_graph_stats(): one GROUP BY room, wing, hall over chroma.sqlite3,
reconstructing build_graph's room_data and the same stats (total_rooms,
tunnel_rooms, total_edges, rooms_per_wing, top_tunnels) with the same
per-drawer filter (room present, != "general", wing present) and edge
semantics (C(wings, 2) * halls per multi-wing room). Same _is_chroma_backend()
guard + client-path fallback as the #1748 overview tools.
Test seeds a real chroma palace mirroring the build_graph parity case in
test_palace_graph, with a tripwire on graph_stats proving the fast path runs
and that "general"/wing-less drawers are excluded. Idea adapted from #1381's
_sqlite_graph_stats.
* fix: address PR review feedback on graph_stats sqlite fast path (#1379)
- Soft-fallback on any exception, not just sqlite3.Error, so an unexpected
schema shape tripping the reconstruction degrades to build_graph() instead
of raising — matching the sibling sqlite fast paths (Copilot).
- Guard an empty/None _config.palace_path before building db_path (Gemini).
- Test: tripwire _get_collection in addition to graph_stats, directly
asserting the fast path never opens the chroma client / cold-loads HNSW
(Copilot).
* fix: percent-encode sqlite read-only URIs so spaced/special-char paths open
sqlite3.connect(f"file:{db_path}?mode=ro", uri=True) mis-parses paths
containing spaces or other URI-reserved characters — common in real home
directories (a Windows "First Last" user folder, many macOS paths), and made
worse by Windows backslashes. The database silently fails to open and the
read-only fast paths fall back (or error) on those machines.
Add config.sqlite_read_uri(), which percent-encodes the path via
urllib.request.pathname2url (lazy-imported to keep config import light), and
route every read-only sqlite reader through it:
- mcp_server._tool_status_via_sqlite
- searcher BM25 sqlite fallback
- repair (status / scan / max-seq read paths)
- backends/chroma (5 readers: counts, wing/room tally, id maps, etc.)
All previously used the same naive f-string construction. Surfaced as a
gemini-code-assist review note on #1837.
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix(mcp): route _sqlite_graph_stats through sqlite_read_uri
The graph_stats sqlite reader (#1837) and the sqlite_read_uri encoding fix
(#1838) landed in parallel, so _sqlite_graph_stats was the one reader left on
the naive f"file:{db_path}?mode=ro" construction that mis-parses paths with
spaces/special chars. Convert it too — now every read-only sqlite reader
percent-encodes its path. sqlite_read_uri is already imported in mcp_server
from #1838, so this is a call-site-only change.
* fix(pgvector): push get(limit, offset) pagination into SQL (#1830)
PgVectorCollection.get(limit=, offset=) ignored pagination at the SQL
layer: scroll_rows ran SELECT ... WHERE <where> with no LIMIT/OFFSET, so
get() fetched the whole table and sliced in Python. prefetch_mined_set
pages the whole palace on every mine, so mining was O(N^2) in rows
transferred and Python objects built as the palace grows; every other
paginating caller (exporter, migrate, repair, hallways, closet_llm,
miner, palace_graph) paid the same cost.
Push LIMIT/OFFSET into scroll_rows/_scroll with ORDER BY id (the primary
key) for stable offset pagination. get() uses the pushed path only for an
unfiltered page (no ids, no where/where_document, non-negative bounds); a
filtered get keeps the full-scan path because the metadata @> ... pushdown
is broader than the exact _matches_where re-filter for array/object
values, so that re-filter must run before pagination. Full-scroll callers
pass no bound, so their SQL is unchanged.
Co-authored-by: hrabbach <181709360+hrabbach@users.noreply.github.com>
* fix(pgvector): replace lone surrogates so a transcript surrogate no longer aborts the mine (#1833)
A lone UTF-16 surrogate (U+D800-U+DFFF) in transcript content has no UTF-8
encoding, so pgvector's bulk upsert_rows makes psycopg raise UnicodeEncodeError
and the whole mine aborts, leaving later files unmined.
Apply config.strip_lone_surrogates (-> U+FFFD) to id, document, and the
serialized metadata JSON in upsert_rows. json.dumps(ensure_ascii=False) leaves a
metadata surrogate raw in the string, so one pass over the serialized JSON covers
it; NUL, by contrast, json-escapes and must be stripped before serialization
(see #1829). Replace rather than drop, matching ChromaDB's document handling.
Verified end to end against live Postgres + pgvector: before, a surrogate in
document or metadata aborts the mine; after, it ingests and round-trips as U+FFFD.
Fixes #1833
* fix(backends): push sqlite_exact get(limit, offset) pagination into SQL
SQLiteExactCollection.get(limit, offset) fetched the whole collection via
_rows() (SELECT ... FROM documents ORDER BY rowid, no LIMIT/OFFSET) and sliced
in Python, so every paginating caller (prefetch_mined_set, status, exporter,
migrate, dedup, sync, ...) re-scanned the entire table per page, making the
sweep O(N^2) in rows materialized.
Push LIMIT/OFFSET into the scan on the unfiltered page (no ids/where/
where_document and non-negative bounds); filtered, id, and negative pages keep
the full-scan plus Python-slice path so the post-filter still runs first. SQLite
requires a LIMIT before OFFSET, so an offset-only page uses LIMIT -1. ORDER BY
rowid keeps pages stable.
* ci: re-trigger checks (unrelated Windows closet flake)
* refactor(qdrant): reuse _rows() in get_all_metadata(); fix sys.modules test pollution
Addresses maintainer review on #1832 (the two non-blocking 🟡 items plus
two 🟢 nits)
* test(mcp_server): add missing _fetch_all_metadata delegation/fallback tests.
* feat(search): add an optional source_file filter to mempalace_search (#1815)
Expose source_file alongside wing/room on mempalace_search. build_where_filter
generalizes to 0/1/2+ clauses and the filter threads through the main vector
path, the index-mismatch fallback, the vector-disabled BM25/SQLite path, and
the union lexical path so it never silently no-ops. Matching is on the exact
full stored value; results now expose source_path (the full path) for round
tripping, since the displayed source_file is a basename. The MCP schema gains
the source_file property and a path-tolerant sanitizer rejects null bytes,
lone surrogates, and overlong values.
Fixes #1815
Co-Authored-By: rendigua2025-gif <253093224+rendigua2025-gif@users.noreply.github.com>
* fix(mcp): reject non-string source_file with a clean error (#1815)
A JSON number or boolean passed for source_file is not coerced by the
string schema type, so it reached _sanitize_optional_source_file and
raised AttributeError from .strip() rather than a clean validation error.
Add an isinstance guard that raises ValueError, which tool_search returns
as a structured error. Regression test added.
* ci: re-trigger Windows (flaky closet-boost test)
* fix(repair): point index-read failures to repair --mode from-sqlite (#1843)
When the chromadb compactor cannot apply the WAL into the drawers HNSW
segment (InternalError: Failed to apply logs to the hnsw segment writer),
the legacy repair paths fail on their first Collection.count() read and
advise re-mining from source files. The drawer rows are intact in
chroma.sqlite3, so repair --mode from-sqlite rebuilds them; re-mining
silently drops drawers added via the MCP server and diary entries that
have no source file.
Both legacy read-failure sites (cmd_repair and rebuild_index) now emit
shared guidance pointing at the from-sqlite recovery, worded conditionally
so it also covers a live server or mine still holding the palace open.
Co-Authored-By: undeadindustries <9536461+undeadindustries@users.noreply.github.com>
* fix: use CREATE_NO_WINDOW so Windows hook miner spawns don't flash a console (#1783)
Fixes #1783
* fix: point diverged-index recovery at from-sqlite, not re-mine (#1843)
A diverged HNSW index (for example after a failed chromadb compaction)
leaves the drawer rows intact in chroma.sqlite3 but the vector index out
of sync. Re-mining to recover silently drops drawers added through the
MCP server and diary entries, which have no source file.
- repair-status now recommends `mempalace repair --mode from-sqlite
--archive-existing` when DIVERGED, instead of the generic `mempalace
repair`, and explains why re-mining loses data.
- The shared recall protocol and the recall skills (Cursor + Claude
plugin) document the compactor / "Not connected" recovery path:
stop the server, rebuild from SQLite, verify, restart — never repair
in-process from the agent.
Complements #1847 (legacy repair error messages); does not duplicate it.
Does not close #1843 — MCP reconnect resilience and honest add_drawer
write signalling remain open.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs: add Windows backup alternative to corrupt-index recovery (#1843)
Gemini review on PR #1849: the optional palace backup step used the
Unix-only `cp -a`, which fails on Windows. MemPalace ships on win32, so
add a PowerShell `Copy-Item` alternative alongside the macOS/Linux form
and note that `--archive-existing` already moves the old palace aside.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: add mempalace_checkpoint batch save tool
Collapse the Cursor auto-save sequence (check_duplicate Nx + add_drawer
Nx + diary_write 1x) into a single mempalace_checkpoint MCP call so the
host UI renders one tool-call card and keeps its spinner up for the whole
save. The new tool reuses the existing single-item handlers, so semantic
dedup, idempotency, and verbatim guarantees are unchanged.
- mcp_server.py: add tool_checkpoint + register mempalace_checkpoint
- service.py: classify mempalace_checkpoint as a write tool
- cursor save hook: followup now drives one mempalace_checkpoint call
- docs: new mcp-tools.md section, help.md entry, 33 -> 34 tool count sweep
- tests: checkpoint add/dedup/malformed/registry + classify_tool
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: harden tool_checkpoint input validation
Address PR review: guard untrusted MCP client payloads in
mempalace_checkpoint so a single malformed item cannot raise deep in
sanitization and abort the whole batch.
- coerce dedup_threshold to float
- require wing/room/content to be non-empty strings (skip + record error)
- validate the diary object and entry type, recording errors instead of
silently ignoring a malformed diary
On a genuine dedup-check error we still file the drawer rather than skip:
verbatim recall is the priority and add_drawer's idempotency blocks exact
duplicates. Adds tests for the non-string, dedup-error, and malformed-diary
paths.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: update Cursor followup assertion for checkpoint tool
The save-hook followup now drives a single mempalace_checkpoint call, so
test_threshold_emits_followup_message must assert that tool name instead
of the old add_drawer/check_duplicate/diary_write trio. Fixes the
test-macos / test-linux CI failures on this branch.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): purge matching closets in delete_by_source (#1722)
delete_by_source removed only the drawers, leaving the matching closets
(the AAAK index layer, keyed independently by source_file) behind as stale
pointers at the now-deleted source. Mirror the closet-purge step used by
sync_palace / purge_file_closets: after the drawer delete, best-effort purge
the closets via push-down delete(where=...) so it survives large palaces and
can never abort an already-committed drawer delete.
Dry run now also reports closet_match_count so the caller sees the full blast
radius; commit reports closets_deleted. Adds tests that seed the closet
collection directly (tool_add_drawer doesn't build closets) and assert the
matching closets are purged on commit and counted on dry run.
* feat(mcp): add opt-in HTTP transport
* fix suggestion of reviewer to avoid a critical race condition and other fixes
* test(mcp): keep HTTP transport tests Python 3.9 compatible
* test(mcp): avoid subprocess flakiness in HTTP transport tests
* test(mcp): bypass proxies in HTTP transport loopback tests
* test(mcp): make HTTP transport loopback tests proxy-free
* test(mcp): make HTTP transport tests network-free
* fix(mcp): move _HTTP_REQUEST_LOCK and _HTTP_MAX_REQUEST_BYTES
* fix(mcp): move _HTTP_REQUEST_LOCK
* fix reviewer: handling JSON-RPC
* fix(tests): rewrite test_mcp_http_transport for Python 3.9-3.13 + Windows
* fix(lint): resolve 7 ruff errors in test_mcp_http_transport
* fix(mcp): harden HTTP transport — DNS-rebinding guard, optional token, real tests
The opt-in HTTP transport reuses the stdio dispatcher and binds loopback by
default, but /mcp was unauthenticated with no protection against a malicious
web page reaching a DNS-rebound localhost server, and its tests reached for
Starlette/uvicorn (not project deps) so they were silently skipped in CI —
the production _serve_http handler had zero coverage.
Hardening:
- Pin the Host header to loopback literals + the bound host on a loopback bind
(DNS-rebinding defense); relaxed for a deliberately non-loopback bind, which
is the operator's call and may sit behind a Host-rewriting proxy.
- Reject any browser Origin that isn't a loopback origin (rebinding/SSRF guard);
non-browser MCP clients omit Origin and are unaffected.
- Optional bearer token via MEMPALACE_MCP_HTTP_TOKEN (constant-time compare);
required on /mcp, never on /healthz so liveness probes work credential-free.
- Warn loudly when bound to a non-loopback host (palace reachable from network).
Testability:
- Split _build_http_server() out of _serve_http() so tests bind 127.0.0.1:0 and
drive the real handler over a loopback socket via stdlib http.client.
- Replace the skipped Starlette reimplementation with 12 tests covering dispatch,
initialize, /healthz, 404, parse-error, the 16 MiB cap, notification 202, and
the Host/Origin/token rejections — no third-party deps.
* ci(test-windows): retry the transient ChromaDB HNSW compaction flake
ChromaDB's rust HNSW core intermittently fails compaction on Windows with
"Failed to apply logs to the hnsw segment writer" during add/update — a
long-standing, non-reproducible-on-Linux/macOS flake that hits different tests
(test_migrate_wings, test_closets) across unrelated commits and has been
turning otherwise-green release/CI runs red at random.
Add pytest-rerunfailures and wire `--reruns 2 --only-rerun "Failed to apply
logs to the hnsw segment writer"` into the test-windows job only. The
--only-rerun scope means a real, deterministic failure still fails on the first
run; only this specific transient native-dependency error is retried. The
Linux and macOS jobs deliberately keep zero reruns so genuine regressions
surface there loudly.
* chore(release): 3.5.0
Bump version to 3.5.0 across version.py, pyproject.toml, the Claude/Codex
plugin manifests, the README badge, and uv.lock. Refresh the "N MCP tools"
prose from 34 to 35 (delete_by_source #1729 and checkpoint #1851 each added a
tool). Add the 3.5.0 CHANGELOG entry.
* fix: tighten local guards and file handling
* fix: restore convo miner scan indentation
* fix: green up CI for hardened file handling
- ruff format llm_client.py and miner.py (lint job)
- _copy_file_no_follow: close src fd if the dst open fails (no leak), and
route the rebuild restore through it so backup + restore share one
no-follow/regular-file path
- update repair tests to assert the unified hardened copy instead of the
removed shutil.copy2 calls; backup paths are now timestamped
- update normalize large-file test to stub fstat (size is checked on the
open fd, not via a pre-open os.path.getsize)
* test(wal): cover crash-safety, idempotent setup, and redaction edge paths
The write-ahead log gained its own module in v3.5.0 but sat at 82% coverage;
the uncovered lines were exactly the failure/guard branches that uphold its
contracts: the cache-hit early return, the restricted-FS chmod/mkdir swallow
paths, and the promise that a WAL write failure is logged and never crashes
the calling tool. Add five tests covering those branches plus the non-string
redaction marker, bringing mempalace/wal.py to 100% and locking the
crash-safety guarantees against regression. Test-only; no production change.
* fix: spawn daemon with CREATE_NO_WINDOW to match hook miner (#1783) (#1857)
daemon.py:_detached_kwargs was the last production spawn site still using DETACHED_PROCESS. Swap it to CREATE_NO_WINDOW, matching the hook miner's _detached_popen_kwargs fixed in #1848 — the dedicated follow-up the review bot asked for. `grep -rn DETACHED_PROCESS mempalace/` now returns zero production hits.
Survivability is unchanged: CREATE_BREAKAWAY_FROM_JOB (escapes the parent Job Object's kill-on-close) plus the daemon never being attached to the launching console carry survive-terminal-close; CREATE_NEW_PROCESS_GROUP (also kept) isolates Ctrl-C/Break. CREATE_NO_WINDOW is ignored when OR'd with DETACHED_PROCESS, so this replaces the flag rather than adding it. The daemon already redirects stdout/stderr to daemon.log and reads no stdin, so it needs no console.
Adds the first tests for _detached_kwargs (posix + windows, cross-platform monkeypatch of the Windows-only flag constants, mirroring the #1848 hooks_cli tests).
* fix(cli): add repair rebuild-index alias (#1670)
* Fix/wing slug special chars (#1852)
* fix: sanitize wing slug for project dirs with special characters
Project folders containing characters outside sanitize_name's set
(e.g. a leading '+') leaked into the derived wing name, producing names
like 'wing_+project' that config.sanitize_name rejects, silently
breaking diary auto-save for that project.
Add _safe_wing_slug(): collapse non-word runs to '_', trim, and fall
back to 'sessions' when a name reduces to nothing. Route the three
wing-derivation sites through it.
Tests: unit cases for the helper plus a hypothesis property test
asserting wing_<slug> always passes sanitize_name for any input.
* fix: preserve dots and apostrophes in wing slug for backward compatibility
The first pass collapsed every non-word character (including dot and
apostrophe) to underscore, renaming existing valid wings — e.g. my.app
became wing_my_app — which would orphan diary entries already filed under
the old name. Keep dot and apostrophe (both accepted by sanitize_name),
collapse consecutive dots to avoid the path-traversal rejection, and trim
edge separators.
Add backward-compatibility tests for previously-valid names plus a
double-dot collapse test.
* fix: cap wing slug length to stay within sanitize_name's limit
sanitize_name rejects names over 128 characters, so a very long project
directory name would produce a wing name that fails validation,
re-triggering the silent auto-save break this PR fixes. Truncate the slug
to 120 chars (the wing_ prefix keeps the total under 128).
Widen the hypothesis property test to max_size=300 so it exercises the
length path, and add an explicit truncation test.
Addresses gemini-code-assist review feedback on PR #1852.
---------
Co-authored-by: Ivan Antsimonau <ivan.antsimonau@katim.com>
* fix(hooks): hide conhost window on Windows in _mine_sync non-daemon path (#1863)
The non-daemon synchronous mine fallback in _mine_sync() spawned the
mine subprocess without CREATE_NO_WINDOW, flashing a visible console
window on every PreCompact fire on Windows. The async paths
(_spawn_mine, _desktop_toast) already pass it via _detached_popen_kwargs();
this sync path was missed. getattr(..., 0) is a no-op off-Windows.
Fixes #1862
Co-authored-by: David Finkelstein <david@finkelstein.us>
* fix: expand tilde in palace_path when read from config file\n\nMempalaceConfig.palace_path correctly called os.path.expanduser() for\nenv-var paths but not for paths read from config.json. If config.json\nstores palace_path as '~/.mempalace/palace' (the default written by\ninit), the tilde was returned unexpanded…
jphein
added a commit
to techempower-org/mempalace
that referenced
this pull request
Jul 2, 2026
…s) (#369) * feat(miner): add C# and .NET file extensions to READABLE_EXTENSIONS Adds .cs, .csproj, .sln, .razor, and .cshtml so C#/.NET projects are indexed by the project miner. .razor/.cshtml are analogous to the already-supported .jsx/.tsx. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(miner): add support for Swift and Kotlin file extensions - Updated READABLE_EXTENSIONS in miner.py to include ".swift", ".kt", and ".kts". - Added tests in test_miner.py to ensure scanning includes Swift and Kotlin files. * feat: add Pi agent JSONL session normalizer Add _try_pi_jsonl parser for Pi agent session files stored at ~/.config/pi/agent/sessions/{encoded-cwd}/{timestamp}_{uuid}.jsonl. Uses type "message" entries with role "user"/"assistant". Skips toolResult messages, model_change, thinking_level_change, and other operational events. Requires session header (type "session" with "version" key) to avoid false positives. Format documented at github.com/badlogic/pi-mono session.md and verified via Context7. Sample data provided by tunnckoCore in #59. Refs: #59 * feat: add Gemini CLI / AI Studio JSON session import support Adds _try_gemini_json parser to normalize.py for three layouts: 1. Gemini API contents format (~/.gemini/sessions/*.json): {"contents": [{"role": "user", "parts": [{"text": "..."}]}, ...]} 2. Messages-wrapper variant: {"messages": [{"role": "user", ...}, {"role": "model", ...}]} 3. Flat top-level list with role="model". This complements the existing _try_gemini_jsonl parser (which handles ~/.gemini/tmp/<hash>/chats/session-*.jsonl with session_metadata sentinel) — JSONL covers Gemini CLI runtime sessions, JSON covers exported / Studio-saved transcripts. ## Review feedback addressed (PR #204) bgauryy review: - #1 Parser-precedence bug: _try_gemini_json runs *before* _try_claude_ai_json so the {"messages":[..., role=model, ...]} layout is no longer silently claimed by the Claude parser. The Gemini parser's has_model_role guard prevents false-positives against Claude / ChatGPT data. - #2 Layout 2a coverage: TestGeminiJson.test_messages_wrapper_format + test_messages_wrapper_does_not_get_claimed_by_claude pin the fix in place. - #3 Test conflicts with current main: rebased onto develop; tests restructured into TestGeminiJson class. - #4 tempfile/os.unlink → pytest tmp_path everywhere. - #5 elif not text → else (the elif branch was dead). - #6 Module docstring updated to mention Google AI Studio. Tests: 9 new cases in TestGeminiJson covering all three layouts, multi-part text joining, non-text part skipping, has_model_role disambiguation, dispatch-chain regression for review #1. * feat(normalize): add Continue.dev session parser Add _try_continue_json() normalizer for Continue.dev AI assistant sessions (~/.continue/sessions/*.json). Parses history array with role/content pairs, handles tool calls, system messages, and metadata. Closes #59 (partial — adds Continue.dev format support) Includes comprehensive test coverage for valid sessions, edge cases, malformed input, and unicode content. * fix: preserve collection name on MCP search retry * feat: add Cursor IDE support (hooks, plugin, skill, docs, tests) Adds first-class Cursor IDE integration alongside the existing Claude Code and Codex hook flows, so Cursor users get the same automatic diary saves, pre-compaction transcript capture, and session-start memory recall — without changing any default behaviour for existing users. What's included --------------- Cursor hook scripts (hooks/cursor/): - mempal_save_hook_cursor.sh — Stop event, counter + loop_count guard, pending-save marker consumption, background mempalace mine, followup_message emission. - mempal_precompact_hook_cursor.sh — synchronous mine before compaction, drops a pending_save marker, returns user_message. - mempal_wake_hook_cursor.sh — sessionStart event, wing-scoped recall guidance via additional_context. - lib/common.sh — shared parsing + state helpers (bash 3.2 safe, no heredoc-in-subshell traps). - install.sh — idempotent installer with --scope, --variant, --dry-run, --uninstall. Recognises existing entries by basename so re-installs across paths work. - STDIN_SHAPE.md, README.md — payload schemas + quick reference. Cursor plugin (.cursor-plugin/ + repo-root components): - plugin.json, marketplace.json, README.md. - skills/mempalace/SKILL.md — model-invocable skill mirroring the Claude plugin's skill surface. - commands/mempalace-{help,init,mine,search,status}.md — slash commands for marketplace-published installs (filename = slug). - mcp.json — auto-registers the mempalace MCP server, wrapped under the documented mcpServers key. Examples + docs: - examples/cursor/hooks.json, hooks.minimal.json + README. - website/guide/cursor-hooks.md + sidebar entry. - README.md and CHANGELOG.md updates. Tests (129 new, all green): - tests/test_cursor_hooks_shell.py — 75 behavioural tests for the three hook scripts: kill switches, input parsing, counter logic, loop prevention, pending markers, wing inference, logging. - tests/test_cursor_hooks_install.py — 19 contract tests for the installer: dry-run, idempotent merge, basename-matched uninstall, refusal to overwrite malformed JSON. - tests/test_cursor_plugin_manifest.py — 35 contract tests for the plugin: manifest validity, version sync with mempalace.version, mcp.json shape, skill/command frontmatter, default-discovery layout invariants. Design notes ------------ - Local-first and zero-API by default; hooks never call external services. Same privacy model as the existing Claude Code hooks. - Fail-open: hook scripts deliberately do not use set -e so a broken hook can never block the user's conversation. - Cursor preCompact cannot block + return a followup, so we synchronously mine the transcript and drop a pending_save marker that the next stop hook consumes — guarantees verbatim capture before context window compression. - Cursor's default plugin discovery requires real commands/, skills/, and mcp.json at the plugin root (verified against the cached cloudflare plugin); .cursor-plugin/{commands,skills} are convenience symlinks back to those canonical locations. - bash 3.2 compatibility throughout: avoids heredoc-in-command- substitution parser bugs; uses python -c for JSON parsing; basename-matched entry recognition in install.sh. - All changes are additive. No existing files are removed, no existing hooks change behaviour, and no new runtime dependencies are introduced. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cursor): address gemini-code-assist review on PR #1632 Five fixes from the Gemini Code Assist review on https://github.com/MemPalace/mempalace/pull/1632 — three real bugs, two cleanups, all consistent with the bash-3.2-compatibility contract documented in the original commit. Bug fixes (high) ---------------- 1. hooks/cursor/lib/common.sh — config.json kill-switch check used a `python3 - <<'PYEOF' ... PYEOF` heredoc inside a `$(...)` command substitution. The heredoc body contains parens which trips the macOS bash 3.2.57 parser bug. Replaced with a `python -c '...'` call passing the config path as argv[1]. Matches the pattern already used in mempal_parse_stdin in the same file. 2. hooks/cursor/install.sh — a relative `--install-dir` was written verbatim into hooks.json. Cursor invokes hook commands from its own working directory (typically the project root), so a relative command path would silently fail to launch the hook. Now resolved to an absolute path against `$PWD` before being baked in. 3. hooks/cursor/mempal_save_hook_cursor.sh — `MEMPAL_SAVE_INTERVAL=0` would crash bash on `$((NEXT % 0))` (division by zero). Extended the existing sanitiser case to coerce 0 to the default interval alongside empty / non-numeric values. Cleanups (medium) ----------------- 4. hooks/cursor/install.sh — the EMPTY_CHECK_PY temp file is now inlined as `python -c '...'`. Removes a small leak window (tmpfile would linger if the script were interrupted between mktemp and rm -f) and shortens the script. 5. hooks/cursor/install.sh — `mktemp -t prefix` has subtly different semantics on BSD (macOS) vs GNU mktemp. Switched to the portable absolute-template form `mktemp "${TMPDIR:-/tmp}/...XXXXXX"` which behaves identically on both. Regression tests ---------------- - tests/test_cursor_hooks_shell.py test_save_interval_zero_is_coerced_to_default — guards fix #3. - tests/test_cursor_hooks_install.py — new TestInstallDirAbsolutePath class: test_relative_install_dir_is_absolutized_in_hooks_json — guards fix #2 against regression. test_absolute_install_dir_is_preserved_verbatim — guards that the relative-to-absolute resolution does not mangle paths that were already absolute. Verification ------------ - bash -n on all three edited scripts: clean. - uv run pytest tests/test_cursor_hooks_*.py tests/test_cursor_plugin_manifest.py: 132 passed (was 129; +3 regression tests). - uv run pytest tests/ --ignore=tests/benchmarks: 2399 passed, 3 skipped (pre-existing). - uv run ruff check . / ruff format --check .: clean. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cursor): address igorls review on PR #1632 Resolves the maintainer review on the Cursor IDE support PR. Cursor-only scope; cross-IDE items (wing-naming convention, shared-file merge order) are coordinated on the separate Antigravity branch. followup_message default (the one "decide before merge" item): - Keep the stop-hook followup ON by default. Cursor's transcript format is undocumented and mempalace/normalize.py has no Cursor parser, so the background `mempalace mine --mode convos` is best-effort only and does not yet yield clean verbatim drawers. The followup is therefore the load-bearing verbatim-capture path; defaulting it off would leave a default Cursor install capturing nothing. - Add an opt-out (MEMPAL_CURSOR_SILENT=1, or MEMPAL_VERBOSE=false) for users who want the Claude-style "zero tokens in chat" behaviour. The hook still mines and keeps its counters/markers when silenced. - Correct the misleading "background mine captures it" comments in the save and precompact hooks; update hooks/cursor/README.md and the guide. Hygiene fixes: - Drop the hardcoded "version" field from .cursor-plugin/plugin.json and marketplace.json (mempalace/version.py is the single source of truth); tests now assert the field stays absent. - Remove the committed .cursor-plugin/{commands,skills} symlinks (they break on Windows clones with core.symlinks=false and were redundant with the real repo-root components that `source: "."` already serves); add a guard test that no symlinks exist under .cursor-plugin/. - Document the preCompact synchronous-mine timeout tradeoff and that an incremental/append-only mine is recoverable if killed (no corruption). - Add a Cursor-namespaced, daily-throttled TTL sweep (MEMPAL_STATE_TTL_DAYS, default 30) to lib/common.sh that GCs stale cursor_*.count/.pending only, after the kill-switch check; shared logs and antigravity_* are untouched. Verification: full suite green (2424 passed, 3 skipped), ruff check + format clean, bash -n clean on all cursor scripts. +30 Cursor tests (followup opt-out, state GC, TTL validation, no-symlink/version guards). Co-authored-by: Cursor <cursoragent@cursor.com> * style(tests): apply ruff 0.4.x format to test_normalize Fixes lint CI: ruff format --check flagged blank-line and long-dict wrapping in the Continue.dev parser tests. * fix(searcher): scope neighbor expansion by parent_drawer_id (#1580) * fix(mcp): drop top-level anyOf from diary_write schema The mempalace_diary_write tool declared a top-level anyOf in its input schema to require either entry or content. Anthropic's Messages API rejects any tool schema with a top-level anyOf/oneOf/allOf and returns a 400 for the entire tools array, so every MCP session failed to start. The entry/content constraint is already enforced at dispatch: content is remapped to entry before the handler runs, and a missing value returns -32602. Removing the combinator restores compatibility without weakening validation. Closes #1711 * docs(openclaw): catch up SKILL.md with 8 newer MCP tools The openclaw skill was last updated when mempalace exposed 19 MCP tools. Since then 13 more agent-facing tools have landed; this PR documents the 8 that openclaw should expose so agents can call them natively instead of falling back to `npx mcporter call ...`: Search & Browse: - mempalace_list_drawers (paginated drawer listing) - mempalace_get_drawer (fetch a single drawer by id) Palace Graph: - mempalace_create_tunnel (explicit cross-wing link) - mempalace_list_tunnels (enumerate explicit tunnels) - mempalace_delete_tunnel (remove an explicit tunnel) - mempalace_follow_tunnels (walk explicit tunnels from a room) Write / Session: - mempalace_update_drawer (mutate content or relocate a drawer) - mempalace_memories_filed_away (ack the silent auto-save hook) The 3 admin-only tools (mempalace_sync, mempalace_hook_settings, mempalace_reconnect) are intentionally left out — they're host/admin operations, not agent-facing memory operations. The Hermes MemoryProvider plugin landing in MemPalace/mempalace#1684 makes the same call. Version bumped 3.3.0 -> 3.4.0 (additive tool surface, no breaking changes to existing tool docs). * docs(openclaw): address review round 1 - Fix mempalace_find_tunnels params: (required) -> optional. The MCP handler defaults both wing_a and wing_b to None (mempalace/mcp_server.py:1277), so the prior docs were factually wrong. Caught by gemini-code-assist on PR #1719. - Clarify implicit-vs-explicit tunnel distinction with consistent casing and a brief in-line definition (implicit = discovered from drawer content overlap; explicit = user/agent-declared link). Suggested by copilot-pull-request-reviewer. - Split the mempalace_memories_filed_away one-liner into a short description plus 'Returns' and 'When to call' sub-bullets for readability. Suggested by copilot-pull-request-reviewer. * fix: detect Java project manifests * fix: handle rootless Java subprojects * fix(mcp): fail closed when add_drawer idempotency pre-check fails * feat: add mempalace-recall skill and optional Cursor recall rule Ports the OpenClaw "search before answering" protocol to the Cursor and Claude plugin surfaces so the agent reads the palace before answering about past work, people, projects, or prior decisions instead of guessing from model memory. - integrations/shared/recall-protocol.md: single source of truth for the recall protocol, referenced by the skill and the rule so they cannot drift. - skills/mempalace-recall/SKILL.md: recall-only skill (the mempalace skill keeps setup/mine/status); cross-linked from the ops skill. - rules/mempalace-recall.mdc: plugin recall rule, alwaysApply: false so it only fires on recall-relevant turns and never adds MCP latency to greenfield work. - examples/cursor/rules/: opt-in copies for non-plugin users, including an aggressive alwaysApply: true variant documented with its latency tradeoff. - .claude-plugin/skills/mempalace-recall/SKILL.md: Claude plugin parity. - tests: assert the recall skill and rules/ discovery layout; the shipped rule must be alwaysApply: false. - docs: .cursor-plugin/README.md and the cursor-hooks guide now describe the three layers of recall (hook + skill + rule). The Antigravity plugin mirror lands as a follow-up on the antigravity branch, where .antigravity-plugin/ exists. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(repair): run post-rebuild FTS5 cleanup on legacy cmd_repair path (#1747) A clean `mempalace repair --yes` (legacy path) finished without _vacuum_and_rebuild_fts5: the bulk delete_collection + re-upsert cycle leaves the FTS5 inverted index inconsistent, so the next repair aborts at the sqlite integrity preflight. rebuild_index() got this cleanup when #1517 was fixed; cmd_repair never did. Extract the shared epilogue _post_rebuild_cleanup() (close chroma handles, then VACUUM + rebuild FTS5) and call it from both full-rebuild paths so they cannot drift apart again. Cleanup runs on the legacy success path only; failure/restore paths are unchanged. Closes #1747 Co-Authored-By: nord- <3777600+nord-@users.noreply.github.com> * test(backends): live-substrate conformance module for pgvector Mirrors the portable fake-client arms of test_pgvector_backend.py against a real PostgreSQL+pgvector server and adds live-only arms the in-memory fake cannot exercise: real <=> operator ground truth, JSONB pushdown vs local-fallback equivalence, cross-namespace isolation on real tables, 8-connection concurrent writers, and the advisory-lock serialization of run_maintenance('reindex') under a 2-connection race. Gated on MEMPALACE_PGVECTOR_LIVE_DSN (same pattern as the qdrant live gate); skips cleanly when unset. First run: 15/15 pass on PostgreSQL 16.10 + pgvector 0.8.2 (+AGE 1.6.0 in the same server), psycopg 3.3.4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review(gemini): marker-race stub, created-list lock, exact-order + exactly-one-ran asserts - Stub _write_marker on the 8 concurrent writer backends: upsert() rewrites the marker on every call with a plain open('w'), so backends sharing one local_path race on the same file (sharing violations on Windows) — a test-design artifact, not the contract under test - Guard the fixture's created list with a lock for the threaded tests - Assert exact distance-ordered ids in the query/filter arms - Reindex race: exactly one 'ran' (index absent beforehand, so the advisory-lock winner must build) Re-run live after changes: 15/15 pass (PG 16.10, pgvector 0.8.2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(embedding): chunk EmbeddinggemmaONNX batches to bound ONNX memory (#1770) One session.run over a repair-scale batch (5000 docs) allocates attention buffers far beyond available RAM and the kernel kills the process. Mirror chromadb's ONNXMiniLM_L6_V2 and embed in sub-batches of 32; per-chunk padding also stops one long doc inflating the whole batch. Co-Authored-By: mojie5 <262519016+mojie5@users.noreply.github.com> * fix(embedding): lock EF cache and lazy load, guard inputs (#1770) Two threads sharing a cold EmbeddinggemmaONNX via _EF_CACHE could each build a full model session, and two factory callers could each keep a private instance. The load is now double-check locked with the session published last, and the factory cache has an atomic check-then-construct behind a lock-free fast path. __call__ wraps a bare string, returns [] for None and empty input before the lazy download, and its annotation matches the accepted types. * fix(hallways): scope hallway-file path to MempalaceConfig.palace_path (#1778) Pre-3.4 the hallway store was hardcoded at ~/.mempalace/hallways.json regardless of the configured palace_path, so two palaces on one host silently shared one file. Mining into palace-A leaked records into palace-B's hallway code paths. Apply the 3.3.6 tunnel-file migration pattern: * MempalaceConfig.hallway_file resolves to <dirname(palace_path)>/hallways.json * hallways._get_hallway_file(config) reads through MempalaceConfig * hallways._legacy_hallway_file() exposes the pre-migration hardcoded path for one-time orphan detection; _load_hallways logs a one-line warning when the legacy file exists but the configured one doesn't, matching palace_graph._load_tunnels behavior. No auto-migration — silent merging risks clobbering newer data. Atomic-write + 0600 semantics unchanged. Module-level _HALLWAY_FILE constant kept and honored when monkey-patched directly, so the three existing test sites that patch it (test_hallways.py, test_hallways_pagination.py, test_mcp_server.py) keep working without modification. New coverage in tests/test_hallways_palace_scoped.py mirrors the analogous tunnel tests: resolver default + custom palace_path + env-var redirect, orphaned-legacy warning + no-warning when paths match, and an end-to-end multi-palace isolation regression guard. Closes #1778 * fixup(hallways): drop _HALLWAY_FILE back-compat shim, migrate existing tests to resolver Replaces the back-compat shim in _load_hallways/_save_hallways (which honored direct monkey-patches of the _HALLWAY_FILE module constant) with a clean single-source-of-truth resolver, matching the palace_graph tunnel-file migration in 3.3.6. The three existing test sites (tests/test_hallways.py, tests/test_hallways_pagination.py, tests/test_mcp_server.py) now monkey-patch _get_hallway_file and _legacy_hallway_file directly, exactly mirroring the helper in tests/test_palace_graph_tunnels.py. Production code now has one branch through the path resolution instead of two. No behavior change. 269/269 hallway + tunnel + mcp-server tests pass on Python 3.11 and 3.12, ruff clean. * fixup(hallways): address gemini-code-assist review on PR #1780 Two catches on tests/test_hallways_palace_scoped.py TestMultiPalaceIsolation.test_save_then_load_under_different_palace_returns_empty: 1. Stale comment referencing the removed _HALLWAY_FILE back-compat shim (deleted in the prior fixup commit). Removed. 2. _legacy_hallway_file was not monkey-patched, so the test isolation gap let _load_hallways check the host's real ~/.mempalace/hallways.json when evaluating the legacy-warning branch. Now patched to a tmp_path sibling, matching the helper pattern used in test_palace_graph_tunnels. * fix(ids): use length-prefixed recipe v3 * fix(mcp): treat chunked drawers as logical drawers * fix(mcp): avoid mutating drawer metadata * fix(ids): simplify v3 length-prefixed hashing * chore(deps): bump docker/metadata-action from 5 to 6 Bumps [docker/metadata-action](https://github.com/docker/metadata-action) from 5 to 6. - [Release notes](https://github.com/docker/metadata-action/releases) - [Commits](https://github.com/docker/metadata-action/compare/v5...v6) --- updated-dependencies: - dependency-name: docker/metadata-action dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump docker/build-push-action from 6 to 7 Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6 to 7. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/v6...v7) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump docker/login-action from 3 to 4 Bumps [docker/login-action](https://github.com/docker/login-action) from 3 to 4. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](https://github.com/docker/login-action/compare/v3...v4) --- updated-dependencies: - dependency-name: docker/login-action dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * fix(hooks): normalize Windows transcript paths in shell hooks * fix(backends): serialize first connect in sqlite_exact and pgvector (#1774, #1775) Co-Authored-By: jphein <19301265+jphein@users.noreply.github.com> * fix(hooks): preserve fail-loud parse diagnostics * fix(hooks): typo regression addressed * fix(tests): run fact_checker __main__ via subprocess to clear runpy warning `tests/test_fact_checker.py` imports symbols from `mempalace.fact_checker` at module top (putting it in sys.modules), then `TestCLI.test_exits_nonzero_when_ issues_found` re-executed the same module as __main__ via `runpy.run_module("mempalace.fact_checker", run_name="__main__")`. runpy warns because it re-runs an already-imported module against a half-initialized state: RuntimeWarning: 'mempalace.fact_checker' found in sys.modules after import of package 'mempalace', but prior to execution of 'mempalace.fact_checker' Run the CLI in a fresh process via `subprocess.run([sys.executable, "-m", "mempalace.fact_checker", ...])` instead — no sys.modules collision, and it exercises the real `python -m` entry point. Assertions are preserved (SystemExit code 1 → returncode 1; captured stdout substring → result.stdout). The child's entity registry (`~/.mempalace/known_entities.json`, resolved via expanduser at import) is redirected by overriding both HOME and USERPROFILE in the subprocess env so it works on POSIX and Windows. Verified: `pytest tests/test_fact_checker.py -W error::RuntimeWarning` passes (26) with the warning promoted to error — proving it no longer fires. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): avoid Chroma open when cached DB disappears * test: stabilize release validation on develop * fix(palace): clean source mine locks safely * fix: close blob seq sqlite migration connection * fix: address mine lock review feedback * test: stabilize closet boost fixture on Windows * chore(release): 3.4.1 Bump version across all sources (version.py, pyproject.toml, both Claude plugin manifests, Codex plugin manifest, README badge, uv.lock) and promote the Unreleased changelog to 3.4.1. Shipping: Cursor IDE plugin + hooks, first-class Antigravity IDE support (with zero-config interpreter resolution), embeddinggemma bulk re-embed OOM fix, and backup-retention pruning. Also rebuilds the CHANGELOG compare-link block, which had been left at v3.2.0: adds the full 3.3.0-3.4.1 chain plus the previously undocumented 3.4.0, and points Unreleased at v3.4.1...HEAD. Every version header now resolves to a compare link. * fix(hooks): portable mtime in macOS hook throttles; doc cleanup Address review feedback surfaced on the 3.4.1 release promotion (#1810). Bug fix — `date -r FILE` is GNU-only. On BSD/macOS `date -r` expects epoch seconds, not a path, so the staleness/throttle checks in the new Cursor and Antigravity hooks silently failed on macOS: the state GC swept on every fire and the pending-save guard was skipped. Replace with a portable `os.path.getmtime` one-liner via the already-resolved $MEMPAL_PYTHON_BIN (cursor/lib, antigravity/lib, antigravity save hook). This restores the "bash 3.2.57 / macOS default" compatibility the Antigravity changelog claims. Docs: - Correct the MCP tool count to 33 (was 19/29/31 in 21 places across plugin manifests, READMEs, and website docs — all drifted from the TOOLS dict / mcp-tools.md reference, which both have 33). - Fix broken CHANGELOG link to the Cursor skill (skills/, not .cursor-plugin/skills/). - Fix one-too-many `../` in skills/mempalace/SKILL.md's cursor-hooks link (resolved above the repo root). - Add the required `mcpServers` wrapper to the mcp.json example in .cursor-plugin/README.md so copy-paste yields a valid Cursor config. Left intentionally unchanged: the os.dup2 fd-1 redirect in mcp_server.py is deliberate (#225 keeps JSON-RPC off fd 1). * style(hooks): single-quote the static python -c mtime snippet The snippet has no shell interpolation — the path arrives via argv, not string interpolation — so single quotes are correct and make it unambiguous that nothing is shell-expanded. Behavior is identical: `sys.argv[1]` contains no `$`, so it was never expanded (verified empirically). Matches the single-quoted `python -c` blocks already in hooks/cursor/lib/common.sh. No functional change. * test(migrate): cover swap-failure rollback Adds end-to-end regression coverage for the migration swap path where os.replace hits EXDEV, the shutil.move fallback fails, and the original palace must be restored from the rename-aside copy. * feat(mcp): add mempalace_delete_by_source bulk-cleanup tool (#1722) Adds an MCP tool to remove every drawer mined from a given source_file exact match, for cleaning up benchmark/test data accidentally mined into a user wing (ShareGPT dumps, results_mempal_*.jsonl, language config JSON) that drowns out real memories in semantic search. Matching is pushed to the backend via delete(where={"source_file": ...}) the same idiom the miner and diary-ingest paths already use so it is not subject to the SQLite variable limit regardless of how many drawers share the source. Defaults to a dry run reporting match count and a sample; dry_run=false commits. Absent source is an idempotent no-op, not an error. * fix(mcp): harden delete_by_source per review — strip surrogates + type guard Address Gemini review on #1729: - normalize source_file with strip_lone_surrogates so exact matching hits rows mined from non-ASCII paths via cp1252 stdin (#1488), mirroring tool_add_drawer's ingestion-side normalization - isinstance(str) guard so a non-string source_file returns a clean error instead of AttributeError - default missing wing/room to "" in the dry-run sample, consistent with the rest of the file - add tests: non-string rejection + surrogate-normalization match * feat(miner): add PHP ecosystem file extensions * fix(claude-plugin): run final mine on SessionEnd * fix reviewer feedback: To prevent a KeyError and provide a clear, actionable assertion failure message if PreCompact is ever missing * fix(chroma): route stale hnsw divergence to sqlite fallback * fix reviewer feedback for chroma and tests * fix(mcp): refuse second writer for same palace * fix(mcp): cache writer lock setup failures * feat: add opt-in local daemon for queued MemPalace writes - New mempalace/daemon.py: long-lived localhost HTTP server (127.0.0.1) with a SQLite WAL job queue, single worker thread, bearer-token auth, and owner-only file perms (0600/0700) on queue DB, token, endpoint, and log. - New mempalace/service.py: transport-neutral job execution surface shared by the daemon, with per-job env isolation so one job's backend/palace switch cannot leak into the next. mcp_tool is allowlisted to write-classified tools only. - Crash recovery re-queues jobs left 'running' by a killed daemon; jobs that already exhausted MAX_ATTEMPTS are dead-lettered to 'failed' instead of being retried (non-idempotent diary_write would otherwise duplicate verbatim content on every restart). - Bounded retention prunes terminal jobs older than 7 days (MEMPALACE_DAEMON_RETENTION_DAYS); queued/running jobs are never touched so a crash mid-prune cannot drop in-flight work. - CLI: --daemon/--background on mine/sync submit to the queue; new `mempalace daemon {start,stop,status,jobs,wait}` subcommand. Strictly opt-in: no flag, env, or config means no daemon and no behavior change. - Hooks opt in via MEMPALACE_HOOKS_DAEMON or config hooks.daemon; when the daemon is not already running, hooks fall back to the existing direct/spawn path so the 500ms hook budget is preserved (hooks never auto-start the daemon). - service.run_sync renders the same operator-facing report shape as the direct CLI sync path (no_source, out_of_scope, by_source, Re-run/Removed hints) and drops the old KeyError-prone 'deleted' read. * fix(mcp): gate startup on sqlite integrity failures * chore(deps-dev): bump ruff from 0.15.15 to 0.15.18 Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.15 to 0.15.18. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/ruff/compare/0.15.15...0.15.18) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.15.18 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * fix(mcp): applied 3 of the 4 reviewer suggestions * fix(mcp): guard remaining None palace_path in _mcp_sqlite_integrity_refusal. Added one regression test calling the function directly with palace_path=None. * fix: unblock daemon PR CI + address review comments CI was red on all three platforms for the daemon-mode draft PR. Root causes and fixes: - Linux 3.9 collection error: `_submit_daemon_job`'s `dedupe_key: str | None` parameter annotation is evaluated at def time, and hooks_cli.py has no `from __future__ import annotations` — `str | None` raises TypeError on 3.9. Reverted to `dedupe_key: str = None` (the original, 3.9-safe). The other `int | None` in the file is a function-local annotation, which is never evaluated, so it was never the problem. - macOS/Windows daemon lifecycle flakes: the 3 HTTP-lifecycle tests failed at the 10s readiness deadline on contended CI runners (localhost bind is sub-second locally but took ~5s when it passed on the macOS fleet, >10s when it didn't), and because the server thread never shuts down on timeout, run_server's `os.environ["MEMPALACE_PALACE_PATH"]` + `os.umask(0o077)` mutations leaked into the rest of the suite — poisoning every later test that reads MempalaceConfig().palace_path (the 60+ test_mcp_server cascade on macOS; the at-exit socket hang → SIGINT on Windows). Bumped the readiness deadline to 30s and added a module-scoped snapshot + autouse fixture in test_daemon.py that force-restores the env + umask to the pre-suite baseline after every daemon test, so a leaked server thread can't poison other test files. Gemini review comments (fixed in code, no thread replies per convention): - daemon.py `_connect()` was a bare `sqlite3.connect` whose `with`-block only managed the transaction, not the connection — an unbounded FD leak in a long-lived daemon running thousands of jobs (also the source of the Windows "unclosed database" ResourceWarning noise). Converted to a closing @contextlib.contextmanager. - `QueueStore.finish()` gained `only_if_running`; `_safe_finish` passes it so a late worker finish can't overwrite a shutdown-cancelled job back to succeeded/failed — removes the reliance on process-exit timing. - `DaemonClient.request` wraps the final `json.loads` in try/except JSONDecodeError → DaemonError, so a non-JSON 2xx response surfaces as a structured error instead of a bare JSONDecodeError. - test_sync.py: removed the module-level `import mempalace.mcp_server` and moved the stdout-rebinding side effect into an autouse fixture scoped to TestServiceRunSyncReport, so the embedder/Chroma import chain is no longer forced at collection time for the existing sync tests. Coverage: added focused happy-path tests for service.run_sync early-returns, run_mine backend application + invalid mode, execute_job kind dispatch, run_diary_write arg forwarding, run_mcp_tool write-tool dispatch, and print_job_result — lifts service.py from 57% to 85% so the new files (service 85%, daemon 80%) don't drag the total below the 80% CI gate now that the daemon tests complete and the gate is actually evaluated. * fix: daemon client bypasses proxy discovery; tests force-shutdown server thread DaemonClient.request now uses a no-proxy opener (build_opener(ProxyHandler({}))) instead of urllib.urlopen. The daemon is always on 127.0.0.1, so a request must never go through an HTTP proxy — this is the correct production choice. It also bypasses urllib's proxy discovery (macOS _scproxy via SystemConfiguration), which runs on the first request to any host and is NOT bounded by the per-request timeout: on a CI runner with no network it hangs for tens of seconds, which looked exactly like the daemon never came up (test_daemon_http_lifecycle_executes_job timed out at 30.18s). With the no-proxy opener the lifecycle runs in 0.78s and no server thread is leaked — which also removes the timing skew that made the sqlite_exact concurrent-connection test flake on macOS CI. The leaked server thread was also the Windows exit-hang root cause: a slow/failed client.shutdown() POST left serve_forever running, and the interpreter blocked on the open listening socket at process exit. Tests now capture the httpd run_server creates (by subclassing daemon.ThreadingHTTPServer) and force httpd.shutdown() + server_close() from the test thread if the normal shutdown path leaves the thread alive, asserting the thread died so a leak becomes a visible failure instead of a silent exit hang. * test: win32-only diagnostic for daemon process-exit hang The Windows CI run passes all 666 tests then hangs at interpreter shutdown (KeyboardInterrupt at socket.py:723) until the runner kills it. All daemon lifecycle tests assert their server threads died, so the hang is a different non-daemon thread blocked on a socket — not the daemon server thread. CI round-trips can't show which thread it is. Add a win32-only session fixture that: - arms faulthandler.dump_traceback_later(130s) to print every thread's stack to stderr once the hang has run a while, and - prints every live thread (name + daemon flag) at session teardown — a non-daemon thread present there is the shutdown blocker. Gated to sys.platform == 'win32' so Linux/macOS CI see no extra output. Remove once the Windows hang is fixed. * fix(daemon): Windows-safe pid liveness probe; finalize cross-platform daemon tests _pid_alive used os.kill(pid, 0) as an existence check. On Windows signal 0 is signal.CTRL_C_EVENT, so Python routes it to GenerateConsoleCtrlEvent and sends a console Ctrl-C to the target's process group rather than probing the pid. DaemonClient polls a same-process endpoint during startup, so on a CI runner with an attached console that Ctrl-C was delivered back to the interpreter as a spurious KeyboardInterrupt — the Windows CI hang that interrupted the suite at the first daemon HTTP-lifecycle test (socket.py recv). Probe via the Win32 OpenProcess/WaitForSingleObject handle API instead, which has no signalling side effects. This is also a real Windows production bug, not just a test artifact. Tests: - Skip the two owner-only (0600) permission tests on Windows: os.chmod cannot represent POSIX mode bits there (files report 0o666); the daemon relies on user-profile ACLs on Windows. - _start_server now captures and re-surfaces a run_server thread crash instead of spinning for 30s and failing with a bare assert (diagnoses the macOS startup flake). - Add a regression test asserting _pid_alive is correct and emits no console control event when hammered like the poll loop. - Remove the temporary win32 exit-hang diagnostic fixture from conftest now that the root cause is fixed. * fix(daemon): skip reverse-DNS in server_bind so startup can't block ~30s HTTPServer.server_bind() resolves server_name via socket.getfqdn(host). For the daemon's 127.0.0.1 bind that lookup is pointless, and on a host with slow or absent reverse DNS it blocks startup until the resolver times out (~30s) — which looks exactly like the daemon never coming up. This is why the first daemon HTTP-lifecycle test timed out on the macOS CI runner (httpd_bound=False after 30s) while every later one bound in seconds once the OS had cached the negative lookup. Bind via TCPServer directly and set server_name from the literal host. * feat(hooks): add a budget-safe SessionEnd save hook for clean exits (#1341) Short sessions that exit cleanly below SAVE_INTERVAL and without a PreCompact were never saved. Add a SessionEnd hook that takes one final flush. Claude Code budgets SessionEnd hooks at 1.5s and a plugin-provided timeout cannot raise it, and a cold mempalace start exceeds that, so the wrapper backgrounds the work and returns immediately; the detached child completes the transcript ingest, project mine, and diary checkpoint after the session exits. The handler validates transcript_path through _validate_transcript_path before any ingest or diary write, so a traversal or wrong-suffix path is rejected while the independent project mine still runs. Adds hook_session_end, both shell wrappers, the plugin hooks.json entry, the session-end CLI choice, and focused tests. (cherry picked from commit 10e1450e04fc7cec72984ab7442d3b4fca1490e8) * fix(claude-plugin): resolve SessionEnd merge semantics * fix(claude-plugin):reviewer feedback for _validate_transcript_path function calls Path.resolve(), which can raise an OSError * fix(daemon): address post-merge review feedback on #1826 Five fixes from the Copilot review of the merged daemon PR: 1. Privacy: the queue DB's SQLite WAL/SHM sidecars hold un-checkpointed verbatim payloads but were created with the caller's umask. Set the owner-only umask in run_server BEFORE DaemonRuntime builds the QueueStore (not only once the HTTP server starts), and harden any existing sidecars in QueueStore._init_db as defense-in-depth. 2. DoS guard: reject a negative Content-Length in the request reader. rfile.read(-1) would block until the client disconnects and bypass the MAX_BODY_BYTES cap. 3. Side effects: extract _wal_log (+ _ensure_wal, _WAL_FILE, _WAL_REDACT_KEYS) into a new side-effect-free mempalace/wal.py. The CLI sync path and the daemon service layer obtained _wal_log via `from .mcp_server import _wal_log`, which runs mcp_server's import-time stdio protection (os.dup2(2, 1); sys.stdout = sys.stderr) in a non-MCP process and misroutes operator output. mcp_server/cli/service now import from mempalace.wal. 4. Correctness: run_mcp_tool treated any dict as success. Write tools that return a bare {"error": ...} (e.g. tool_create_tunnel/tool_delete_tunnel validation) were recorded as succeeded; now the "error" key infers failure. 5. Hook budget: get_client_if_running()/health() take an explicit timeout, and the hook "is the daemon up?" precheck uses a short HOOK_PROBE_TIMEOUT (0.5s) so a wedged daemon can't stall the hook for the default 5s. Adds tests/test_wal.py (import isolation + redaction) and daemon tests for the umask ordering, negative Content-Length, run_mcp_tool error inference, and the short probe timeout. * fix(backends): single-scroll bulk metadata fetch for Qdrant; bump scroll page size (#1796) * fix: apply suggested reviewer suggestions * updated tests/test_qdrant_bulk_metadata_scroll.py because of CI failure after push the 2nd commit * perf(embedding): cap ORT intra-op threads so a background mine doesn't pin every core (#1068) ChromaDB's ONNX embedder builds its InferenceSession without a thread cap, so ORT's intra-op pool defaults to the physical core count. OMP_NUM_THREADS is inert against it (ORT owns its own pool), so a background `mempalace mine` pins 4-5 cores and stacked Stop-hook fires turn the machine into a thermal event. Add an `embedding_threads` config knob (env MEMPALACE_EMBEDDING_THREADS or config.json). Unset/"auto" caps the intra-op pool at half the logical CPUs so a fresh install stays usable out of the box; a positive integer sets an exact count; 0/negative leaves ORT uncapped for users who want max throughput. The cap is applied via SessionOptions at session construction: - `_MempalaceONNX` (default minilm) overrides the `model` cached_property to rebuild the session the same way upstream does plus the cap, falling back to upstream's uncapped build if chromadb internals shift. - `EmbeddinggemmaONNX` builds its session through the shared `_intra_op_session_options()` helper. * perf(mcp): answer overview tools from the sqlite aggregate to fix large-palace timeouts (#1748, #1379) tool_status / list_wings / list_rooms / get_taxonomy paged the entire collection metadata through the chroma client (`_fetch_all_metadata`, a 1000-row offset loop), which cold-loads the HNSW index and materializes hundreds of MB of dicts. On six-figure palaces these exceed the MCP host tool-call limit (180k drawers ~3-4 min; 349k times out at 120-240s). The 5s metadata cache only dedups repeat calls — it does not stop the cold-call timeout. A correct single-query SQL cross-tab already exists (`backends.chroma._sqlite_wing_room_counts`) and is already the CLI default (`miner.status`), but the MCP tools never used it — and the MCP-side sqlite reader only ran behind the `vector_disabled` recovery path. Add `_sqlite_taxonomy()` (guards on `_is_chroma_backend()`, returns None to fall back) and wire it as the default path into all four overview tools. They now answer from one GROUP BY without touching HNSW. Non-chroma backends (qdrant, sqlite_exact) and unbootstrapped/legacy layouts fall back to the existing client path unchanged. graph_stats (also named in #1379) builds an in-memory graph via build_graph() and needs its own treatment — tracked separately. * fix(pgvector): strip NUL bytes so a transcript NUL no longer aborts the mine (#1829) PostgreSQL cannot store NUL (0x00) in text or jsonb. On the pgvector write path a NUL in `document` is rejected by psycopg ("PostgreSQL text fields cannot contain NUL (0x00) bytes") and a NUL in `metadata` becomes a JSON unicode escape the jsonb cast rejects ("unsupported Unicode escape sequence"). `_execute` re-wraps either as BackendError and `_mine_impl` re-raises, so the whole mine exits non-zero and every file after the offending one is left unmined. ChromaDB, SQLite, and Qdrant store the byte verbatim, so only pgvector hard-fails. Add a recursive `_strip_nul` helper and apply it to id, document, and metadata in `_PgVectorClient.upsert_rows`, mirroring the backend-layer sanitization `_sanitize_documents_for_chromadb` already does for lone surrogates on the same bulk-ingest paths. ids are SHA-256 hashes and metadata keys are fixed field names, so the id and key passes are no-ops in practice; only transcript-derived values change. Co-authored-by: hrabbach <181709360+hrabbach@users.noreply.github.com> * fix: address PR review feedback (preserve "unknown" label; use super().model) #1748: normalize the sqlite fast path's "?" COALESCE placeholder (and None) back to "unknown" inside _sqlite_taxonomy, so drawers missing wing/room metadata keep the client path's output contract — no observable API change for MCP clients on legacy/partial drawers. #1068: invoke the parent embedder build via super().model instead of reaching into cached_property's .func attribute, so the uncapped/fallback path survives chromadb changing `model` to a plain @property or other descriptor. * perf(mcp): sqlite fast path for graph_stats to fix large-palace timeouts (#1379) tool_graph_stats built the whole palace graph via build_graph(), which pages every metadata row (col.get limit/offset) and cold-loads the HNSW index — the remaining overview-tool timeout from #1379 (#1836 fixed status / list_wings / list_rooms / get_taxonomy but deliberately left graph_stats out, as it builds an in-memory graph rather than a flat tally). Add _sqlite_graph_stats(): one GROUP BY room, wing, hall over chroma.sqlite3, reconstructing build_graph's room_data and the same stats (total_rooms, tunnel_rooms, total_edges, rooms_per_wing, top_tunnels) with the same per-drawer filter (room present, != "general", wing present) and edge semantics (C(wings, 2) * halls per multi-wing room). Same _is_chroma_backend() guard + client-path fallback as the #1748 overview tools. Test seeds a real chroma palace mirroring the build_graph parity case in test_palace_graph, with a tripwire on graph_stats proving the fast path runs and that "general"/wing-less drawers are excluded. Idea adapted from #1381's _sqlite_graph_stats. * fix: address PR review feedback on graph_stats sqlite fast path (#1379) - Soft-fallback on any exception, not just sqlite3.Error, so an unexpected schema shape tripping the reconstruction degrades to build_graph() instead of raising — matching the sibling sqlite fast paths (Copilot). - Guard an empty/None _config.palace_path before building db_path (Gemini). - Test: tripwire _get_collection in addition to graph_stats, directly asserting the fast path never opens the chroma client / cold-loads HNSW (Copilot). * fix: percent-encode sqlite read-only URIs so spaced/special-char paths open sqlite3.connect(f"file:{db_path}?mode=ro", uri=True) mis-parses paths containing spaces or other URI-reserved characters — common in real home directories (a Windows "First Last" user folder, many macOS paths), and made worse by Windows backslashes. The database silently fails to open and the read-only fast paths fall back (or error) on those machines. Add config.sqlite_read_uri(), which percent-encodes the path via urllib.request.pathname2url (lazy-imported to keep config import light), and route every read-only sqlite reader through it: - mcp_server._tool_status_via_sqlite - searcher BM25 sqlite fallback - repair (status / scan / max-seq read paths) - backends/chroma (5 readers: counts, wing/room tally, id maps, etc.) All previously used the same naive f-string construction. Surfaced as a gemini-code-assist review note on #1837. * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(mcp): route _sqlite_graph_stats through sqlite_read_uri The graph_stats sqlite reader (#1837) and the sqlite_read_uri encoding fix (#1838) landed in parallel, so _sqlite_graph_stats was the one reader left on the naive f"file:{db_path}?mode=ro" construction that mis-parses paths with spaces/special chars. Convert it too — now every read-only sqlite reader percent-encodes its path. sqlite_read_uri is already imported in mcp_server from #1838, so this is a call-site-only change. * fix(pgvector): push get(limit, offset) pagination into SQL (#1830) PgVectorCollection.get(limit=, offset=) ignored pagination at the SQL layer: scroll_rows ran SELECT ... WHERE <where> with no LIMIT/OFFSET, so get() fetched the whole table and sliced in Python. prefetch_mined_set pages the whole palace on every mine, so mining was O(N^2) in rows transferred and Python objects built as the palace grows; every other paginating caller (exporter, migrate, repair, hallways, closet_llm, miner, palace_graph) paid the same cost. Push LIMIT/OFFSET into scroll_rows/_scroll with ORDER BY id (the primary key) for stable offset pagination. get() uses the pushed path only for an unfiltered page (no ids, no where/where_document, non-negative bounds); a filtered get keeps the full-scan path because the metadata @> ... pushdown is broader than the exact _matches_where re-filter for array/object values, so that re-filter must run before pagination. Full-scroll callers pass no bound, so their SQL is unchanged. Co-authored-by: hrabbach <181709360+hrabbach@users.noreply.github.com> * fix(pgvector): replace lone surrogates so a transcript surrogate no longer aborts the mine (#1833) A lone UTF-16 surrogate (U+D800-U+DFFF) in transcript content has no UTF-8 encoding, so pgvector's bulk upsert_rows makes psycopg raise UnicodeEncodeError and the whole mine aborts, leaving later files unmined. Apply config.strip_lone_surrogates (-> U+FFFD) to id, document, and the serialized metadata JSON in upsert_rows. json.dumps(ensure_ascii=False) leaves a metadata surrogate raw in the string, so one pass over the serialized JSON covers it; NUL, by contrast, json-escapes and must be stripped before serialization (see #1829). Replace rather than drop, matching ChromaDB's document handling. Verified end to end against live Postgres + pgvector: before, a surrogate in document or metadata aborts the mine; after, it ingests and round-trips as U+FFFD. Fixes #1833 * fix(backends): push sqlite_exact get(limit, offset) pagination into SQL SQLiteExactCollection.get(limit, offset) fetched the whole collection via _rows() (SELECT ... FROM documents ORDER BY rowid, no LIMIT/OFFSET) and sliced in Python, so every paginating caller (prefetch_mined_set, status, exporter, migrate, dedup, sync, ...) re-scanned the entire table per page, making the sweep O(N^2) in rows materialized. Push LIMIT/OFFSET into the scan on the unfiltered page (no ids/where/ where_document and non-negative bounds); filtered, id, and negative pages keep the full-scan plus Python-slice path so the post-filter still runs first. SQLite requires a LIMIT before OFFSET, so an offset-only page uses LIMIT -1. ORDER BY rowid keeps pages stable. * ci: re-trigger checks (unrelated Windows closet flake) * refactor(qdrant): reuse _rows() in get_all_metadata(); fix sys.modules test pollution Addresses maintainer review on #1832 (the two non-blocking 🟡 items plus two 🟢 nits) * test(mcp_server): add missing _fetch_all_metadata delegation/fallback tests. * feat(search): add an optional source_file filter to mempalace_search (#1815) Expose source_file alongside wing/room on mempalace_search. build_where_filter generalizes to 0/1/2+ clauses and the filter threads through the main vector path, the index-mismatch fallback, the vector-disabled BM25/SQLite path, and the union lexical path so it never silently no-ops. Matching is on the exact full stored value; results now expose source_path (the full path) for round tripping, since the displayed source_file is a basename. The MCP schema gains the source_file property and a path-tolerant sanitizer rejects null bytes, lone surrogates, and overlong values. Fixes #1815 Co-Authored-By: rendigua2025-gif <253093224+rendigua2025-gif@users.noreply.github.com> * fix(mcp): reject non-string source_file with a clean error (#1815) A JSON number or boolean passed for source_file is not coerced by the string schema type, so it reached _sanitize_optional_source_file and raised AttributeError from .strip() rather than a clean validation error. Add an isinstance guard that raises ValueError, which tool_search returns as a structured error. Regression test added. * ci: re-trigger Windows (flaky closet-boost test) * fix(repair): point index-read failures to repair --mode from-sqlite (#1843) When the chromadb compactor cannot apply the WAL into the drawers HNSW segment (InternalError: Failed to apply logs to the hnsw segment writer), the legacy repair paths fail on their first Collection.count() read and advise re-mining from source files. The drawer rows are intact in chroma.sqlite3, so repair --mode from-sqlite rebuilds them; re-mining silently drops drawers added via the MCP server and diary entries that have no source file. Both legacy read-failure sites (cmd_repair and rebuild_index) now emit shared guidance pointing at the from-sqlite recovery, worded conditionally so it also covers a live server or mine still holding the palace open. Co-Authored-By: undeadindustries <9536461+undeadindustries@users.noreply.github.com> * fix: use CREATE_NO_WINDOW so Windows hook miner spawns don't flash a console (#1783) Fixes #1783 * fix: point diverged-index recovery at from-sqlite, not re-mine (#1843) A diverged HNSW index (for example after a failed chromadb compaction) leaves the drawer rows intact in chroma.sqlite3 but the vector index out of sync. Re-mining to recover silently drops drawers added through the MCP server and diary entries, which have no source file. - repair-status now recommends `mempalace repair --mode from-sqlite --archive-existing` when DIVERGED, instead of the generic `mempalace repair`, and explains why re-mining loses data. - The shared recall protocol and the recall skills (Cursor + Claude plugin) document the compactor / "Not connected" recovery path: stop the server, rebuild from SQLite, verify, restart — never repair in-process from the agent. Complements #1847 (legacy repair error messages); does not duplicate it. Does not close #1843 — MCP reconnect resilience and honest add_drawer write signalling remain open. Co-authored-by: Cursor <cursoragent@cursor.com> * docs: add Windows backup alternative to corrupt-index recovery (#1843) Gemini review on PR #1849: the optional palace backup step used the Unix-only `cp -a`, which fails on Windows. MemPalace ships on win32, so add a PowerShell `Copy-Item` alternative alongside the macOS/Linux form and note that `--archive-existing` already moves the old palace aside. Co-authored-by: Cursor <cursoragent@cursor.com> * feat: add mempalace_checkpoint batch save tool Collapse the Cursor auto-save sequence (check_duplicate Nx + add_drawer Nx + diary_write 1x) into a single mempalace_checkpoint MCP call so the host UI renders one tool-call card and keeps its spinner up for the whole save. The new tool reuses the existing single-item handlers, so semantic dedup, idempotency, and verbatim guarantees are unchanged. - mcp_server.py: add tool_checkpoint + register mempalace_checkpoint - service.py: classify mempalace_checkpoint as a write tool - cursor save hook: followup now drives one mempalace_checkpoint call - docs: new mcp-tools.md section, help.md entry, 33 -> 34 tool count sweep - tests: checkpoint add/dedup/malformed/registry + classify_tool Co-authored-by: Cursor <cursoragent@cursor.com> * fix: harden tool_checkpoint input validation Address PR review: guard untrusted MCP client payloads in mempalace_checkpoint so a single malformed item cannot raise deep in sanitization and abort the whole batch. - coerce dedup_threshold to float - require wing/room/content to be non-empty strings (skip + record error) - validate the diary object and entry type, recording errors instead of silently ignoring a malformed diary On a genuine dedup-check error we still file the drawer rather than skip: verbatim recall is the priority and add_drawer's idempotency blocks exact duplicates. Adds tests for the non-string, dedup-error, and malformed-diary paths. Co-authored-by: Cursor <cursoragent@cursor.com> * test: update Cursor followup assertion for checkpoint tool The save-hook followup now drives a single mempalace_checkpoint call, so test_threshold_emits_followup_message must assert that tool name instead of the old add_drawer/check_duplicate/diary_write trio. Fixes the test-macos / test-linux CI failures on this branch. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): purge matching closets in delete_by_source (#1722) delete_by_source removed only the drawers, leaving the matching closets (the AAAK index layer, keyed independently by source_file) behind as stale pointers at the now-deleted source. Mirror the closet-purge step used by sync_palace / purge_file_closets: after the drawer delete, best-effort purge the closets via push-down delete(where=...) so it survives large palaces and can never abort an already-committed drawer delete. Dry run now also reports closet_match_count so the caller sees the full blast radius; commit reports closets_deleted. Adds tests that seed the closet collection directly (tool_add_drawer doesn't build closets) and assert the matching closets are purged on commit and counted on dry run. * feat(mcp): add opt-in HTTP transport * fix suggestion of reviewer to avoid a critical race condition and other fixes * test(mcp): keep HTTP transport tests Python 3.9 compatible * test(mcp): avoid subprocess flakiness in HTTP transport tests * test(mcp): bypass proxies in HTTP transport loopback tests * test(mcp): make HTTP transport loopback tests proxy-free * test(mcp): make HTTP transport tests network-free * fix(mcp): move _HTTP_REQUEST_LOCK and _HTTP_MAX_REQUEST_BYTES * fix(mcp): move _HTTP_REQUEST_LOCK * fix reviewer: handling JSON-RPC * fix(tests): rewrite test_mcp_http_transport for Python 3.9-3.13 + Windows * fix(lint): resolve 7 ruff errors in test_mcp_http_transport * fix(mcp): harden HTTP transport — DNS-rebinding guard, optional token, real tests The opt-in HTTP transport reuses the stdio dispatcher and binds loopback by default, but /mcp was unauthenticated with no protection against a malicious web page reaching a DNS-rebound localhost server, and its tests reached for Starlette/uvicorn (not project deps) so they were silently skipped in CI — the production _serve_http handler had zero coverage. Hardening: - Pin the Host header to loopback literals + the bound host on a loopback bind (DNS-rebinding defense); relaxed for a deliberately non-loopback bind, which is the operator's call and may sit behind a Host-rewriting proxy. - Reject any browser Origin that isn't a loopback origin (rebinding/SSRF guard); non-browser MCP clients omit Origin and are unaffected. - Optional bearer token via MEMPALACE_MCP_HTTP_TOKEN (constant-time compare); required on /mcp, never on /healthz so liveness probes work credential-free. - Warn loudly when bound to a non-loopback host (palace reachable from network). Testability: - Split _build_http_server() out of _serve_http() so tests bind 127.0.0.1:0 and drive the real handler over a loopback socket via stdlib http.client. - Replace the skipped Starlette reimplementation with 12 tests covering dispatch, initialize, /healthz, 404, parse-error, the 16 MiB cap, notification 202, and the Host/Origin/token rejections — no third-party deps. * ci(test-windows): retry the transient ChromaDB HNSW compaction flake ChromaDB's rust HNSW core intermittently fails compaction on Windows with "Failed to apply logs to the hnsw segment writer" during add/update — a long-standing, non-reproducible-on-Linux/macOS flake that hits different tests (test_migrate_wings, test_closets) across unrelated commits and has been turning otherwise-green release/CI runs red at random. Add pytest-rerunfailures and wire `--reruns 2 --only-rerun "Failed to apply logs to the hnsw segment writer"` into the test-windows job only. The --only-rerun scope means a real, deterministic failure still fails on the first run; only this specific transient native-dependency error is retried. The Linux and macOS jobs deliberately keep zero reruns so genuine regressions surface there loudly. * chore(release): 3.5.0 Bump version to 3.5.0 across version.py, pyproject.toml, the Claude/Codex plugin manifests, the README badge, and uv.lock. Refresh the "N MCP tools" prose from 34 to 35 (delete_by_source #1729 and checkpoint #1851 each added a tool). Add the 3.5.0 CHANGELOG entry. * fix: tighten local guards and file handling * fix: restore convo miner scan indentation * fix: green up CI for hardened file handling - ruff format llm_client.py and miner.py (lint job) - _copy_file_no_follow: close src fd if the dst open fails (no leak), and route the rebuild restore through it so backup + restore share one no-follow/regular-file path - update repair tests to assert the unified hardened copy instead of the removed shutil.copy2 calls; backup paths are now timestamped - update normalize large-file test to stub fstat (size is checked on the open fd, not via a pre-open os.path.getsize) * test(wal): cover crash-safety, idempotent setup, and redaction edge paths The write-ahead log gained its own module in v3.5.0 but sat at 82% coverage; the uncovered lines were exactly the failure/guard branches that uphold its contracts: the cache-hit early return, the restricted-FS chmod/mkdir swallow paths, and the promise that a WAL write failure is logged and never crashes the calling tool. Add five tests covering those branches plus the non-string redaction marker, bringing mempalace/wal.py to 100% and locking the crash-safety guarantees against regression. Test-only; no production change. * fix: spawn daemon with CREATE_NO_WINDOW to match hook miner (#1783) (#1857) daemon.py:_detached_kwargs was the last production spawn site still using DETACHED_PROCESS. Swap it to CREATE_NO_WINDOW, matching the hook miner's _detached_popen_kwargs fixed in #1848 — the dedicated follow-up the review bot asked for. `grep -rn DETACHED_PROCESS mempalace/` now returns zero production hits. Survivability is unchanged: CREATE_BREAKAWAY_FROM_JOB (escapes the parent Job Object's kill-on-close) plus the daemon never being attached to the launching console carry survive-terminal-close; CREATE_NEW_PROCESS_GROUP (also kept) isolates Ctrl-C/Break. CREATE_NO_WINDOW is ignored when OR'd with DETACHED_PROCESS, so this replaces the flag rather than adding it. The daemon already redirects stdout/stderr to daemon.log and reads no stdin, so it needs no console. Adds the first tests for _detached_kwargs (posix + windows, cross-platform monkeypatch of the Windows-only flag constants, mirroring the #1848 hooks_cli tests). * fix(cli): add repair rebuild-index alias (#1670) * Fix/wing slug special chars (#1852) * fix: sanitize wing slug for project dirs with special characters Project folders containing characters outside sanitize_name's set (e.g. a leading '+') leaked into the derived wing name, producing names like 'wing_+project' that config.sanitize_name rejects, silently breaking diary auto-save for that project. Add _safe_wing_slug(): collapse non-word runs to '_', trim, and fall back to 'sessions' when a name reduces to nothing. Route the three wing-derivation sites through it. Tests: unit cases for the helper plus a hypothesis property test asserting wing_<slug> always passes sanitize_name for any inp…
xcarbo
pushed a commit
to xcarbo/mempalace
that referenced
this pull request
Jul 3, 2026
(cherry picked from commit 8654634)
5 tasks
jphein
added a commit
to techempower-org/mempalace
that referenced
this pull request
Aug 9, 2026
#394) * fix(mine): route SKIP to stderr and cover stat() OSError arm (#923) The original commit printed SKIP for oversized files to stdout but the sibling SKIP for symlinks in the same scan_project / scan_convos already went to stderr. Align the new line with that convention. Also adds a SKIP-with-error log for the except OSError arm right below the size check. Files whose stat() raises (permission denied, racing delete, broken symlink that survived the earlier is_symlink check) were the same bug class as the silent oversize drop. Tests switched from captured.out to .err and tightened to the full template; new test covers the OSError arm via a selective Path.stat monkeypatch with a follow_symlinks gate for Python 3.10+. * fix: spawn daemon with CREATE_NO_WINDOW to match hook miner (#1783) (#1857) daemon.py:_detached_kwargs was the last production spawn site still using DETACHED_PROCESS. Swap it to CREATE_NO_WINDOW, matching the hook miner's _detached_popen_kwargs fixed in #1848 — the dedicated follow-up the review bot asked for. `grep -rn DETACHED_PROCESS mempalace/` now returns zero production hits. Survivability is unchanged: CREATE_BREAKAWAY_FROM_JOB (escapes the parent Job Object's kill-on-close) plus the daemon never being attached to the launching console carry survive-terminal-close; CREATE_NEW_PROCESS_GROUP (also kept) isolates Ctrl-C/Break. CREATE_NO_WINDOW is ignored when OR'd with DETACHED_PROCESS, so this replaces the flag rather than adding it. The daemon already redirects stdout/stderr to daemon.log and reads no stdin, so it needs no console. Adds the first tests for _detached_kwargs (posix + windows, cross-platform monkeypatch of the Windows-only flag constants, mirroring the #1848 hooks_cli tests). * fix(cli): add repair rebuild-index alias (#1670) * Fix/wing slug special chars (#1852) * fix: sanitize wing slug for project dirs with special characters Project folders containing characters outside sanitize_name's set (e.g. a leading '+') leaked into the derived wing name, producing names like 'wing_+project' that config.sanitize_name rejects, silently breaking diary auto-save for that project. Add _safe_wing_slug(): collapse non-word runs to '_', trim, and fall back to 'sessions' when a name reduces to nothing. Route the three wing-derivation sites through it. Tests: unit cases for the helper plus a hypothesis property test asserting wing_<slug> always passes sanitize_name for any input. * fix: preserve dots and apostrophes in wing slug for backward compatibility The first pass collapsed every non-word character (including dot and apostrophe) to underscore, renaming existing valid wings — e.g. my.app became wing_my_app — which would orphan diary entries already filed under the old name. Keep dot and apostrophe (both accepted by sanitize_name), collapse consecutive dots to avoid the path-traversal rejection, and trim edge separators. Add backward-compatibility tests for previously-valid names plus a double-dot collapse test. * fix: cap wing slug length to stay within sanitize_name's limit sanitize_name rejects names over 128 characters, so a very long project directory name would produce a wing name that fails validation, re-triggering the silent auto-save break this PR fixes. Truncate the slug to 120 chars (the wing_ prefix keeps the total under 128). Widen the hypothesis property test to max_size=300 so it exercises the length path, and add an explicit truncation test. Addresses gemini-code-assist review feedback on PR #1852. --------- Co-authored-by: Ivan Antsimonau <ivan.antsimonau@katim.com> * fix(hooks): hide conhost window on Windows in _mine_sync non-daemon path (#1863) The non-daemon synchronous mine fallback in _mine_sync() spawned the mine subprocess without CREATE_NO_WINDOW, flashing a visible console window on every PreCompact fire on Windows. The async paths (_spawn_mine, _desktop_toast) already pass it via _detached_popen_kwargs(); this sync path was missed. getattr(..., 0) is a no-op off-Windows. Fixes #1862 Co-authored-by: David Finkelstein <david@finkelstein.us> * fix: expand tilde in palace_path when read from config file\n\nMempalaceConfig.palace_path correctly called os.path.expanduser() for\nenv-var paths but not for paths read from config.json. If config.json\nstores palace_path as '~/.mempalace/palace' (the default written by\ninit), the tilde was returned unexpanded.\n\nDownstream callers such as cli.py cmd_mine did call expanduser when\n--palace was passed explicitly, but fell through to MempalaceConfig()\nwhen no flag was given, inheriting the unexpanded string. Python's\nos.makedirs and chromadb.PersistentClient treat a leading tilde as a\nliteral directory name rather than the home directory, so the palace\nwas silently written to a CWD-relative path such as\nmy_project/~/.mempalace/palace.\n\nThe fix is a single os.path.expanduser() call on line 343 of\nconfig.py, mirroring the existing env-var branch on line 342. Since\nDEFAULT_PALACE_PATH is already expanded at module load (line 197),\nexpanduser on an absolute path is a no-op, so the default case is\nunaffected.\n\nSymptoms: scattered {project}/~/.mempalace/palace directories, palace\nalways appears empty after mine, search returns Collection does not\nexist, launchd-driven nightly mine writes to a different location than\ninteractive mine.\n\nCo-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>n (#1865) * fix(chroma): stop quarantining valid all-layer-0 HNSW segments (#1716) An empty link_lists.bin is not corruption on its own: hnswlib stores the layer-0 graph inside data_level0.bin and only writes link_lists.bin for elements promoted to level > 0. A small/low-fanout index where every element stays on layer 0 serializes an empty link_lists.bin and loads fine. Flagging that shape as corrupt produced a self-perpetuating quarantine loop — repair rebuilt the byte-identical all-layer-0 segment, the next cold start re-quarantined it, accumulating drift dirs (221 MB in the reported case) with no ingestion involved. Use the persist-completion marker as the discriminator instead. ChromaDB writes index_metadata.pickle last, so an intact pickle envelope proves the flush finished and the empty link_lists.bin is the legitimate all-layer-0 shape. Only treat an empty link_lists.bin as a partial flush when there is real payload AND no completion marker (absent or truncated pickle). The #1457 partial-flush protection (real payload, no/truncated marker) is preserved; the byte-sniff is factored into _hnsw_metadata_marker_intact and reused by _segment_appears_healthy. Also fixes the related single-writer stale-quarantine false positive (#1564), which shares this all-layer-0 root cause. * fix(repair): auto-heal isolated FTS5 inverted-index corruption (#1596) Concurrent killed-mid-write mines can leave embedding_fulltext_search in a malformed-inverted-index state that fails PRAGMA quick_check while the underlying rows stay intact (integrity_check ok). The repair preflight then hard-aborts before reaching the FTS5 rebuild step, so `mempalace repair` refuses to run and full-text search stays broken — the exact loop #1596 reports. The MineValidationError banner even promises "repair --yes rebuilds the FTS5 virtual table automatically," which the preflight abort made false. Add maybe_autoheal_fts5_index(): when every quick_check error is an isolated "malformed inverted index for FTS5 table" failure, rebuild the index in place from the intact embedding_fulltext_search_content table (INSERT ... VALUES('rebuild')) under mine_palace_lock, then re-run quick_check. The rebuild touches no drawer rows. Wired into both repair preflights (rebuild_index and cli cmd_repair). Any non-FTS5 error in the set, a lock held by a live mine, or a rebuild that does not clear quick_check leaves the errors unchanged so the caller still aborts with the recovery banner — broader corruption is never silently rebuilt over. * fix(mcp): stop clobbering host app root logger at import (#1860) (#1885) * fix(mcp): stop clobbering host app root logger at import (#1860) _init_logging() ran at import and called logging.basicConfig(force=True), resetting the root logger's level, format, and handlers unconditionally. An app that configured logging before importing mempalace.mcp_server lost its setup: a host on DEBUG dropped to INFO, custom formatters and handlers were replaced. force=True existed (#1495) only to keep MEMPALACE_LOG_FILE working when root already had handlers. This keeps that contract without the reset: configure root only when it is unconfigured (standalone); otherwise attach a mempalace-filtered file handler additively and leave the host's config alone. Adds _MempalaceLogFilter so the file captures every mempalace logger (the dotted mempalace.* family plus the flat mempalace_* names) and nothing else. * fix(mcp): survive importlib.reload and pin file log format (#1860) Addresses review on #1885. Restore _logging_configured from globals() so the idempotency guard survives importlib.reload: a reload re-executes the module body, and a plain reset would let _init_logging() stack a duplicate file handler on root. Set an explicit "%(message)s" formatter on the file handler so the embedded path does not depend on logging's default formatter (which already renders the same, but is now pinned and identical to the standalone path). Adds a reload regression test and a format-pin assertion. * fix(layers): order L1 wake-up by recency so it surfaces the latest moments (#1630) L1's generate() scored drawers by importance/emotional_weight/weight, and the docstring promised "prefer high importance, recent filing". But no ingest path (miner, convo_miner, diary, add_drawer) writes any of those fields, so the sort collapsed to insertion order (oldest first) and recency was never consulted. A scoped `wake-up --wing X` therefore surfaced the *oldest* moments: the opposite of useful. Add filed_at (present on every drawer, ISO-8601, lexically chronological) as the secondary sort key. Importance stays primary for the day a scoring pass populates it; filed_at is the effective signal today, making the "recent filing" half of the promise true with data already present. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> * fix(cli): force UTF-8 when reading/writing .gitignore in init (#1648) On Windows, Path.read_text() and open(path, 'a') use locale encoding (GBK on Chinese-locale systems) before PEP 686 / Python 3.15. A valid UTF-8 .gitignore with non-ASCII comments crashes _ensure_mempalace_files_gitignored() with UnicodeDecodeError, which aborts 'mempalace init' on Windows for any user whose .gitignore contains non-ASCII text. Force encoding='utf-8' on both read and append, with errors='replace' on read as a defensive fallback for legacy mixed-encoding files. Co-authored-by: ALaDingAhmad <16530935@qq.com> Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> * chore(deps): bump actions/checkout from 6 to 7 (#1882) Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v6...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump docker/setup-qemu-action from 3 to 4 (#1880) Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 3 to 4. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/v3...v4) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump docker/setup-buildx-action from 3 to 4 (#1881) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3 to 4. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](https://github.com/docker/setup-buildx-action/compare/v3...v4) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps-dev): bump ruff from 0.15.18 to 0.15.20 (#1883) Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.18 to 0.15.20. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/ruff/compare/0.15.18...0.15.20) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.15.20 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(backends): require SQLite magic header for chroma + sqlite_exact detect() (#1893) (#1896) * fix(chroma): require SQLite magic header for ChromaBackend.detect() (#1893) Closes #1893. ChromaBackend.detect() was returning True for a 0-byte chroma.sqlite3 file because the check was just os.path.isfile(...). On a palace that has any other backend marker alongside a stale 0-byte chroma.sqlite3, resolve_backend_name then raises BackendMismatchError and the palace becomes unopenable until the user manually rm's the empty file. The 0-byte file appears as a side effect of any sqlite3.connect() on a missing path — Python creates the file immediately but writes the SQLite header only on the first statement. So any code path that touches the chroma.sqlite3 path with bare sqlite3.connect(), including chromadb's own PersistentClient lazy-init (see the comment at backends/chroma.py:2052), can leave a 0-byte artifact behind. Fix: detect() now reads the first 16 bytes and compares to the SQLite magic prefix b"SQLite format 3\x00" instead of relying on file presence alone. One extra open() + 16-byte read; detect() isn't a hot path. Properties: - Rejects 0-byte files (the symptom #1893 is about). - Rejects non-SQLite garbage at the canonical path (partial writes, etc.). - Doesn't false-negative on real chroma palaces: any chroma palace whose PersistentClient has done any work has the magic header on disk (verified — CREATE TABLE is enough to land the header). - Doesn't couple detect() to chroma's specific schema; the magic header is stable across chromadb releases. Test sweep: many test files used (chroma.sqlite3).touch() or .write_bytes(b"") as a "fake palace" shortcut, exploiting the loose isfile() check (one such site even had the comment "# pass the isfile guard"). After this change, those stand-ins no longer register as chroma palaces. Introduced tests/_chroma_palace_helper.py::make_minimal_chroma_sqlite following the existing _backend_conformance.py precedent, and updated 15 call sites across 8 test files to use it. The existing test_chroma_detect_matches_palace_with_chroma_sqlite (which encoded the buggy semantics with write_bytes(b"")) is renamed to test_chroma_detect_matches_palace_with_sqlite_header and now writes a real SQLite database via the helper. Added two new tests for the rejection paths (empty file, non-SQLite garbage). Full env-cleared suite: 3137 passed, 20 skipped, 0 failed. ruff check and ruff format --check both clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA * fix(sqlite_exact): require SQLite magic header for SQLiteExactBackend.detect() Per gemini-code-assist review on #1892 PR #1896: SQLiteExactBackend has the same os.path.isfile() detection pattern as ChromaBackend did, with the same 0-byte-file vulnerability. Mirrors the chroma fix for repo-wide consistency. - SQLiteExactBackend.detect() now does the same 16-byte SQLite magic-prefix check as ChromaBackend.detect(). - _chroma_palace_helper.py: factored its body into a private _write_minimal_sqlite_file() and gained a sibling make_minimal_sqlite_exact_sqlite() for the sqlite_exact filename. No churn to any existing chroma call sites. - test_sqlite_exact_backend.py:426 (the one site that wrote b"" for sqlite_exact.sqlite3) updated to use the new helper. - Three new tests in test_sqlite_exact_backend.py mirror the chroma trio: matches with valid header, rejects empty file, rejects non-SQLite garbage. Full env-cleared suite: 3140 passed, 20 skipped, 0 failed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * fix(pgvector): skip document column for metadata-only fetches (#1840 follow-up) (#1892) * fix(pgvector): skip document column for metadata-only fetches (#1840 follow-up) Closes the explicit "separate follow-up to keep this low-risk" callout in PR #1840's description. For remote pgvector deployments (TLS over WAN), `mempalace_status` and every other metadata-only consumer was transferring the full `document` column over the wire even when nothing read it. A single scroll over a 177K-drawer palace on a 175 ms-RTT link moved ~150 MB of document text plus ~50 MB of metadata; this PR drops that to ~50 MB. scroll_rows / _scroll gain `with_document: bool = True`. When False, SELECT projects NULL::text instead of the document column. Positional _row parser unchanged (record[1] stays the document slot, just receives NULL). Existing callers default to True and see byte-for-byte identical behavior. PgVectorCollection.get_all_metadata override: where=None path goes single-scroll with with_document=False. Filtered path falls back to base to keep _matches_where running on array/object metadata values (same correctness contract as #1840's filtered-path decision). Tests: - Update _FakePgVectorClient.scroll_rows to accept with_document; mirror the NULL-becomes-empty-string semantics when False - Update 5 existing scroll_calls assertions to include with_document=True (unchanged intent) - test_pgvector_get_all_metadata_skips_document_column: assert exactly one scroll call with with_document=False - test_pgvector_get_all_metadata_filtered_falls_back_to_base: assert filtered path preserves with_document=True Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA * fix(pgvector): extend with_document=False fast path to filtered get_all_metadata Per gemini-code-assist review feedback on #1892: _matches_where only reads metadata, so the where=None vs where=set conditional fall-back was unnecessary. The filtered path can use the same single-scroll with_document=False fast path and apply the post-filter locally on metadata dicts — extending the wire-byte win to every get_all_metadata caller, not just unfiltered ones. Mirrors the pushdown + local _matches_where pattern already used by _rows in the same file: pushdown when _requires_local_filter is False, post-filter in Python otherwise. Same correctness contract as #1840's filtered get path. Renames test_pgvector_get_all_metadata_filtered_falls_back_to_base to test_pgvector_get_all_metadata_filtered_uses_fast_path and asserts the new behavior (with_document=False + pushdown forwards the equality filter to SQL). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * feat(convo): preserve authored timestamp from transcripts (#1890) * feat(convo): preserve authored timestamp from transcripts Conversation drawers only carried `filed_at` (ingest time), so a bulk re-mine collapsed every drawer to a single instant and the chronological signal was lost — even though each Claude Code / Codex JSONL line already carries an ISO-8601 `timestamp`. The recency-window fallback and any date-aware consumer then saw ingest order, not when content was written. - convo_miner: derive `authored_at` (per-file max line `timestamp`) and store it as drawer metadata; falls back to `filed_at` when absent - searcher: surface `authored_at` in search results, and break exact hybrid-score ties toward the more recently authored drawer (ISO strings sort chronologically; missing dates sort oldest) — benchmark-neutral as it only reorders exact ties - tests: cover `_extract_authored_at` (latest wins, skips/tolerates lines without timestamps, non-jsonl/missing -> None) and the tie-break Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(search): surface authored_at in CLI + backfill for existing data Completes the authored_at work so the field is visible end-to-end and existing palaces can adopt it without re-mining. - layers: CLI `search` output shows an `authored:` date line per result (peer of the existing date; markdown drawers fall back to filed_at) - scripts/backfill_authored_at.py: in-place migration that stamps authored_at on convos drawers from their source transcripts — metadata only (no re-embedding), idempotent, dry-run by default - docs/authored-at.md: documents created_at (ingest) vs authored_at (written) and both backfill paths (in-place / drop-and-recreate) - tests: backfill integration tests over an ephemeral ChromaDB collection Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(search): address review — non-string timestamp guard + top-level authored_at tiebreak Two correctness fixes from the PR review: - _extract_authored_at: only compare when the parsed `timestamp` is a str. A non-string timestamp on a malformed/foreign JSONL line previously raised TypeError outside the try and could crash the mine. - _hybrid_rank: the tie-break read `authored_at` only from nested `metadata`, but the search_memories path (MCP / Claude Code) carries it at the top level of each hit — so the tie-break silently no-op'd there. Read both shapes. - tests: non-string timestamp cases, and a top-level-shape tie-break test (which fails before this fix). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: apply ruff format to authored_at changes CI ruff format --check flagged 4 files; ruff check already passed. Formatting only — no behavior change. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> * fix(palace): process-wide mine_palace_lock re-entrancy so the HTTP transport can write (#1859) * fix(palace): process-wide mine_palace_lock re-entrancy for threaded HTTP transport The MCP HTTP transport (ThreadingHTTPServer) acquires the long-lived writer-lease on one thread (_acquire_mcp_writer_lock) but dispatches each write request on a different worker thread. The lock re-entrancy guard was thread-local, so write handlers (add_drawer/update_drawer) failed to see the process-held lease, re-acquired the flock, and self-conflicted with "palace ... is held by PID <self>". Reads worked (no lock); writes over the HTTP transport were impossible. Make the re-entrancy record process-wide (pid-tagged, guarded by a threading.Lock) so a write from any thread of the process that already holds the lease passes through. Safe: flock is per-process and HTTP writes are serialized by _HTTP_REQUEST_LOCK. Preserves fork-safety, same-thread nesting (miner.mine -> ChromaCollection.upsert), and cross-process protection (MineAlreadyRunning still raised between processes). Add cross-thread same-process regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(palace): reset lock guard on fork to avoid inherited-locked deadlock Address review (PR #1859): `_palace_lock_guard` is a threading.Lock, so a child forked while another thread held it would inherit it locked (the holder thread is gone in the child) and deadlock on the next acquire. Register an os.register_at_fork(after_in_child=...) handler that replaces the guard with a fresh unlocked lock and clears state; the child must reacquire the flock anyway. Guarded by hasattr(os, "register_at_fork") for Windows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> * feat(mcp): add since/before date filter to list_drawers (#1128) (#1891) * feat(mcp): add since/before date filter to list_drawers (#1128) mempalace_list_drawers previously filtered only by wing/room. This adds optional since/before ISO date bounds on filed_at: since is inclusive, before is exclusive. The filter runs in Python after the rows are fetched. ChromaDB 1.5.7 rejects string operands for $gte/$lt and filed_at is stored as an ISO string, so a server-side where comparison is not available; the tool already collapses and paginates the full result set in Python. Drawers whose filed_at is missing or unparseable are excluded while a bound is active, and inverted bounds (since >= before) return a clear error. * test: close chromadb clients between tests to fix Windows handle leak (#1128) chromadb 1.5.7 caches one System per palace path and only frees the SQLite/HNSW file handles on client.close(); the collection fixture and the per-test MCP cache reset only dereferenced the client, so handles leaked across the session. Harmless on POSIX (rmtree unlinks open files), but on Windows the handles stay locked and accumulate until an HNSW segment write in a later test's setup fails, which surfaced here as TestDeleteBySource::test_commit_purges_matching_closets asserting 0 == 2. Close the client in the collection fixture and in _reset_mcp_cache so the handles are released between tests. * test: release backend chromadb clients between tests (#1128) palace.get_collection() caches one PersistentClient per palace_path on the process-wide backend singleton and never closes it; sweep, repair and several CLI tests reach the store through it. chromadb frees the rust-side SQLite/HNSW file handles only on client.close(), so the handles leak across the whole session: a 30-palace probe shows ~200 open file descriptors into the palace tree, dropping to 0 once the clients are closed. On POSIX the open handles are harmless (rmtree unlinks open files), but on Windows they stay locked and accumulate until a later test's HNSW segment write fails ("Failed to apply logs to the hnsw segment writer"), e.g. test_sweeper.py::TestSweeperTandem::test_sweep_recovers_untaken_message_at_cursor_timestamp. Drain the cached clients in the autouse _reset_mcp_cache teardown via close_palace(), which closes each PersistentClient (releasing its handles) without marking the backend closed so it stays reusable. Complements the collection-fixture and _client_cache close() added earlier. * feat: optimize metadata counting using Qdrant server-side facets (#1868) * feat: add metadata facet support for qdrant * added benchmark * updated benchmark * chore: remove tracking for local scratch benchmark * feat: add metadata facet support for qdrant -clean * Update mempalace/mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update mempalace/backends/qdrant.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update mempalace/backends/qdrant.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update tests/test_qdrant_backend.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update tests/test_qdrant_backend.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update tests/test_qdrant_backend.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update tests/test_mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * /fix always working tool_status() fallback fixed * /fix fallback added to tool_list_rooms * Update mempalace/mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update mempalace/mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update tests/test_qdrant_backend.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * /fix rebuilt the room populating logic * /add added temporary files for atomic transactions * Update mempalace/mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update mempalace/mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update tests/test_mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * /fix ai slop * /fix added default facet limit * Update tests/test_qdrant_backend.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * /fix added max workers pool * Update mempalace/mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update mempalace/mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update mempalace/backends/qdrant.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * /fix added clear() * Update tests/test_mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * fix(qdrant): validate facet filter before existence check; fix taxonomy test - facet_counts now validates the where filter and rejects local-only filters before the _remote_exists() short-circuit, so an unsupported filter raises UnsupportedCapabilityError even on an unmaterialized collection (matches get()/lexical_search() ordering). - test_tool_get_taxonomy_uses_metadata_facets compared concurrent room facet calls via set(), but a call() with a dict kwarg is unhashable; compare order-independently via membership instead. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> * feat(graph): auto-populate the associative graph from mined sessions (#1895) * feat(graph): auto-populate the associative graph from mined sessions Conversation mining never set the `entities` drawer metadata that hallways consume, so mined sessions produced an empty associative graph (and starved the entity-navigation / tunnel-recommendation features built on top of it). Add a no-LLM structural entity extractor and wire it into the convos mine: - entities: structural-only extractor (author-quoted code spans, URLs, file paths, qualified identifiers, CamelCase / snake_case symbols). No wordlists, no NLP models, precision-biased so prose doesn't pollute the graph. - convo_miner: set `entities` per chunk, and compute hallways after a convos mine (mirroring the project-file path). Hallways run before the FTS5 validation, which opens a direct sqlite connection that can invalidate the live Chroma collection handle on some Chroma builds. - cli: `mempalace hallways` lists the associative graph (CLI parity with the list_hallways MCP tool). - tests: extractor precision/ranking, entities metadata at mine time, CLI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(graph): address review — semicolon safety, leading-underscore snake, negative limit - entities `_clean`: strip `;` out of tokens so a URL query string or backtick span can't split the `;`-joined entities metadata field - entities `_SNAKE`: optional leading/trailing `_?` so `_extract_authored_at` and similar are matched in plain text (previously only caught via backticks) - cli `hallways`: clamp `--limit` with max(0, ...) so a negative value shows nothing instead of slicing from the end - tests for all three Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> * docs(guide): add Remote / Team Server deployment guide (#1877) (#1897) Documents running MemPalace as a central memory service for a team: HTTP MCP transport (--transport http with bearer-token auth), a networked backend (Qdrant via REST, no extra dep; or pgvector), and optional GPU embedding. Covers the security model (non-loopback token requirement, Host/Origin DNS-rebinding guard, TLS-in-front), client connection, and operating notes. Adds the page to the guide sidebar. Addresses #1877. * fix(backends): forward facet_counts + get_all_metadata in EmbeddingCollection Both methods are concrete on ``BaseCollection`` (``facet_counts`` raises ``UnsupportedCapabilityError``; ``get_all_metadata`` pages through ``self.get(include=["metadatas"])``). Python MRO resolves them on ``EmbeddingCollection`` before ``__getattr__`` ever fires, so without an explicit forwarder the wrapper silently runs the base default instead of delegating to the wrapped backend's optimized implementation. The pattern matches the existing explicit forwarders for ``distance_metric``, ``lexical_search``, and the embedder-identity trio — all added to fix the same shadow. What this means in production for the three backends that get wrapped (``EmbeddingCollection`` only applies to ``requires_explicit_embeddings`` backends — qdrant, pgvector, sqlite_exact; chroma is unwrapped and unaffected): - **facet_counts shadow (#1868 regression)**: every ``mempalace_status``, ``list_wings``, ``list_rooms``, ``get_taxonomy`` call routes through the gated ``col.facet_counts(...)`` path. The capability check passes (``supports_metadata_facets`` is on the backend), but the call hits the wrapper's MRO-resolved ``BaseCollection.facet_counts`` and raises ``UnsupportedCapabilityError``. ``mcp_server``'s broad ``except`` swallows it, logs ``WARN Failed to fetch metadata facets, falling back to client- side loop: backend does not support facet_counts``, and counts via the O(n) Python loop — the exact behavior #1868 was designed to eliminate. - **get_all_metadata shadow (#1796 / #1892 regression)**: the BaseCollection default pages through ``self.get(include=["metadatas"])`` — fine for Chroma's SQL OFFSET cursor, but on wrapped backends (qdrant, pgvector) the inner's overridden ``get_all_metadata`` is unreachable. For pgvector specifically, this means #1892's ``with_document=False`` fast path is never taken even though it's implemented — every metadata-only fetch transfers the full document column over the wire. On a 13k-drawer remote pgvector palace over WAN that's ~13MB per call, dominating wall time. Why no test caught it: backend tests (``test_qdrant_backend.py``, ``test_pgvector_backend.py``) call the methods directly on the raw collection, not through the wrapper. ``test_mcp_server.py`` facet tests use ``MagicMock()`` for the collection, which synthesizes attributes on demand and bypasses MRO entirely. Neither path covers the seam where the bug lives: ``palace.get_collection() -> EmbeddingCollection -> .method()``. Three tests pin both the fix and the bug class: - ``test_facet_counts_forwards_to_inner`` — direct integration through the wrapper, asserts the inner's recorded call matches. - ``test_get_all_metadata_forwards_to_inner`` — same shape, plus a sentinel ``get()`` on the inner so a missing forwarder would route to the base default and pick up the wrong data (observable failure, not silent). - ``test_wrapper_forwards_all_concrete_basecollection_methods`` — meta-test that enumerates every concrete public method on ``BaseCollection`` via ``inspect.getmembers`` and asserts each one is explicitly defined on ``EmbeddingCollection``. Catches the bug *class*: any future ``BaseCollection`` method with a concrete default body becomes a CI failure the moment it's added without a wrapper forwarder, with a message pointing straight at the file to edit. Full env-cleared suite: 3205 passed, 20 skipped. ``ruff check`` and ``ruff format --check`` both clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA * fix(backends): annotate EmbeddingCollection.facet_counts return type Per Gemini review (PR #1898 comment r3489013681): the forwarder lacked the ``-> dict[str, int]`` return annotation that ``BaseCollection.facet_counts`` and the sibling ``get_all_metadata`` forwarder both carry. One-line consistency fix, no behavior change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA * feat(serve): turnkey secure remote MCP server (#1877) (#1900) * feat(serve): turnkey secure remote MCP server (#1877) Add `mempalace serve`: a secure-by-default wrapper over the HTTP MCP transport so a team can stand up a shared central palace with one command. Server capabilities (mempalace/mcp_server.py): - Native TLS via --tls-cert/--tls-key (env MEMPALACE_MCP_TLS_CERT/_KEY): wraps the socket in a TLS 1.2+ context, validated before bind. Token is still required on a non-loopback bind (TLS != auth). - Read-only mode via --read-only (env MEMPALACE_MCP_READ_ONLY): the 24 mutating tools are hidden from tools/list and refused at dispatch (-32003), enforced before arg handling — not merely hidden. Turnkey command (mempalace/cli.py): - Auto-generates a strong bearer token for non-loopback binds, stored 0600 under ~/.mempalace/server/ and printed once; reused across restarts. Token rides in the child env, never argv, so it can't leak via ps. - Prints a ready-to-paste client config (scheme reflects TLS), then foreground-execs the real server so Docker/systemd own the lifecycle. Deployment (deploy/): - docker-compose.server.yml wires the server + Qdrant with a /healthz healthcheck and persistent volumes. - server.env.example documents the env surface. - mempalace-server.service is a hardened systemd unit template. Tests: TLS handshake (openssl-gated), read-only enforcement, token autogen/0600/reuse, token-not-in-argv, secure-by-default gates. Docs: remote-server guide now leads with `mempalace serve` plus Compose and systemd subsections. * test(serve): fix Windows — don't patch os.name; gate 0600 asserts to POSIX Patching os.name to 'posix' broke Path.home() on Windows (pathlib mixed POSIX home resolution with Windows drive parsing). Capture both exec branches (os.execve + subprocess.run) instead, and guard the POSIX permission-bit assertions behind os.name == 'posix' (Windows files report 0o666). * feat: add LaTeX (.tex, .bib) to readable and prose extensions LaTeX source files and BibTeX bibliographies are prose-rich content that benefits from both palace mining and entity detection. Adds the two extensions to the two extension lists most relevant to them, each with a matching test. - ``mempalace/miner.py:READABLE_EXTENSIONS`` — ``.tex`` / ``.bib`` join the mining allowlist (parallel to the Swift/Kotlin PR #1368 and the PHP ecosystem PR #1819). - ``mempalace/entity_detector.py:PROSE_EXTENSIONS`` — ``.tex`` / ``.bib`` also join the *preferred* entity-detection bucket alongside ``.md`` / ``.rst`` / ``.csv``, NOT the broader code-file fallback. The reason ``PROSE_EXTENSIONS`` exists separately is documented in-code: programming-language files have lots of capitalized identifiers (class names, function names) that produce false-positive person matches. LaTeX/BibTeX don't have that problem — they're typesetting languages for prose documents. ``.bib`` in particular is almost entirely author names, one of the highest real-entity densities of any file type the detector scans. Tests follow the patterns established by the prior extension PRs: ``tests/test_miner.py::test_scan_project_includes_latex_files`` mirrors the Swift/Kotlin scan tests, and ``tests/test_entity_detector.py::test_scan_for_detection_includes_latex_prose`` mirrors ``test_scan_for_detection_finds_prose``. The existing ``test_prose_extensions`` was extended to assert the two new entries. Full env-cleared suite: 3216 passed, 20 skipped. ``ruff check .`` and ``ruff format --check .`` both clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA * docs(config): add storage backends configuration reference Establish guide/configuration.md as the canonical home for per-backend connection settings, with a compatibility table and connection-variable reference for the chroma, sqlite_exact, qdrant, and pgvector backends. remote-server.md already links Postgres + pgvector to /guide/configuration, but the page had no backend section; this populates that target. New backends add one table row plus a connection subsection, keeping README's compatibility table in sync rather than accreting a prose paragraph per backend. * docs(config): clarify backend selection vs configuration in table Rename the table's 'Select with' column to 'Configure with' and list each backend's primary connection knob, since a connection variable (e.g. MEMPALACE_QDRANT_URL) configures a backend but does not select it — selection is uniform via --backend / MEMPALACE_BACKEND, covered in the prose below the table. Also state the concrete MEMPALACE_QDRANT_TIMEOUT default (10.0s). * fix(docs): stop wide tables from clipping; slim backend table The custom theme set `.vp-doc table { overflow: hidden }` to clip its rounded corners, which also overrode VitePress's default `overflow-x: auto` — so any table wider than the content column was clipped with no way to scroll to the hidden columns (visible on the storage-backends table). Switch to `overflow-x: auto` so wide tables scroll, keeping the rounded corners. Also shorten the storage-backends table's two capability headers (Namespace isolation -> Namespaces, Lexical search -> Lexical) so the table fits the content column without needing the scrollbar. * fix(docs): make backend comparison table fit the content column Browser-validated the table layout across desktop (1280) and mobile (375): - Denser doc-table cell padding (8px 16px -> 8px 12px) so comparison tables fit the content column instead of needing a horizontal scrollbar. - `overflow-wrap: break-word` on table-cell code so only genuinely long values (e.g. a Postgres DSN) wrap, while short identifiers like `palace_path` keep natural column sizing and stay on one line. - Drop the redundant 'Configure with' column from the storage-backends table (each backend's connection variables are documented in full in its own subsection right below) and shorten 'Local (exact cosine)' -> 'Local (exact)'. The comparison table is now five columns and fits cleanly. Verified no clipping and no page-level horizontal overflow on the configuration, remote-server, reference (cli/mcp-tools/python-api), claude-code, and knowledge-graph pages; wide tables scroll within their own container on mobile. * docs(openclaw): document full MCP tool surface * feat: add Milvus backend Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com> * fix: use native Milvus lexical search Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com> * fix: enable native Milvus Lite lexical search Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com> * fix: refine Milvus backend consistency Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com> * fix: address Milvus backend review feedback Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com> * fix: skip startup SQLite integrity check on oversized palace The MCP server ran PRAGMA quick_check on the full chroma.sqlite3 during startup, before answering the initialize handshake. quick_check is O(database size); on multi-GB palaces it exceeds the MCP client's ~30s connection timeout, so the server never finishes starting and the client drops the connection (observed >2min on a 4.6GB palace). Skip the startup probe when chroma.sqlite3 exceeds MEMPALACE_STARTUP_INTEGRITY_MAX_MB (default 512MB; 0 disables). The gate lives in _refresh_sqlite_integrity_status, the single choke point for the startup calls and every lazy consumer. `mempalace repair` preflight still runs the full quick_check via repair.sqlite_integrity_errors, so SQLite-layer corruption is still caught on the destructive path. Refs #1818. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jq495N7e2D4wY2Mp2AQvg7 * fix: auto-heal isolated FTS5 corruption in mine, not just repair mempalace mine aborts with the "ABORT: SQLite-layer corruption detected" banner on an isolated FTS5 inverted-index corruption -- the specific case maybe_autoheal_fts5_index already exists to fix in place. That helper is wired into cmd_repair's preflight, but not into mine's own post-mine validation (palace._validate_palace_fts5_after_mine), so mine forces a manual `mempalace repair` for a corruption class that's already safely self-healable. This wires the same auto-heal call into _validate_palace_fts5_after_mine, before it raises MineValidationError. maybe_autoheal_fts5_index returns the *remaining* errors after the heal attempt, so MineValidationError still raises whenever the corruption isn't the isolated, fully-healable case -- this only changes behavior when the heal has verifiably and fully cleared the corruption. Verified against a real-world repro (mining a real Claude Code project directory deterministically triggered this corruption after all files filed successfully, zero concurrency, single uninterrupted process): mempalace repair --yes confirmed the auto-heal path clears it before proceeding to a full rebuild. With this patch, mine self-heals the same case directly -- no abort, Files processed: <n>, Done, PRAGMA quick_check clean afterward. Test suite: 3214 passed, 20 skipped -- no new failures. Two pre-existing unrelated failures in test_repair.py (a SQLite-version-dependent FTS5 corruption message-wording mismatch, tracked separately) reproduce identically on unmodified develop. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: include checkpoint and delete_by_source in _MUTATING_TOOLS mempalace_checkpoint and mempalace_delete_by_source (added in 3.5.0) were missing from _MUTATING_TOOLS, so a server started with --read-only / MEMPALACE_MCP_READ_ONLY=1 still allowed writing drawers and bulk-deleting by source. The same gap affected the peer-writer lock gate, which uses the same frozenset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(repair): recognize newer SQLite FTS5 corruption message wording _errors_are_isolated_fts5 gated auto-heal on one specific message shape: malformed inverted index for FTS5 table SQLite >= ~3.5x (confirmed on 3.53.2 / Python 3.13.7) reports the same isolated-FTS5 condition with different wording instead: fts5: corruption found reading blob N from table "embedding_fulltext_search" The narrow regex never matched this phrasing, so maybe_autoheal_fts5_index silently declined to heal on any machine running a recent-enough SQLite, falling straight through to the hard-abort path -- the exact condition the whole auto-heal feature (#1926/#1928) exists to avoid. Widened the pattern to match either wording. Caught by running this repo's own test suite on this machine: test_repair.py's two auto-heal tests were failing (not, as assumed earlier, pre-existing/unrelated flakiness -- that assumption was never actually verified). Traced to this exact classification gap. Fixing this correctly also exposed that four tests in test_miner_fts5_validation.py had been passing for the wrong reason: they manufacture the exact "reporter-shaped" isolated-FTS5 corruption (#1926's actual bug shape) and asserted mine() must raise MineValidationError for it -- true only because the classifier bug prevented auto-heal from ever engaging. With the classifier fixed, that corruption is now correctly auto-healed and mine() succeeds instead, so those tests' expectations were stale, not their fixtures being invalid: - test_helper_raises_on_fts5_segment_corruption -> renamed test_helper_auto_heals_fts5_segment_corruption; asserts no raise + a clean post-heal quick_check, instead of expecting a raise. - test_full_chain_raises_through_mine_impl and test_mine_impl_does_not_print_partial_summary_on_validation_error: their real purpose is exception-passthrough / banner-suppression when the validator DOES raise, not proving any particular corruption triggers it. Switched from real file corruption to a monkeypatched raise. (Tried swapping to _page_mangle's non-isolated corruption first -- that made ChromaDB's own Rust bindings panic just opening the file for the re-mine's get_collection() call, a native crash rather than a catchable Python exception, before the validator ever ran. Different failure mode than what these tests are about, and not reliable to depend on.) - test_mine_formats_full_chain_raises_when_fts5_corrupt: same fix, mirrors the miner-path change for the extract path. - Added test_full_chain_auto_heals_isolated_fts5_corruption and test_mine_formats_full_chain_auto_heals_isolated_fts5_corruption as companions, proving the full mine()/mine_formats() chain -- not just the standalone validator -- actually auto-heals and succeeds end-to-end for the isolated case now that it's correctly classified. - test_errors_are_isolated_fts5_classification: added the new message wording as an explicit regression fixture (pinned literally, not dependent on whatever this machine's SQLite happens to emit). Full suite: 3302 passed, 20 skipped, 0 failed -- first fully clean run this session. ruff check / ruff format -- clean. * fix: sanitize embedded NUL bytes before they reach ChromaDB Fixes #1927.\n\nVerified locally on Windows with targeted NUL/surrogate/miner/FTS5 tests plus ruff check and ruff format --check. * fix: half-open as-of interval for KG supersession Fixes #1913.\n\nVerified locally on Windows with focused knowledge graph/MCP KG tests plus ruff check and ruff format --check. * fix: keep status from taking writer lease * fix(repair): use os.rename for in-place archive, not shutil.move shutil.move's fallback for a failed os.rename is copytree + rmtree. On Windows, when any file inside the palace is held open by another process (a live MCP server, a running mine, another harness), the rename fails and shutil.move falls back to deleting the live palace file-by-file via rmtree -- which itself then fails partway through on the first locked file, leaving the palace partially gutted next to a partial (or empty) archive copy. Reproduced live twice (Windows 11, 2026-07-05 and 2026-07-06): running `mempalace repair --mode from-sqlite --yes --archive-existing` while an MCP server / detached mine held palace/*/data_level0.bin open threw mid-rmtree in both cases. The palace itself survived only because the specific locked files could not be unlinked -- a different lock pattern (e.g. a lock on a file rmtree reaches first) would have lost data with no way back. os.rename is atomic on both platforms it matters on (POSIX rename(2), Windows MoveFileEx) -- it either fully succeeds or fails without touching anything. Catch the failure and abort cleanly with actionable guidance instead of a raw traceback. * fix(mcp): mark sqlite_integrity not-applicable on non-chroma backends (#1931) mempalace_status reported a passing SQLite integrity check on non-chroma backends (checked/ok true, sqlite_path pointing at a chroma.sqlite3 that does not exist) even though _refresh_sqlite_integrity_status short-circuits the check there. _sqlite_integrity_payload now reports the check as not-applicable (checked false, ok null, reason) for non-chroma backends, keeping the chroma payload shape and error surfacing unchanged. Co-Authored-By: Zoz92 <66385795+Zoz92@users.noreply.github.com> * fix(mine): address review feedback on FTS5 auto-heal (#1928) Two review comments on this PR, both addressed: - gemini-code-assist flagged that maybe_autoheal_fts5_index's default progress=print goes straight to stdout. _validate_palace_fts5_after_mine runs inside the MCP server process too (mcp_server.tool_mine -> miner.mine), where stdout is the JSON-RPC transport -- a stray print() there would corrupt the protocol stream and crash the connection. Pass progress=logger.info instead; palace.py already has the module logger. - nikkunikku corroborated the fix from a real 1.4GB production palace (278 repeated abort-loop iterations before the fix) and pointed out a real test gap: the fixture-based auto-heal tests fabricate real FTS5 corruption via direct shadow-table writes, which some SQLite builds refuse (existing pytest.skip paths in test_miner_fts5_validation.py, related to #1925) -- so on those builds the auto-heal wiring in _validate_palace_fts5_after_mine is never actually exercised. Added their suggested build-independent tests, adapted to this file's fixture helpers: test_validator_suppresses_raise_when_autoheal_clears and test_validator_still_raises_when_autoheal_cannot_clear, stubbing mempalace.repair.sqlite_integrity_errors/maybe_autoheal_fts5_index directly instead of fabricating corruption. Added a third test, test_validator_passes_logger_progress_not_print_to_autoheal, covering the specific progress= wiring: the two tests above mock maybe_autoheal_fts5_index entirely and discard its kwargs, so neither would have caught the progress=print regression this commit actually fixes. The new test captures the real kwargs and asserts progress is a bound method of palace.py's own logger (not print), without pinning to logger.info specifically -- severity level is a verbosity choice, not a correctness requirement, so the assertion shouldn't fail on a reasonable future change to e.g. logger.debug. Verified both directions: fails against the pre-fix `print` default (and shows the leaked stdout line to prove it), passes at .info and at .debug alike. Full suite: 3221 passed, 20 skipped (unchanged skip count). ruff check/format clean. * feat: add exclude_patterns config key to mempalace.yaml Allow projects to specify .gitignore-style patterns that the miner should skip, without relying on .gitignore for mining control. A new optional exclude_patterns list in mempalace.yaml is parsed by the existing GitignoreMatcher class via a new from_patterns() classmethod — same syntax, same semantics as .gitignore, no new dependency. exclude_patterns: - '*.md' - '*.yaml' - 'docs/' # dir-only: prunes entire tree without descending - 'dist/' - 'coverage/' Key behaviour: - Patterns follow .gitignore rules: anchoring (/pattern), dir-only - dirs[:] pruning via GitignoreMatcher.matches(..., is_dir=True) so excluded subtrees are never walked - Checked after .gitignore filtering; force_include (--include-ignored) bypasses exclude_patterns - Pre-scanned files lists (init double-scan optimisation) are filtered too - Backwards compatible: omitting exclude_patterns changes nothing Changes: - GitignoreMatcher.from_patterns(): new classmethod, same rule parser as from_dir(), reads from a list instead of a file on disk - scan_project(): builds one exclude_matcher before os.walk; used for both dirs[:] pruning and per-file filtering - _mine_impl(): applies the exclude matcher to pre-scanned files lists when provided by the caller - tests/test_miner.py: three new tests test_scan_project_exclude_patterns_skips_matching_files test_scan_project_exclude_patterns_prunes_entire_directory test_scan_project_exclude_patterns_include_ignored_bypasses_exclusion * refactor(miner): extract exclude_patterns prescan filter into a helper Rebasing Lochness's exclude_patterns work (#1213) onto current develop pushed _mine_impl's cyclomatic complexity to 26, tripping the repo's max-complexity=25 ruff gate (clean on develop before this rebase). Extracted the pre-scanned-file-list filtering branch into _apply_exclude_patterns_to_prescanned_files -- same behavior, no test changes needed, complexity back under the gate. * fix(convo_miner): treat transcripts as mutable, not immutable Conversation transcripts were assumed immutable once mined: the bulk skip-check (prefetch_mined_set) only tracked "have we seen this source_file before at the current normalize_version", with no mtime comparison at all. That's wrong for how Claude Code sessions actually work -- a session keeps appending to its own JSONL file while active, and /compact or /clear can rewrite one in place. Once a session file was mined, any content appended after that point would silently never get mined, with no error or warning -- the file just looked "already filed" forever. palace.py: - prefetch_mined_set() now returns dict[source_file, stored_mtime] instead of a bare set[source_file]. `if src in mined_set` still works identically (dict `in` checks keys), so this is a source-compatible change for that access pattern; a caller that wants staleness detection reads mined_set[src] and compares against the file's current mtime itself. None means no mtime was ever stored (or getmtime failed when the drawer was written) and must be treated as stale, not "unknown, assume unchanged". - Removed bulk_check_mined(): it already existed for exactly this purpose (bulk mtime prefetch) but had zero callers anywhere in the codebase and was missing the normalize_version/extract_mode filtering prefetch_mined_set has -- folded its intent into prefetch_mined_set instead of maintaining two subtly-different, overlapping bulk scans over the same underlying data. - file_already_mined()'s docstring corrected: it previously claimed "transcripts are assumed immutable" for convo mining. That's no longer true; corrected to describe the actual current split (convo miner's bulk skip-check uses prefetch_mined_set's stored mtimes; this function's check_mtime=True path is now only its per-file, lock-held race-condition recheck). convo_miner.py: - New _is_unchanged_since_last_mine() helper (extracted to keep _mine_convos_impl under the repo's cyclomatic-complexity gate): false whenever the file isn't in the prefetched map, its stored mtime is None, getmtime fails, or the mtimes don't match -- true only when genuinely unchanged. - _file_chunks_locked's metadata now stamps source_mtime on every real drawer (mirroring miner.py's existing pattern), and its in-lock recheck now passes check_mtime=True. - _register_file's 0-chunk sentinel also stamps source_mtime, so a short file that later grows past MIN_CHUNK_SIZE is detected as changed instead of being skipped forever by the sentinel. One-time cost worth flagging: no existing convo drawer has source_mtime stored (this field never existed for convo mining before now), so the first `mempalace mine --mode convos` after this ships will see every already-mined file as stale and fully re-mine it. Not a bug -- _file_chunks_locked's existing purge-before-insert means no duplication results -- just a real, one-time cost across a large corpus. tests/test_convo_miner.py: 7 new tests -- grown-file re-mine picks up new content, unchanged file still skipped (the mtime check must not regress the existing optimization), grown-file re-mine purges stale drawers rather than accumulating duplicates (checked via unique content markers, not raw counts -- ChromaDB collections can carry unrelated bookkeeping rows), prefetch_mined_set's returned mtime matches the real file, None handling for a drawer with no stored mtime, a legacy drawer (no source_mtime field, simulating pre-this-change data) is correctly re-mined rather than skipped forever, and the sentinel path stamps source_mtime too. Full suite: 3327 passed, 20 skipped, 0 failed. ruff check / ruff format -- clean. * fix(mcp): self-heal writer lease instead of latching read-only for life The #1818 peer-writer guard latched _MCP_WRITER_READ_ONLY=True on the first MineAlreadyRunning and short-circuited every subsequent acquisition attempt, so a server that came up read-only (a peer held the per-palace flock at startup) stayed read-only for its entire process lifetime — even long after the peer exited and the OS released the flock. In the common case of several overlapping Claude sessions (one server per session, all on the same palace), whichever session started second was stranded: mutating tools kept refusing with -32001 and the only remedy was killing/restarting that server. _mcp_peer_writer_refusal already calls _acquire_mcp_writer_lock() on every mutating tool, so the retry hook existed — the sticky latch just suppressed it. Drop the read-only short-circuit: when read-only we now re-attempt the non-blocking flock each call and transparently promote to writer once the peer is gone. Race-safe — fcntl LOCK_NB is kernel-arbitrated, so two servers can never both win. The genuinely-broken-lock path (_MCP_WRITER_LOCK_FAILED) is still cached, since retrying a broken lock mechanism can't help. Adds test_peer_writer_readonly_self_heals_after_peer_exits. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(tests): satisfy ruff format check for peer-writer self-heal test PR #1960 merged with a red `lint` job: `ruff format --check .` wanted to collapse the multi-line `MineAlreadyRunning(...)` raise in the new `test_peer_writer_readonly_self_heals_after_peer_exits` onto one line (it fits the line-length limit). All six real test jobs passed; only the formatter check failed, which left `develop` red on lint. Reformat that one statement so `ruff format --check .` is clean again. No logic change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: retrigger CI (unrelated test-windows flake on prior run) * fix(mcp): answer initialize immediately — run startup preflight in a background thread The stdio loop ran _refresh_sqlite_integrity_status() and _refresh_vector_disabled_flag() before reading the first request. PRAGMA quick_check reads every page of chroma.sqlite3, so on multi-GB palaces the probe alone (measured: 20.3s on a 1.72 GB / 326k-drawer palace, 40-46s under disk/lock contention) starves the MCP client's 60s connect timeout — even though the initialize response itself never touches the database. The HTTP transport already starts without the synchronous probe. Move both probes to a daemon thread (mcp-startup-preflight). The #1222 intent is preserved: the probe still starts at startup and logs its warning as soon as it finishes. Consumers that need the verdict (_ensure_sqlite_integrity_status via the tool-call integrity gate) serialize on a new _sqlite_integrity_refresh_lock with double-checked locking, so a tool call arriving mid-probe waits for the in-flight verdict instead of running a second O(database size) quick_check — and never proceeds unverified. Measured on the 1.72 GB palace with the >512 MB startup gate disabled (MEMPALACE_STARTUP_INTEGRITY_MAX_MB=0, full quick_check in flight): initialize 1.4s (was 20-46s); first tool call after probe completion 3.4s with sqlite_integrity checked=true ok=true. Complements c54531a: the oversized-palace skip still applies to the background probe, but the handshake no longer depends on it. * style: ruff format test file * fix(palace): pair the mine_palace_lock holder-set update with its release _mark_held(palace_key) ran before the try: whose finally runs _mark_released(). An async exception (SIGINT/KeyboardInterrupt) landing after _mark_held() and before the try: skips _mark_released(), stranding the key in the process-wide _palace_lock_keys set while the outer finally frees the flock. The in-memory hold then outlives the OS lock: a later re-entrant acquire passes through and writes without the flock while another process can acquire it, i.e. two writers into one palace. Move _…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds
mempalace repair rebuild-indexas a CLI alias for rebuilding a palace index after changingMEMPALACE_EMBEDDING_MODEL.The alias routes to the existing SQLite rebuild path with archive enabled, so the repair command avoids opening the old Chroma collection with the new embedding function. The embedding mismatch hint now points to the working command and quotes the palace path safely.
Related to #1663.
How to test
.venv/bin/python -m pytest tests/test_cli.py -q.venv/bin/python -m pytest tests/test_backends.py::test_get_collection_translates_ef_mismatch_to_helpful_error -q.venv/bin/ruff check mempalace/cli.py mempalace/backends/chroma.py tests/test_cli.py tests/test_backends.pyChecklist