feat(miner): add support for Swift and Kotlin file extensions - #1368
Merged
igorls merged 1 commit intoJun 18, 2026
Conversation
- Updated READABLE_EXTENSIONS in miner.py to include ".swift", ".kt", and ".kts". - Added tests in test_miner.py to ensure scanning includes Swift and Kotlin files.
igorls
added
enhancement
Contributor
This was referenced Jun 22, 2026
Merged
4 tasks
igorls
pushed a commit
that referenced
this pull request
Jun 30, 2026
LaTeX source files and BibTeX bibliographies are prose-rich content that benefits from both palace mining and entity detection. Adds the two extensions to the two extension lists most relevant to them, each with a matching test. - ``mempalace/miner.py:READABLE_EXTENSIONS`` — ``.tex`` / ``.bib`` join the mining allowlist (parallel to the Swift/Kotlin PR #1368 and the PHP ecosystem PR #1819). - ``mempalace/entity_detector.py:PROSE_EXTENSIONS`` — ``.tex`` / ``.bib`` also join the *preferred* entity-detection bucket alongside ``.md`` / ``.rst`` / ``.csv``, NOT the broader code-file fallback. The reason ``PROSE_EXTENSIONS`` exists separately is documented in-code: programming-language files have lots of capitalized identifiers (class names, function names) that produce false-positive person matches. LaTeX/BibTeX don't have that problem — they're typesetting languages for prose documents. ``.bib`` in particular is almost entirely author names, one of the highest real-entity densities of any file type the detector scans. Tests follow the patterns established by the prior extension PRs: ``tests/test_miner.py::test_scan_project_includes_latex_files`` mirrors the Swift/Kotlin scan tests, and ``tests/test_entity_detector.py::test_scan_for_detection_includes_latex_prose`` mirrors ``test_scan_for_detection_finds_prose``. The existing ``test_prose_extensions`` was extended to assert the two new entries. Full env-cleared suite: 3216 passed, 20 skipped. ``ruff check .`` and ``ruff format --check .`` both clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
jphein
added a commit
to techempower-org/mempalace
that referenced
this pull request
Aug 9, 2026
#394) * fix(mine): route SKIP to stderr and cover stat() OSError arm (#923) The original commit printed SKIP for oversized files to stdout but the sibling SKIP for symlinks in the same scan_project / scan_convos already went to stderr. Align the new line with that convention. Also adds a SKIP-with-error log for the except OSError arm right below the size check. Files whose stat() raises (permission denied, racing delete, broken symlink that survived the earlier is_symlink check) were the same bug class as the silent oversize drop. Tests switched from captured.out to .err and tightened to the full template; new test covers the OSError arm via a selective Path.stat monkeypatch with a follow_symlinks gate for Python 3.10+. * fix: spawn daemon with CREATE_NO_WINDOW to match hook miner (#1783) (#1857) daemon.py:_detached_kwargs was the last production spawn site still using DETACHED_PROCESS. Swap it to CREATE_NO_WINDOW, matching the hook miner's _detached_popen_kwargs fixed in #1848 — the dedicated follow-up the review bot asked for. `grep -rn DETACHED_PROCESS mempalace/` now returns zero production hits. Survivability is unchanged: CREATE_BREAKAWAY_FROM_JOB (escapes the parent Job Object's kill-on-close) plus the daemon never being attached to the launching console carry survive-terminal-close; CREATE_NEW_PROCESS_GROUP (also kept) isolates Ctrl-C/Break. CREATE_NO_WINDOW is ignored when OR'd with DETACHED_PROCESS, so this replaces the flag rather than adding it. The daemon already redirects stdout/stderr to daemon.log and reads no stdin, so it needs no console. Adds the first tests for _detached_kwargs (posix + windows, cross-platform monkeypatch of the Windows-only flag constants, mirroring the #1848 hooks_cli tests). * fix(cli): add repair rebuild-index alias (#1670) * Fix/wing slug special chars (#1852) * fix: sanitize wing slug for project dirs with special characters Project folders containing characters outside sanitize_name's set (e.g. a leading '+') leaked into the derived wing name, producing names like 'wing_+project' that config.sanitize_name rejects, silently breaking diary auto-save for that project. Add _safe_wing_slug(): collapse non-word runs to '_', trim, and fall back to 'sessions' when a name reduces to nothing. Route the three wing-derivation sites through it. Tests: unit cases for the helper plus a hypothesis property test asserting wing_<slug> always passes sanitize_name for any input. * fix: preserve dots and apostrophes in wing slug for backward compatibility The first pass collapsed every non-word character (including dot and apostrophe) to underscore, renaming existing valid wings — e.g. my.app became wing_my_app — which would orphan diary entries already filed under the old name. Keep dot and apostrophe (both accepted by sanitize_name), collapse consecutive dots to avoid the path-traversal rejection, and trim edge separators. Add backward-compatibility tests for previously-valid names plus a double-dot collapse test. * fix: cap wing slug length to stay within sanitize_name's limit sanitize_name rejects names over 128 characters, so a very long project directory name would produce a wing name that fails validation, re-triggering the silent auto-save break this PR fixes. Truncate the slug to 120 chars (the wing_ prefix keeps the total under 128). Widen the hypothesis property test to max_size=300 so it exercises the length path, and add an explicit truncation test. Addresses gemini-code-assist review feedback on PR #1852. --------- Co-authored-by: Ivan Antsimonau <ivan.antsimonau@katim.com> * fix(hooks): hide conhost window on Windows in _mine_sync non-daemon path (#1863) The non-daemon synchronous mine fallback in _mine_sync() spawned the mine subprocess without CREATE_NO_WINDOW, flashing a visible console window on every PreCompact fire on Windows. The async paths (_spawn_mine, _desktop_toast) already pass it via _detached_popen_kwargs(); this sync path was missed. getattr(..., 0) is a no-op off-Windows. Fixes #1862 Co-authored-by: David Finkelstein <david@finkelstein.us> * fix: expand tilde in palace_path when read from config file\n\nMempalaceConfig.palace_path correctly called os.path.expanduser() for\nenv-var paths but not for paths read from config.json. If config.json\nstores palace_path as '~/.mempalace/palace' (the default written by\ninit), the tilde was returned unexpanded.\n\nDownstream callers such as cli.py cmd_mine did call expanduser when\n--palace was passed explicitly, but fell through to MempalaceConfig()\nwhen no flag was given, inheriting the unexpanded string. Python's\nos.makedirs and chromadb.PersistentClient treat a leading tilde as a\nliteral directory name rather than the home directory, so the palace\nwas silently written to a CWD-relative path such as\nmy_project/~/.mempalace/palace.\n\nThe fix is a single os.path.expanduser() call on line 343 of\nconfig.py, mirroring the existing env-var branch on line 342. Since\nDEFAULT_PALACE_PATH is already expanded at module load (line 197),\nexpanduser on an absolute path is a no-op, so the default case is\nunaffected.\n\nSymptoms: scattered {project}/~/.mempalace/palace directories, palace\nalways appears empty after mine, search returns Collection does not\nexist, launchd-driven nightly mine writes to a different location than\ninteractive mine.\n\nCo-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>n (#1865) * fix(chroma): stop quarantining valid all-layer-0 HNSW segments (#1716) An empty link_lists.bin is not corruption on its own: hnswlib stores the layer-0 graph inside data_level0.bin and only writes link_lists.bin for elements promoted to level > 0. A small/low-fanout index where every element stays on layer 0 serializes an empty link_lists.bin and loads fine. Flagging that shape as corrupt produced a self-perpetuating quarantine loop — repair rebuilt the byte-identical all-layer-0 segment, the next cold start re-quarantined it, accumulating drift dirs (221 MB in the reported case) with no ingestion involved. Use the persist-completion marker as the discriminator instead. ChromaDB writes index_metadata.pickle last, so an intact pickle envelope proves the flush finished and the empty link_lists.bin is the legitimate all-layer-0 shape. Only treat an empty link_lists.bin as a partial flush when there is real payload AND no completion marker (absent or truncated pickle). The #1457 partial-flush protection (real payload, no/truncated marker) is preserved; the byte-sniff is factored into _hnsw_metadata_marker_intact and reused by _segment_appears_healthy. Also fixes the related single-writer stale-quarantine false positive (#1564), which shares this all-layer-0 root cause. * fix(repair): auto-heal isolated FTS5 inverted-index corruption (#1596) Concurrent killed-mid-write mines can leave embedding_fulltext_search in a malformed-inverted-index state that fails PRAGMA quick_check while the underlying rows stay intact (integrity_check ok). The repair preflight then hard-aborts before reaching the FTS5 rebuild step, so `mempalace repair` refuses to run and full-text search stays broken — the exact loop #1596 reports. The MineValidationError banner even promises "repair --yes rebuilds the FTS5 virtual table automatically," which the preflight abort made false. Add maybe_autoheal_fts5_index(): when every quick_check error is an isolated "malformed inverted index for FTS5 table" failure, rebuild the index in place from the intact embedding_fulltext_search_content table (INSERT ... VALUES('rebuild')) under mine_palace_lock, then re-run quick_check. The rebuild touches no drawer rows. Wired into both repair preflights (rebuild_index and cli cmd_repair). Any non-FTS5 error in the set, a lock held by a live mine, or a rebuild that does not clear quick_check leaves the errors unchanged so the caller still aborts with the recovery banner — broader corruption is never silently rebuilt over. * fix(mcp): stop clobbering host app root logger at import (#1860) (#1885) * fix(mcp): stop clobbering host app root logger at import (#1860) _init_logging() ran at import and called logging.basicConfig(force=True), resetting the root logger's level, format, and handlers unconditionally. An app that configured logging before importing mempalace.mcp_server lost its setup: a host on DEBUG dropped to INFO, custom formatters and handlers were replaced. force=True existed (#1495) only to keep MEMPALACE_LOG_FILE working when root already had handlers. This keeps that contract without the reset: configure root only when it is unconfigured (standalone); otherwise attach a mempalace-filtered file handler additively and leave the host's config alone. Adds _MempalaceLogFilter so the file captures every mempalace logger (the dotted mempalace.* family plus the flat mempalace_* names) and nothing else. * fix(mcp): survive importlib.reload and pin file log format (#1860) Addresses review on #1885. Restore _logging_configured from globals() so the idempotency guard survives importlib.reload: a reload re-executes the module body, and a plain reset would let _init_logging() stack a duplicate file handler on root. Set an explicit "%(message)s" formatter on the file handler so the embedded path does not depend on logging's default formatter (which already renders the same, but is now pinned and identical to the standalone path). Adds a reload regression test and a format-pin assertion. * fix(layers): order L1 wake-up by recency so it surfaces the latest moments (#1630) L1's generate() scored drawers by importance/emotional_weight/weight, and the docstring promised "prefer high importance, recent filing". But no ingest path (miner, convo_miner, diary, add_drawer) writes any of those fields, so the sort collapsed to insertion order (oldest first) and recency was never consulted. A scoped `wake-up --wing X` therefore surfaced the *oldest* moments: the opposite of useful. Add filed_at (present on every drawer, ISO-8601, lexically chronological) as the secondary sort key. Importance stays primary for the day a scoring pass populates it; filed_at is the effective signal today, making the "recent filing" half of the promise true with data already present. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> * fix(cli): force UTF-8 when reading/writing .gitignore in init (#1648) On Windows, Path.read_text() and open(path, 'a') use locale encoding (GBK on Chinese-locale systems) before PEP 686 / Python 3.15. A valid UTF-8 .gitignore with non-ASCII comments crashes _ensure_mempalace_files_gitignored() with UnicodeDecodeError, which aborts 'mempalace init' on Windows for any user whose .gitignore contains non-ASCII text. Force encoding='utf-8' on both read and append, with errors='replace' on read as a defensive fallback for legacy mixed-encoding files. Co-authored-by: ALaDingAhmad <16530935@qq.com> Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> * chore(deps): bump actions/checkout from 6 to 7 (#1882) Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v6...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump docker/setup-qemu-action from 3 to 4 (#1880) Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 3 to 4. - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](https://github.com/docker/setup-qemu-action/compare/v3...v4) --- updated-dependencies: - dependency-name: docker/setup-qemu-action dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump docker/setup-buildx-action from 3 to 4 (#1881) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3 to 4. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](https://github.com/docker/setup-buildx-action/compare/v3...v4) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps-dev): bump ruff from 0.15.18 to 0.15.20 (#1883) Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.18 to 0.15.20. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/ruff/compare/0.15.18...0.15.20) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.15.20 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(backends): require SQLite magic header for chroma + sqlite_exact detect() (#1893) (#1896) * fix(chroma): require SQLite magic header for ChromaBackend.detect() (#1893) Closes #1893. ChromaBackend.detect() was returning True for a 0-byte chroma.sqlite3 file because the check was just os.path.isfile(...). On a palace that has any other backend marker alongside a stale 0-byte chroma.sqlite3, resolve_backend_name then raises BackendMismatchError and the palace becomes unopenable until the user manually rm's the empty file. The 0-byte file appears as a side effect of any sqlite3.connect() on a missing path — Python creates the file immediately but writes the SQLite header only on the first statement. So any code path that touches the chroma.sqlite3 path with bare sqlite3.connect(), including chromadb's own PersistentClient lazy-init (see the comment at backends/chroma.py:2052), can leave a 0-byte artifact behind. Fix: detect() now reads the first 16 bytes and compares to the SQLite magic prefix b"SQLite format 3\x00" instead of relying on file presence alone. One extra open() + 16-byte read; detect() isn't a hot path. Properties: - Rejects 0-byte files (the symptom #1893 is about). - Rejects non-SQLite garbage at the canonical path (partial writes, etc.). - Doesn't false-negative on real chroma palaces: any chroma palace whose PersistentClient has done any work has the magic header on disk (verified — CREATE TABLE is enough to land the header). - Doesn't couple detect() to chroma's specific schema; the magic header is stable across chromadb releases. Test sweep: many test files used (chroma.sqlite3).touch() or .write_bytes(b"") as a "fake palace" shortcut, exploiting the loose isfile() check (one such site even had the comment "# pass the isfile guard"). After this change, those stand-ins no longer register as chroma palaces. Introduced tests/_chroma_palace_helper.py::make_minimal_chroma_sqlite following the existing _backend_conformance.py precedent, and updated 15 call sites across 8 test files to use it. The existing test_chroma_detect_matches_palace_with_chroma_sqlite (which encoded the buggy semantics with write_bytes(b"")) is renamed to test_chroma_detect_matches_palace_with_sqlite_header and now writes a real SQLite database via the helper. Added two new tests for the rejection paths (empty file, non-SQLite garbage). Full env-cleared suite: 3137 passed, 20 skipped, 0 failed. ruff check and ruff format --check both clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA * fix(sqlite_exact): require SQLite magic header for SQLiteExactBackend.detect() Per gemini-code-assist review on #1892 PR #1896: SQLiteExactBackend has the same os.path.isfile() detection pattern as ChromaBackend did, with the same 0-byte-file vulnerability. Mirrors the chroma fix for repo-wide consistency. - SQLiteExactBackend.detect() now does the same 16-byte SQLite magic-prefix check as ChromaBackend.detect(). - _chroma_palace_helper.py: factored its body into a private _write_minimal_sqlite_file() and gained a sibling make_minimal_sqlite_exact_sqlite() for the sqlite_exact filename. No churn to any existing chroma call sites. - test_sqlite_exact_backend.py:426 (the one site that wrote b"" for sqlite_exact.sqlite3) updated to use the new helper. - Three new tests in test_sqlite_exact_backend.py mirror the chroma trio: matches with valid header, rejects empty file, rejects non-SQLite garbage. Full env-cleared suite: 3140 passed, 20 skipped, 0 failed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * fix(pgvector): skip document column for metadata-only fetches (#1840 follow-up) (#1892) * fix(pgvector): skip document column for metadata-only fetches (#1840 follow-up) Closes the explicit "separate follow-up to keep this low-risk" callout in PR #1840's description. For remote pgvector deployments (TLS over WAN), `mempalace_status` and every other metadata-only consumer was transferring the full `document` column over the wire even when nothing read it. A single scroll over a 177K-drawer palace on a 175 ms-RTT link moved ~150 MB of document text plus ~50 MB of metadata; this PR drops that to ~50 MB. scroll_rows / _scroll gain `with_document: bool = True`. When False, SELECT projects NULL::text instead of the document column. Positional _row parser unchanged (record[1] stays the document slot, just receives NULL). Existing callers default to True and see byte-for-byte identical behavior. PgVectorCollection.get_all_metadata override: where=None path goes single-scroll with with_document=False. Filtered path falls back to base to keep _matches_where running on array/object metadata values (same correctness contract as #1840's filtered-path decision). Tests: - Update _FakePgVectorClient.scroll_rows to accept with_document; mirror the NULL-becomes-empty-string semantics when False - Update 5 existing scroll_calls assertions to include with_document=True (unchanged intent) - test_pgvector_get_all_metadata_skips_document_column: assert exactly one scroll call with with_document=False - test_pgvector_get_all_metadata_filtered_falls_back_to_base: assert filtered path preserves with_document=True Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA * fix(pgvector): extend with_document=False fast path to filtered get_all_metadata Per gemini-code-assist review feedback on #1892: _matches_where only reads metadata, so the where=None vs where=set conditional fall-back was unnecessary. The filtered path can use the same single-scroll with_document=False fast path and apply the post-filter locally on metadata dicts — extending the wire-byte win to every get_all_metadata caller, not just unfiltered ones. Mirrors the pushdown + local _matches_where pattern already used by _rows in the same file: pushdown when _requires_local_filter is False, post-filter in Python otherwise. Same correctness contract as #1840's filtered get path. Renames test_pgvector_get_all_metadata_filtered_falls_back_to_base to test_pgvector_get_all_metadata_filtered_uses_fast_path and asserts the new behavior (with_document=False + pushdown forwards the equality filter to SQL). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * feat(convo): preserve authored timestamp from transcripts (#1890) * feat(convo): preserve authored timestamp from transcripts Conversation drawers only carried `filed_at` (ingest time), so a bulk re-mine collapsed every drawer to a single instant and the chronological signal was lost — even though each Claude Code / Codex JSONL line already carries an ISO-8601 `timestamp`. The recency-window fallback and any date-aware consumer then saw ingest order, not when content was written. - convo_miner: derive `authored_at` (per-file max line `timestamp`) and store it as drawer metadata; falls back to `filed_at` when absent - searcher: surface `authored_at` in search results, and break exact hybrid-score ties toward the more recently authored drawer (ISO strings sort chronologically; missing dates sort oldest) — benchmark-neutral as it only reorders exact ties - tests: cover `_extract_authored_at` (latest wins, skips/tolerates lines without timestamps, non-jsonl/missing -> None) and the tie-break Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(search): surface authored_at in CLI + backfill for existing data Completes the authored_at work so the field is visible end-to-end and existing palaces can adopt it without re-mining. - layers: CLI `search` output shows an `authored:` date line per result (peer of the existing date; markdown drawers fall back to filed_at) - scripts/backfill_authored_at.py: in-place migration that stamps authored_at on convos drawers from their source transcripts — metadata only (no re-embedding), idempotent, dry-run by default - docs/authored-at.md: documents created_at (ingest) vs authored_at (written) and both backfill paths (in-place / drop-and-recreate) - tests: backfill integration tests over an ephemeral ChromaDB collection Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(search): address review — non-string timestamp guard + top-level authored_at tiebreak Two correctness fixes from the PR review: - _extract_authored_at: only compare when the parsed `timestamp` is a str. A non-string timestamp on a malformed/foreign JSONL line previously raised TypeError outside the try and could crash the mine. - _hybrid_rank: the tie-break read `authored_at` only from nested `metadata`, but the search_memories path (MCP / Claude Code) carries it at the top level of each hit — so the tie-break silently no-op'd there. Read both shapes. - tests: non-string timestamp cases, and a top-level-shape tie-break test (which fails before this fix). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: apply ruff format to authored_at changes CI ruff format --check flagged 4 files; ruff check already passed. Formatting only — no behavior change. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> * fix(palace): process-wide mine_palace_lock re-entrancy so the HTTP transport can write (#1859) * fix(palace): process-wide mine_palace_lock re-entrancy for threaded HTTP transport The MCP HTTP transport (ThreadingHTTPServer) acquires the long-lived writer-lease on one thread (_acquire_mcp_writer_lock) but dispatches each write request on a different worker thread. The lock re-entrancy guard was thread-local, so write handlers (add_drawer/update_drawer) failed to see the process-held lease, re-acquired the flock, and self-conflicted with "palace ... is held by PID <self>". Reads worked (no lock); writes over the HTTP transport were impossible. Make the re-entrancy record process-wide (pid-tagged, guarded by a threading.Lock) so a write from any thread of the process that already holds the lease passes through. Safe: flock is per-process and HTTP writes are serialized by _HTTP_REQUEST_LOCK. Preserves fork-safety, same-thread nesting (miner.mine -> ChromaCollection.upsert), and cross-process protection (MineAlreadyRunning still raised between processes). Add cross-thread same-process regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(palace): reset lock guard on fork to avoid inherited-locked deadlock Address review (PR #1859): `_palace_lock_guard` is a threading.Lock, so a child forked while another thread held it would inherit it locked (the holder thread is gone in the child) and deadlock on the next acquire. Register an os.register_at_fork(after_in_child=...) handler that replaces the guard with a fresh unlocked lock and clears state; the child must reacquire the flock anyway. Guarded by hasattr(os, "register_at_fork") for Windows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> * feat(mcp): add since/before date filter to list_drawers (#1128) (#1891) * feat(mcp): add since/before date filter to list_drawers (#1128) mempalace_list_drawers previously filtered only by wing/room. This adds optional since/before ISO date bounds on filed_at: since is inclusive, before is exclusive. The filter runs in Python after the rows are fetched. ChromaDB 1.5.7 rejects string operands for $gte/$lt and filed_at is stored as an ISO string, so a server-side where comparison is not available; the tool already collapses and paginates the full result set in Python. Drawers whose filed_at is missing or unparseable are excluded while a bound is active, and inverted bounds (since >= before) return a clear error. * test: close chromadb clients between tests to fix Windows handle leak (#1128) chromadb 1.5.7 caches one System per palace path and only frees the SQLite/HNSW file handles on client.close(); the collection fixture and the per-test MCP cache reset only dereferenced the client, so handles leaked across the session. Harmless on POSIX (rmtree unlinks open files), but on Windows the handles stay locked and accumulate until an HNSW segment write in a later test's setup fails, which surfaced here as TestDeleteBySource::test_commit_purges_matching_closets asserting 0 == 2. Close the client in the collection fixture and in _reset_mcp_cache so the handles are released between tests. * test: release backend chromadb clients between tests (#1128) palace.get_collection() caches one PersistentClient per palace_path on the process-wide backend singleton and never closes it; sweep, repair and several CLI tests reach the store through it. chromadb frees the rust-side SQLite/HNSW file handles only on client.close(), so the handles leak across the whole session: a 30-palace probe shows ~200 open file descriptors into the palace tree, dropping to 0 once the clients are closed. On POSIX the open handles are harmless (rmtree unlinks open files), but on Windows they stay locked and accumulate until a later test's HNSW segment write fails ("Failed to apply logs to the hnsw segment writer"), e.g. test_sweeper.py::TestSweeperTandem::test_sweep_recovers_untaken_message_at_cursor_timestamp. Drain the cached clients in the autouse _reset_mcp_cache teardown via close_palace(), which closes each PersistentClient (releasing its handles) without marking the backend closed so it stays reusable. Complements the collection-fixture and _client_cache close() added earlier. * feat: optimize metadata counting using Qdrant server-side facets (#1868) * feat: add metadata facet support for qdrant * added benchmark * updated benchmark * chore: remove tracking for local scratch benchmark * feat: add metadata facet support for qdrant -clean * Update mempalace/mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update mempalace/backends/qdrant.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update mempalace/backends/qdrant.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update tests/test_qdrant_backend.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update tests/test_qdrant_backend.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update tests/test_qdrant_backend.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update tests/test_mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * /fix always working tool_status() fallback fixed * /fix fallback added to tool_list_rooms * Update mempalace/mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update mempalace/mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update tests/test_qdrant_backend.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * /fix rebuilt the room populating logic * /add added temporary files for atomic transactions * Update mempalace/mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update mempalace/mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update tests/test_mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * /fix ai slop * /fix added default facet limit * Update tests/test_qdrant_backend.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * /fix added max workers pool * Update mempalace/mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update mempalace/mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update mempalace/backends/qdrant.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * /fix added clear() * Update tests/test_mcp_server.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * fix(qdrant): validate facet filter before existence check; fix taxonomy test - facet_counts now validates the where filter and rejects local-only filters before the _remote_exists() short-circuit, so an unsupported filter raises UnsupportedCapabilityError even on an unmaterialized collection (matches get()/lexical_search() ordering). - test_tool_get_taxonomy_uses_metadata_facets compared concurrent room facet calls via set(), but a call() with a dict kwarg is unhashable; compare order-independently via membership instead. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> * feat(graph): auto-populate the associative graph from mined sessions (#1895) * feat(graph): auto-populate the associative graph from mined sessions Conversation mining never set the `entities` drawer metadata that hallways consume, so mined sessions produced an empty associative graph (and starved the entity-navigation / tunnel-recommendation features built on top of it). Add a no-LLM structural entity extractor and wire it into the convos mine: - entities: structural-only extractor (author-quoted code spans, URLs, file paths, qualified identifiers, CamelCase / snake_case symbols). No wordlists, no NLP models, precision-biased so prose doesn't pollute the graph. - convo_miner: set `entities` per chunk, and compute hallways after a convos mine (mirroring the project-file path). Hallways run before the FTS5 validation, which opens a direct sqlite connection that can invalidate the live Chroma collection handle on some Chroma builds. - cli: `mempalace hallways` lists the associative graph (CLI parity with the list_hallways MCP tool). - tests: extractor precision/ranking, entities metadata at mine time, CLI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(graph): address review — semicolon safety, leading-underscore snake, negative limit - entities `_clean`: strip `;` out of tokens so a URL query string or backtick span can't split the `;`-joined entities metadata field - entities `_SNAKE`: optional leading/trailing `_?` so `_extract_authored_at` and similar are matched in plain text (previously only caught via backticks) - cli `hallways`: clamp `--limit` with max(0, ...) so a negative value shows nothing instead of slicing from the end - tests for all three Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> * docs(guide): add Remote / Team Server deployment guide (#1877) (#1897) Documents running MemPalace as a central memory service for a team: HTTP MCP transport (--transport http with bearer-token auth), a networked backend (Qdrant via REST, no extra dep; or pgvector), and optional GPU embedding. Covers the security model (non-loopback token requirement, Host/Origin DNS-rebinding guard, TLS-in-front), client connection, and operating notes. Adds the page to the guide sidebar. Addresses #1877. * fix(backends): forward facet_counts + get_all_metadata in EmbeddingCollection Both methods are concrete on ``BaseCollection`` (``facet_counts`` raises ``UnsupportedCapabilityError``; ``get_all_metadata`` pages through ``self.get(include=["metadatas"])``). Python MRO resolves them on ``EmbeddingCollection`` before ``__getattr__`` ever fires, so without an explicit forwarder the wrapper silently runs the base default instead of delegating to the wrapped backend's optimized implementation. The pattern matches the existing explicit forwarders for ``distance_metric``, ``lexical_search``, and the embedder-identity trio — all added to fix the same shadow. What this means in production for the three backends that get wrapped (``EmbeddingCollection`` only applies to ``requires_explicit_embeddings`` backends — qdrant, pgvector, sqlite_exact; chroma is unwrapped and unaffected): - **facet_counts shadow (#1868 regression)**: every ``mempalace_status``, ``list_wings``, ``list_rooms``, ``get_taxonomy`` call routes through the gated ``col.facet_counts(...)`` path. The capability check passes (``supports_metadata_facets`` is on the backend), but the call hits the wrapper's MRO-resolved ``BaseCollection.facet_counts`` and raises ``UnsupportedCapabilityError``. ``mcp_server``'s broad ``except`` swallows it, logs ``WARN Failed to fetch metadata facets, falling back to client- side loop: backend does not support facet_counts``, and counts via the O(n) Python loop — the exact behavior #1868 was designed to eliminate. - **get_all_metadata shadow (#1796 / #1892 regression)**: the BaseCollection default pages through ``self.get(include=["metadatas"])`` — fine for Chroma's SQL OFFSET cursor, but on wrapped backends (qdrant, pgvector) the inner's overridden ``get_all_metadata`` is unreachable. For pgvector specifically, this means #1892's ``with_document=False`` fast path is never taken even though it's implemented — every metadata-only fetch transfers the full document column over the wire. On a 13k-drawer remote pgvector palace over WAN that's ~13MB per call, dominating wall time. Why no test caught it: backend tests (``test_qdrant_backend.py``, ``test_pgvector_backend.py``) call the methods directly on the raw collection, not through the wrapper. ``test_mcp_server.py`` facet tests use ``MagicMock()`` for the collection, which synthesizes attributes on demand and bypasses MRO entirely. Neither path covers the seam where the bug lives: ``palace.get_collection() -> EmbeddingCollection -> .method()``. Three tests pin both the fix and the bug class: - ``test_facet_counts_forwards_to_inner`` — direct integration through the wrapper, asserts the inner's recorded call matches. - ``test_get_all_metadata_forwards_to_inner`` — same shape, plus a sentinel ``get()`` on the inner so a missing forwarder would route to the base default and pick up the wrong data (observable failure, not silent). - ``test_wrapper_forwards_all_concrete_basecollection_methods`` — meta-test that enumerates every concrete public method on ``BaseCollection`` via ``inspect.getmembers`` and asserts each one is explicitly defined on ``EmbeddingCollection``. Catches the bug *class*: any future ``BaseCollection`` method with a concrete default body becomes a CI failure the moment it's added without a wrapper forwarder, with a message pointing straight at the file to edit. Full env-cleared suite: 3205 passed, 20 skipped. ``ruff check`` and ``ruff format --check`` both clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA * fix(backends): annotate EmbeddingCollection.facet_counts return type Per Gemini review (PR #1898 comment r3489013681): the forwarder lacked the ``-> dict[str, int]`` return annotation that ``BaseCollection.facet_counts`` and the sibling ``get_all_metadata`` forwarder both carry. One-line consistency fix, no behavior change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA * feat(serve): turnkey secure remote MCP server (#1877) (#1900) * feat(serve): turnkey secure remote MCP server (#1877) Add `mempalace serve`: a secure-by-default wrapper over the HTTP MCP transport so a team can stand up a shared central palace with one command. Server capabilities (mempalace/mcp_server.py): - Native TLS via --tls-cert/--tls-key (env MEMPALACE_MCP_TLS_CERT/_KEY): wraps the socket in a TLS 1.2+ context, validated before bind. Token is still required on a non-loopback bind (TLS != auth). - Read-only mode via --read-only (env MEMPALACE_MCP_READ_ONLY): the 24 mutating tools are hidden from tools/list and refused at dispatch (-32003), enforced before arg handling — not merely hidden. Turnkey command (mempalace/cli.py): - Auto-generates a strong bearer token for non-loopback binds, stored 0600 under ~/.mempalace/server/ and printed once; reused across restarts. Token rides in the child env, never argv, so it can't leak via ps. - Prints a ready-to-paste client config (scheme reflects TLS), then foreground-execs the real server so Docker/systemd own the lifecycle. Deployment (deploy/): - docker-compose.server.yml wires the server + Qdrant with a /healthz healthcheck and persistent volumes. - server.env.example documents the env surface. - mempalace-server.service is a hardened systemd unit template. Tests: TLS handshake (openssl-gated), read-only enforcement, token autogen/0600/reuse, token-not-in-argv, secure-by-default gates. Docs: remote-server guide now leads with `mempalace serve` plus Compose and systemd subsections. * test(serve): fix Windows — don't patch os.name; gate 0600 asserts to POSIX Patching os.name to 'posix' broke Path.home() on Windows (pathlib mixed POSIX home resolution with Windows drive parsing). Capture both exec branches (os.execve + subprocess.run) instead, and guard the POSIX permission-bit assertions behind os.name == 'posix' (Windows files report 0o666). * feat: add LaTeX (.tex, .bib) to readable and prose extensions LaTeX source files and BibTeX bibliographies are prose-rich content that benefits from both palace mining and entity detection. Adds the two extensions to the two extension lists most relevant to them, each with a matching test. - ``mempalace/miner.py:READABLE_EXTENSIONS`` — ``.tex`` / ``.bib`` join the mining allowlist (parallel to the Swift/Kotlin PR #1368 and the PHP ecosystem PR #1819). - ``mempalace/entity_detector.py:PROSE_EXTENSIONS`` — ``.tex`` / ``.bib`` also join the *preferred* entity-detection bucket alongside ``.md`` / ``.rst`` / ``.csv``, NOT the broader code-file fallback. The reason ``PROSE_EXTENSIONS`` exists separately is documented in-code: programming-language files have lots of capitalized identifiers (class names, function names) that produce false-positive person matches. LaTeX/BibTeX don't have that problem — they're typesetting languages for prose documents. ``.bib`` in particular is almost entirely author names, one of the highest real-entity densities of any file type the detector scans. Tests follow the patterns established by the prior extension PRs: ``tests/test_miner.py::test_scan_project_includes_latex_files`` mirrors the Swift/Kotlin scan tests, and ``tests/test_entity_detector.py::test_scan_for_detection_includes_latex_prose`` mirrors ``test_scan_for_detection_finds_prose``. The existing ``test_prose_extensions`` was extended to assert the two new entries. Full env-cleared suite: 3216 passed, 20 skipped. ``ruff check .`` and ``ruff format --check .`` both clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA * docs(config): add storage backends configuration reference Establish guide/configuration.md as the canonical home for per-backend connection settings, with a compatibility table and connection-variable reference for the chroma, sqlite_exact, qdrant, and pgvector backends. remote-server.md already links Postgres + pgvector to /guide/configuration, but the page had no backend section; this populates that target. New backends add one table row plus a connection subsection, keeping README's compatibility table in sync rather than accreting a prose paragraph per backend. * docs(config): clarify backend selection vs configuration in table Rename the table's 'Select with' column to 'Configure with' and list each backend's primary connection knob, since a connection variable (e.g. MEMPALACE_QDRANT_URL) configures a backend but does not select it — selection is uniform via --backend / MEMPALACE_BACKEND, covered in the prose below the table. Also state the concrete MEMPALACE_QDRANT_TIMEOUT default (10.0s). * fix(docs): stop wide tables from clipping; slim backend table The custom theme set `.vp-doc table { overflow: hidden }` to clip its rounded corners, which also overrode VitePress's default `overflow-x: auto` — so any table wider than the content column was clipped with no way to scroll to the hidden columns (visible on the storage-backends table). Switch to `overflow-x: auto` so wide tables scroll, keeping the rounded corners. Also shorten the storage-backends table's two capability headers (Namespace isolation -> Namespaces, Lexical search -> Lexical) so the table fits the content column without needing the scrollbar. * fix(docs): make backend comparison table fit the content column Browser-validated the table layout across desktop (1280) and mobile (375): - Denser doc-table cell padding (8px 16px -> 8px 12px) so comparison tables fit the content column instead of needing a horizontal scrollbar. - `overflow-wrap: break-word` on table-cell code so only genuinely long values (e.g. a Postgres DSN) wrap, while short identifiers like `palace_path` keep natural column sizing and stay on one line. - Drop the redundant 'Configure with' column from the storage-backends table (each backend's connection variables are documented in full in its own subsection right below) and shorten 'Local (exact cosine)' -> 'Local (exact)'. The comparison table is now five columns and fits cleanly. Verified no clipping and no page-level horizontal overflow on the configuration, remote-server, reference (cli/mcp-tools/python-api), claude-code, and knowledge-graph pages; wide tables scroll within their own container on mobile. * docs(openclaw): document full MCP tool surface * feat: add Milvus backend Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com> * fix: use native Milvus lexical search Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com> * fix: enable native Milvus Lite lexical search Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com> * fix: refine Milvus backend consistency Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com> * fix: address Milvus backend review feedback Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com> * fix: skip startup SQLite integrity check on oversized palace The MCP server ran PRAGMA quick_check on the full chroma.sqlite3 during startup, before answering the initialize handshake. quick_check is O(database size); on multi-GB palaces it exceeds the MCP client's ~30s connection timeout, so the server never finishes starting and the client drops the connection (observed >2min on a 4.6GB palace). Skip the startup probe when chroma.sqlite3 exceeds MEMPALACE_STARTUP_INTEGRITY_MAX_MB (default 512MB; 0 disables). The gate lives in _refresh_sqlite_integrity_status, the single choke point for the startup calls and every lazy consumer. `mempalace repair` preflight still runs the full quick_check via repair.sqlite_integrity_errors, so SQLite-layer corruption is still caught on the destructive path. Refs #1818. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jq495N7e2D4wY2Mp2AQvg7 * fix: auto-heal isolated FTS5 corruption in mine, not just repair mempalace mine aborts with the "ABORT: SQLite-layer corruption detected" banner on an isolated FTS5 inverted-index corruption -- the specific case maybe_autoheal_fts5_index already exists to fix in place. That helper is wired into cmd_repair's preflight, but not into mine's own post-mine validation (palace._validate_palace_fts5_after_mine), so mine forces a manual `mempalace repair` for a corruption class that's already safely self-healable. This wires the same auto-heal call into _validate_palace_fts5_after_mine, before it raises MineValidationError. maybe_autoheal_fts5_index returns the *remaining* errors after the heal attempt, so MineValidationError still raises whenever the corruption isn't the isolated, fully-healable case -- this only changes behavior when the heal has verifiably and fully cleared the corruption. Verified against a real-world repro (mining a real Claude Code project directory deterministically triggered this corruption after all files filed successfully, zero concurrency, single uninterrupted process): mempalace repair --yes confirmed the auto-heal path clears it before proceeding to a full rebuild. With this patch, mine self-heals the same case directly -- no abort, Files processed: <n>, Done, PRAGMA quick_check clean afterward. Test suite: 3214 passed, 20 skipped -- no new failures. Two pre-existing unrelated failures in test_repair.py (a SQLite-version-dependent FTS5 corruption message-wording mismatch, tracked separately) reproduce identically on unmodified develop. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: include checkpoint and delete_by_source in _MUTATING_TOOLS mempalace_checkpoint and mempalace_delete_by_source (added in 3.5.0) were missing from _MUTATING_TOOLS, so a server started with --read-only / MEMPALACE_MCP_READ_ONLY=1 still allowed writing drawers and bulk-deleting by source. The same gap affected the peer-writer lock gate, which uses the same frozenset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(repair): recognize newer SQLite FTS5 corruption message wording _errors_are_isolated_fts5 gated auto-heal on one specific message shape: malformed inverted index for FTS5 table SQLite >= ~3.5x (confirmed on 3.53.2 / Python 3.13.7) reports the same isolated-FTS5 condition with different wording instead: fts5: corruption found reading blob N from table "embedding_fulltext_search" The narrow regex never matched this phrasing, so maybe_autoheal_fts5_index silently declined to heal on any machine running a recent-enough SQLite, falling straight through to the hard-abort path -- the exact condition the whole auto-heal feature (#1926/#1928) exists to avoid. Widened the pattern to match either wording. Caught by running this repo's own test suite on this machine: test_repair.py's two auto-heal tests were failing (not, as assumed earlier, pre-existing/unrelated flakiness -- that assumption was never actually verified). Traced to this exact classification gap. Fixing this correctly also exposed that four tests in test_miner_fts5_validation.py had been passing for the wrong reason: they manufacture the exact "reporter-shaped" isolated-FTS5 corruption (#1926's actual bug shape) and asserted mine() must raise MineValidationError for it -- true only because the classifier bug prevented auto-heal from ever engaging. With the classifier fixed, that corruption is now correctly auto-healed and mine() succeeds instead, so those tests' expectations were stale, not their fixtures being invalid: - test_helper_raises_on_fts5_segment_corruption -> renamed test_helper_auto_heals_fts5_segment_corruption; asserts no raise + a clean post-heal quick_check, instead of expecting a raise. - test_full_chain_raises_through_mine_impl and test_mine_impl_does_not_print_partial_summary_on_validation_error: their real purpose is exception-passthrough / banner-suppression when the validator DOES raise, not proving any particular corruption triggers it. Switched from real file corruption to a monkeypatched raise. (Tried swapping to _page_mangle's non-isolated corruption first -- that made ChromaDB's own Rust bindings panic just opening the file for the re-mine's get_collection() call, a native crash rather than a catchable Python exception, before the validator ever ran. Different failure mode than what these tests are about, and not reliable to depend on.) - test_mine_formats_full_chain_raises_when_fts5_corrupt: same fix, mirrors the miner-path change for the extract path. - Added test_full_chain_auto_heals_isolated_fts5_corruption and test_mine_formats_full_chain_auto_heals_isolated_fts5_corruption as companions, proving the full mine()/mine_formats() chain -- not just the standalone validator -- actually auto-heals and succeeds end-to-end for the isolated case now that it's correctly classified. - test_errors_are_isolated_fts5_classification: added the new message wording as an explicit regression fixture (pinned literally, not dependent on whatever this machine's SQLite happens to emit). Full suite: 3302 passed, 20 skipped, 0 failed -- first fully clean run this session. ruff check / ruff format -- clean. * fix: sanitize embedded NUL bytes before they reach ChromaDB Fixes #1927.\n\nVerified locally on Windows with targeted NUL/surrogate/miner/FTS5 tests plus ruff check and ruff format --check. * fix: half-open as-of interval for KG supersession Fixes #1913.\n\nVerified locally on Windows with focused knowledge graph/MCP KG tests plus ruff check and ruff format --check. * fix: keep status from taking writer lease * fix(repair): use os.rename for in-place archive, not shutil.move shutil.move's fallback for a failed os.rename is copytree + rmtree. On Windows, when any file inside the palace is held open by another process (a live MCP server, a running mine, another harness), the rename fails and shutil.move falls back to deleting the live palace file-by-file via rmtree -- which itself then fails partway through on the first locked file, leaving the palace partially gutted next to a partial (or empty) archive copy. Reproduced live twice (Windows 11, 2026-07-05 and 2026-07-06): running `mempalace repair --mode from-sqlite --yes --archive-existing` while an MCP server / detached mine held palace/*/data_level0.bin open threw mid-rmtree in both cases. The palace itself survived only because the specific locked files could not be unlinked -- a different lock pattern (e.g. a lock on a file rmtree reaches first) would have lost data with no way back. os.rename is atomic on both platforms it matters on (POSIX rename(2), Windows MoveFileEx) -- it either fully succeeds or fails without touching anything. Catch the failure and abort cleanly with actionable guidance instead of a raw traceback. * fix(mcp): mark sqlite_integrity not-applicable on non-chroma backends (#1931) mempalace_status reported a passing SQLite integrity check on non-chroma backends (checked/ok true, sqlite_path pointing at a chroma.sqlite3 that does not exist) even though _refresh_sqlite_integrity_status short-circuits the check there. _sqlite_integrity_payload now reports the check as not-applicable (checked false, ok null, reason) for non-chroma backends, keeping the chroma payload shape and error surfacing unchanged. Co-Authored-By: Zoz92 <66385795+Zoz92@users.noreply.github.com> * fix(mine): address review feedback on FTS5 auto-heal (#1928) Two review comments on this PR, both addressed: - gemini-code-assist flagged that maybe_autoheal_fts5_index's default progress=print goes straight to stdout. _validate_palace_fts5_after_mine runs inside the MCP server process too (mcp_server.tool_mine -> miner.mine), where stdout is the JSON-RPC transport -- a stray print() there would corrupt the protocol stream and crash the connection. Pass progress=logger.info instead; palace.py already has the module logger. - nikkunikku corroborated the fix from a real 1.4GB production palace (278 repeated abort-loop iterations before the fix) and pointed out a real test gap: the fixture-based auto-heal tests fabricate real FTS5 corruption via direct shadow-table writes, which some SQLite builds refuse (existing pytest.skip paths in test_miner_fts5_validation.py, related to #1925) -- so on those builds the auto-heal wiring in _validate_palace_fts5_after_mine is never actually exercised. Added their suggested build-independent tests, adapted to this file's fixture helpers: test_validator_suppresses_raise_when_autoheal_clears and test_validator_still_raises_when_autoheal_cannot_clear, stubbing mempalace.repair.sqlite_integrity_errors/maybe_autoheal_fts5_index directly instead of fabricating corruption. Added a third test, test_validator_passes_logger_progress_not_print_to_autoheal, covering the specific progress= wiring: the two tests above mock maybe_autoheal_fts5_index entirely and discard its kwargs, so neither would have caught the progress=print regression this commit actually fixes. The new test captures the real kwargs and asserts progress is a bound method of palace.py's own logger (not print), without pinning to logger.info specifically -- severity level is a verbosity choice, not a correctness requirement, so the assertion shouldn't fail on a reasonable future change to e.g. logger.debug. Verified both directions: fails against the pre-fix `print` default (and shows the leaked stdout line to prove it), passes at .info and at .debug alike. Full suite: 3221 passed, 20 skipped (unchanged skip count). ruff check/format clean. * feat: add exclude_patterns config key to mempalace.yaml Allow projects to specify .gitignore-style patterns that the miner should skip, without relying on .gitignore for mining control. A new optional exclude_patterns list in mempalace.yaml is parsed by the existing GitignoreMatcher class via a new from_patterns() classmethod — same syntax, same semantics as .gitignore, no new dependency. exclude_patterns: - '*.md' - '*.yaml' - 'docs/' # dir-only: prunes entire tree without descending - 'dist/' - 'coverage/' Key behaviour: - Patterns follow .gitignore rules: anchoring (/pattern), dir-only - dirs[:] pruning via GitignoreMatcher.matches(..., is_dir=True) so excluded subtrees are never walked - Checked after .gitignore filtering; force_include (--include-ignored) bypasses exclude_patterns - Pre-scanned files lists (init double-scan optimisation) are filtered too - Backwards compatible: omitting exclude_patterns changes nothing Changes: - GitignoreMatcher.from_patterns(): new classmethod, same rule parser as from_dir(), reads from a list instead of a file on disk - scan_project(): builds one exclude_matcher before os.walk; used for both dirs[:] pruning and per-file filtering - _mine_impl(): applies the exclude matcher to pre-scanned files lists when provided by the caller - tests/test_miner.py: three new tests test_scan_project_exclude_patterns_skips_matching_files test_scan_project_exclude_patterns_prunes_entire_directory test_scan_project_exclude_patterns_include_ignored_bypasses_exclusion * refactor(miner): extract exclude_patterns prescan filter into a helper Rebasing Lochness's exclude_patterns work (#1213) onto current develop pushed _mine_impl's cyclomatic complexity to 26, tripping the repo's max-complexity=25 ruff gate (clean on develop before this rebase). Extracted the pre-scanned-file-list filtering branch into _apply_exclude_patterns_to_prescanned_files -- same behavior, no test changes needed, complexity back under the gate. * fix(convo_miner): treat transcripts as mutable, not immutable Conversation transcripts were assumed immutable once mined: the bulk skip-check (prefetch_mined_set) only tracked "have we seen this source_file before at the current normalize_version", with no mtime comparison at all. That's wrong for how Claude Code sessions actually work -- a session keeps appending to its own JSONL file while active, and /compact or /clear can rewrite one in place. Once a session file was mined, any content appended after that point would silently never get mined, with no error or warning -- the file just looked "already filed" forever. palace.py: - prefetch_mined_set() now returns dict[source_file, stored_mtime] instead of a bare set[source_file]. `if src in mined_set` still works identically (dict `in` checks keys), so this is a source-compatible change for that access pattern; a caller that wants staleness detection reads mined_set[src] and compares against the file's current mtime itself. None means no mtime was ever stored (or getmtime failed when the drawer was written) and must be treated as stale, not "unknown, assume unchanged". - Removed bulk_check_mined(): it already existed for exactly this purpose (bulk mtime prefetch) but had zero callers anywhere in the codebase and was missing the normalize_version/extract_mode filtering prefetch_mined_set has -- folded its intent into prefetch_mined_set instead of maintaining two subtly-different, overlapping bulk scans over the same underlying data. - file_already_mined()'s docstring corrected: it previously claimed "transcripts are assumed immutable" for convo mining. That's no longer true; corrected to describe the actual current split (convo miner's bulk skip-check uses prefetch_mined_set's stored mtimes; this function's check_mtime=True path is now only its per-file, lock-held race-condition recheck). convo_miner.py: - New _is_unchanged_since_last_mine() helper (extracted to keep _mine_convos_impl under the repo's cyclomatic-complexity gate): false whenever the file isn't in the prefetched map, its stored mtime is None, getmtime fails, or the mtimes don't match -- true only when genuinely unchanged. - _file_chunks_locked's metadata now stamps source_mtime on every real drawer (mirroring miner.py's existing pattern), and its in-lock recheck now passes check_mtime=True. - _register_file's 0-chunk sentinel also stamps source_mtime, so a short file that later grows past MIN_CHUNK_SIZE is detected as changed instead of being skipped forever by the sentinel. One-time cost worth flagging: no existing convo drawer has source_mtime stored (this field never existed for convo mining before now), so the first `mempalace mine --mode convos` after this ships will see every already-mined file as stale and fully re-mine it. Not a bug -- _file_chunks_locked's existing purge-before-insert means no duplication results -- just a real, one-time cost across a large corpus. tests/test_convo_miner.py: 7 new tests -- grown-file re-mine picks up new content, unchanged file still skipped (the mtime check must not regress the existing optimization), grown-file re-mine purges stale drawers rather than accumulating duplicates (checked via unique content markers, not raw counts -- ChromaDB collections can carry unrelated bookkeeping rows), prefetch_mined_set's returned mtime matches the real file, None handling for a drawer with no stored mtime, a legacy drawer (no source_mtime field, simulating pre-this-change data) is correctly re-mined rather than skipped forever, and the sentinel path stamps source_mtime too. Full suite: 3327 passed, 20 skipped, 0 failed. ruff check / ruff format -- clean. * fix(mcp): self-heal writer lease instead of latching read-only for life The #1818 peer-writer guard latched _MCP_WRITER_READ_ONLY=True on the first MineAlreadyRunning and short-circuited every subsequent acquisition attempt, so a server that came up read-only (a peer held the per-palace flock at startup) stayed read-only for its entire process lifetime — even long after the peer exited and the OS released the flock. In the common case of several overlapping Claude sessions (one server per session, all on the same palace), whichever session started second was stranded: mutating tools kept refusing with -32001 and the only remedy was killing/restarting that server. _mcp_peer_writer_refusal already calls _acquire_mcp_writer_lock() on every mutating tool, so the retry hook existed — the sticky latch just suppressed it. Drop the read-only short-circuit: when read-only we now re-attempt the non-blocking flock each call and transparently promote to writer once the peer is gone. Race-safe — fcntl LOCK_NB is kernel-arbitrated, so two servers can never both win. The genuinely-broken-lock path (_MCP_WRITER_LOCK_FAILED) is still cached, since retrying a broken lock mechanism can't help. Adds test_peer_writer_readonly_self_heals_after_peer_exits. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(tests): satisfy ruff format check for peer-writer self-heal test PR #1960 merged with a red `lint` job: `ruff format --check .` wanted to collapse the multi-line `MineAlreadyRunning(...)` raise in the new `test_peer_writer_readonly_self_heals_after_peer_exits` onto one line (it fits the line-length limit). All six real test jobs passed; only the formatter check failed, which left `develop` red on lint. Reformat that one statement so `ruff format --check .` is clean again. No logic change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: retrigger CI (unrelated test-windows flake on prior run) * fix(mcp): answer initialize immediately — run startup preflight in a background thread The stdio loop ran _refresh_sqlite_integrity_status() and _refresh_vector_disabled_flag() before reading the first request. PRAGMA quick_check reads every page of chroma.sqlite3, so on multi-GB palaces the probe alone (measured: 20.3s on a 1.72 GB / 326k-drawer palace, 40-46s under disk/lock contention) starves the MCP client's 60s connect timeout — even though the initialize response itself never touches the database. The HTTP transport already starts without the synchronous probe. Move both probes to a daemon thread (mcp-startup-preflight). The #1222 intent is preserved: the probe still starts at startup and logs its warning as soon as it finishes. Consumers that need the verdict (_ensure_sqlite_integrity_status via the tool-call integrity gate) serialize on a new _sqlite_integrity_refresh_lock with double-checked locking, so a tool call arriving mid-probe waits for the in-flight verdict instead of running a second O(database size) quick_check — and never proceeds unverified. Measured on the 1.72 GB palace with the >512 MB startup gate disabled (MEMPALACE_STARTUP_INTEGRITY_MAX_MB=0, full quick_check in flight): initialize 1.4s (was 20-46s); first tool call after probe completion 3.4s with sqlite_integrity checked=true ok=true. Complements c54531a: the oversized-palace skip still applies to the background probe, but the handshake no longer depends on it. * style: ruff format test file * fix(palace): pair the mine_palace_lock holder-set update with its release _mark_held(palace_key) ran before the try: whose finally runs _mark_released(). An async exception (SIGINT/KeyboardInterrupt) landing after _mark_held() and before the try: skips _mark_released(), stranding the key in the process-wide _palace_lock_keys set while the outer finally frees the flock. The in-memory hold then outlives the OS lock: a later re-entrant acquire passes through and writes without the flock while another process can acquire it, i.e. two writers into one palace. Move _…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adding swift & kotlin files to the miners supported files
How to test
I tested it with a kotlin multiplatform project. i remined with the new READABLE_EXTENSIONS and got results for both languages on the search
Checklist
python -m pytest tests/ -v)`cd [PATH] && python3 -m venv .venv && .venv/bin/pip install -e ".[dev]" -q && .venv/bin/python -m pytest tests/test_miner.py::test_scan_project_includes_swift_files tests/test_miner.py::test_scan_project_includes_kotlin_files -v
tests/test_miner.py::test_scan_project_includes_swift_files PASSED [ 50%]
tests/test_miner.py::test_scan_project_includes_kotlin_files PASSED [100%]`
ruff check .)cd [PATH] && (.venv/bin/ruff check . 2>/dev/null || ruff check .) All checks passed!