Conversation
The website copy of the contributing guide had drifted from the authoritative root CONTRIBUTING.md, sending contributors down paths that create avoidable triage load: - PR target said `main`; the repo merges PRs into `develop` (`main` is tagged releases only). This is the contradiction reported in #1980. - Getting Started cloned the upstream repo directly, with no fork and no upstream remote, contradicting the "Fork the repo" step below it. - The `pre-commit install` step was missing entirely, so the ruff version pin CI enforces never got set up locally: code passes local lint, then fails CI on push. Bring the setup steps and PR target into parity with root. Fixes #1980 Co-authored-by: et1975 <623703+et1975@users.noreply.github.com>
) The #2221 fix reached develop through #2228 at an earlier revision of the branch, so three guards and their tests did not come with it. One of the three is a regression the gate that did land introduced. sweep_directory: the new gate probes the file type with f.stat() inside a try, and its except OSError printed SKIP and continued. A dangling symlink, a symlink loop and a file unlinked between rglob and the gate all raise there. Before the gate existed each of them reached sweep(), raised, and was appended to failures — so the gate turned "could not read this transcript" into a silent skip and a successful exit. A probe that FAILS is an error, not a benign file type: log it, print WARNING, book it in failures. A probe that succeeds and reports a non-regular file still skips silently. _parse_gradle: the is_file() gate sat in front of the try whose except OSError the parser already had, so a manifest under a directory with r but no x raised PermissionError out of a call that used to answer "no manifest name". The gate moves inside that try. _collect_manifest_names stats with os.path.isfile, which reports instead of raising, matching the parsers it guards. split_file: the type gate in main() covers the files the glob listed, but split_file builds its output names itself, so a pre-existing FIFO at one of them wedged write_text in the kernel waiting for a reader. Output names that are anything but a regular file are skipped. That gate asks os.path.lexists, not os.path.exists. exists() follows the link, so a DANGLING symlink at an output name reads as "nothing there" and the write goes through it, creating the target — a chunk landing wherever the link points rather than in the output directory. Measured: the two calls differ on that one case and agree on every other (regular file, symlink to a file, missing name, FIFO, symlink to FIFO, directory). test_gather_origin_samples_survives_an_unreadable_directory broke under root rather than passing vacuously: CAP_DAC_OVERRIDE walks into the 0o444 directory, the walled-off file stays readable, and the count assertion sees two samples instead of one. It now carries the same needs_unprivileged_posix gate as the three new permission tests. miner._read_text_no_follow: comment fix only. F_SETLEASE on a FIFO fails EINVAL, not ENXIO — measured on Linux 6.18 / glibc 2.39. The code branches on EAGAIN and is unaffected.
…2278) An isolated `malformed inverted index for FTS5 table ...` says the inverted index and `embedding_fulltext_search_content` disagree; it does not say which of them is wrong. `maybe_autoheal_fts5_index` rebuilt the index from that content table regardless and reported "rebuilt from intact content". Damaging the content table produces that same wording on SQLite 3.45.1, 3.47.1 and 3.51.2 against chroma's trigram table, and the rebuild then overwrites the index that still held the drawers' own terms. Measured on a mined 30-drawer palace with 12 drawers carrying one distinctive word: `lexical_search` returns 12 before the damage, 12 while quick_check is dirty, and 0 after the heal, with `embedding_metadata` still holding the word in all 12 rows. On the `mine` path nothing re-files afterwards and the heal prints nothing, so the loss is silent and permanent. Chroma writes each document twice -- into `embedding_metadata` under `chroma:document` and into the FTS5 table at `rowid = embeddings.id` -- and every read path returns the metadata copy. The heal now checks the content table against it, restores the rows that disagree, and rebuilds, in one transaction under the mine lock. It declines when the comparison raises and when no content row has a document to check it against. Rows the authority cannot speak for keep their content and are counted in the output; the rebuild still indexes them as they stand, which this does not fix. `typeof(m.id) = 'integer'` guards the restore: `embedding_metadata.id` is nullable, and NULL into the content table's `INTEGER PRIMARY KEY` auto-assigns rather than conflicts, so such a row would never reconcile and every later mine would fail. The test fixture now builds the three tables chroma writes rather than a bare FTS5 table, and indexes them with `trigram`: `unicode61` reports content-side damage as `fts5: checksum mismatch` from SQLite 3.51.2 on, which the classifier deliberately does not match, so the tests would otherwise pass or fail by which SQLite the runner links. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bensig no longer has write access, so GitHub was rejecting every rule naming them — `repos/.../codeowners/errors` reported six "Unknown owner" errors. A rule with an unknown owner is ignored rather than enforced, so the affected paths had no reviewer assignment at all. Four paths were owned by @bensig alone (.github/, .claude-plugin/, .codex-plugin/, integrations/), so the name is replaced rather than removed — deleting it would have left those rules ownerless, which fails the same way it did before, just silently. Also covers .antigravity-plugin/ and .cursor-plugin/, which exist in the tree but matched only the catch-all, and adds a header noting the write-access requirement so the next stale entry is easier to spot.
…e-owner fix(ci): replace inactive owner in CODEOWNERS
CoreML supports only ~280 of the 1647 nodes in embeddinggemma's quantized graph (100+ partitions) and computes it wrongly without raising: last_hidden_state comes back all-NaN, so the pooled sentence_embedding is NaN or all-zero depending on how that run partitioned. Since embedding_device defaults to "auto" and CoreML sits ahead of CPU, every Apple Silicon user on this model embedded degenerate vectors silently — and a `repair rebuild-index` would have written them over the whole palace. Two layers: - _AUTO_PROVIDER_DENYLIST keeps `auto` from ever picking CoreML for this model. CUDA/DirectML and other models are untouched (chromadb already strips CoreML for minilm on its own). CoreML is also ~2x slower here when it does run: 7.9 vs 16.0 docs/s on an M4 Max. - A witness embedding at load time makes any non-CPU provider prove it computes, so an explicit embedding_device=coreml is honoured but falls back to CPU with a warning instead of returning garbage. No healthy provider left raises rather than storing unsearchable vectors. CPU-only sessions skip the probe — CPU is the fallback, so checking it could only add a forward pass to every cold start. describe_device is model-aware for the same reason: the status header would otherwise name a provider we will not use. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bumps [docker/login-action](https://github.com/docker/login-action) from 4 to 4.5.2. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@v4...v4.5.2) --- updated-dependencies: - dependency-name: docker/login-action dependency-version: 4.5.2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6 to 7. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v6...v7) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…2097) * ci: attach provenance and SBOM attestations to the published image * ci: restore trailing newline at end of file
* docs: document KG supersede usage Requested-by: Grace Gettert <grace.gettert@carpe.io> * docs: update recall skills for KG supersede Requested-by: Grace Gettert <grace.gettert@carpe.io> * docs: update Claude recall skill supersede guidance Requested-by: Grace Gettert <grace.gettert@carpe.io>
Co-authored-by: seoseo-ai <261797377+seoseo-ai@users.noreply.github.com>
* test: add knowledge_graph metadata coverage * style: format knowledge_graph metadata tests
CLAUDE.md lists two non-negotiable latency targets under Design Principles:
- Hooks under 500ms.
- Startup injection under 100ms.
Until now those were prose claims with no enforcement. Anyone could add an
eager 'import chromadb' to mempalace/__init__.py and the promise would
silently regress — a lean package import is the concrete lower bound for
both claims, so the first thing to protect is the import path.
Adds tests/benchmarks/test_performance_budgets.py with two tests that run
each measurement in a fresh Python subprocess (so the import is truly
cold, not polluted by whatever pytest already loaded):
- 'import mempalace' must finish under 100ms (3x CI multiplier)
- 'from mempalace import cli' must also finish under 100ms — this is
the path hooks take, so a regression here adds latency to every
Stop/PreCompact invocation before any real work begins
Lives in tests/benchmarks/ so it is excluded from the default
'pytest tests/ --ignore=tests/benchmarks' run and does not slow the main
CI loop. Invoke explicitly when validating performance-sensitive changes.
Local measurements right now:
import mempalace ~6ms
from mempalace import cli ~9ms
Both comfortably under the 100ms target; test failure means real drift.
… Windows (#1382) Follow-up to #1204 per @mschultheiss83's request — applies the same narrow Windows encoding fix to the three remaining LongMemEval-shaped benchmark runners that share the audit pattern jphein flagged in #1204's review (`benchmarks/{locomo,membench,convomem}_bench.py`). For each file: - All `open(path)` / `open(path, "w")` calls gained `encoding="utf-8"` so cached benchmark JSON, palace-cache files, and result files are always read/written as UTF-8 instead of inheriting the platform default (cp1252 on Windows, GBK on Chinese Windows, etc.). Same pattern as #1204's #2917 / #2927 / #2957 / #2998 / #3031. - Replaced non-ASCII separator characters in `print(...)` chrome with ASCII equivalents (`─` → `-`, `→` → `->`) so the runners don't raise `UnicodeEncodeError` on a default cp1252 console. Comments and docstrings (which never hit stdout) are untouched. - `urllib.request.urlopen(...)` calls left alone — they don't open local files, the original audit didn't flag them. Audit was already done in #1204's thread; this PR carries the mechanical follow-through. No behavior change beyond Windows correctness. Refs #1203 (the original Windows reproducer), #1204 (sibling PR that fixed `longmemeval_bench.py`). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: remove hardcoded fallback names from split_mega_files Replace developer's family names in _FALLBACK_KNOWN_PEOPLE with an empty list. Names like "Max" and "Sam" matched unrelated content in new installations without known_names.json configured. Name detection now requires explicit opt-in via ~/.mempalace/known_names.json. * fix: update stale comment after removing fallback names The comment said "otherwise generic fallback" but the fallback list is now empty. Updated to match. * fix: remove empty fallback constant and add no-names test Address remaining review concerns from @bgauryy on #166: - Remove _FALLBACK_KNOWN_PEOPLE constant — inline return [] in _load_known_people() since the constant was always empty - Update docstring to reflect opt-in-only name detection - Update existing test to assert against [] directly - Add test_extract_people_empty_when_no_config verifying that common names like Max and Sam are not detected without config
) ChromaBackend._client caches a PersistentClient per palace and keys the cache on chroma.sqlite3's (inode, mtime) so that a write by another process forces a rebuild rather than serving a stale HNSW segment. Constructing a PersistentClient writes to chroma.sqlite3, and so do the collection opens that follow it. The stamp was taken immediately after the constructor, so it was already behind by the time the surrounding operation finished its own writes, and the next _client() call read that footprint as an external change. A search opens mempalace_drawers and then mempalace_closets, so the first open moved the mtime the second one checked and the cache missed on essentially every request. Each miss rebuilt the client and reloaded both HNSW segments, and the client being displaced was overwritten in the dict without being closed, so its native index allocation (max_elements * size_data_per_element, ~440 MB per collection on a 165k vector palace) was never released. Re-baseline the stat once the backend's own operation completes, so the recorded value means "chroma.sqlite3 as this backend last left it" and a later difference is genuinely somebody else's write. Cover writes made through ChromaCollection as well, via its existing _write_lock context manager, so the file-a-drawer-then-search cycle stops reloading the index. Close the displaced client on the rebuild path. An external write that lands while one of our own operations is in flight is folded into the new stamp and picked up on the next change. mtime cannot distinguish writers on its own, and PRAGMA data_version does not help: it reports writes by any other connection, and chromadb's connection is foreign to a probe connection, so our own opens would register as external there too. Eight searches against the same palace, HTTP transport: before 951 MB -> 3522 MB after 940 MB -> 965 MB
sqlite_exact.query() loaded every document and metadata blob to do Python-loop cosine on the whole collection. On a 167k-drawer palace that made mempalace_search 6-9s and mempalace_status 7s (paging every metadata row because sqlite_exact had no facet_counts). Rank from the embedding column with vectorized numpy cosine, hydrate only the top-k documents, and cache the matrix plus wing/room/source_file on the long-lived handle. Add facet_counts and sqlite_wing_room_counts so status/list_wings use one GROUP BY. Advertise supports_metadata_facets and expose collection._backend so the MCP facet path runs through the embedding wrapper. Live palace (167k drawers / 166k closets): status 6997ms -> 1045ms search warm 6364ms -> 210-1600ms search cold 8236ms -> 5299ms (first matrix load)
The peer sync loop is started only by _serve_http, and the estate it builds (_PEER_SYNC_STATE, _KNOWN_PROFILES) is process memory. But mempalace_mesh_peers ships in every transport, including the stdio servers agents connect through, and those processes never run a sync round. So they answered from a permanently empty estate: every peer reduced to a bare name and url with no reachable, last_success_at, remote_version_vector or profile; origin_profiles holding only this node; and configured peers reported as unnamed_origins -- "known only transitively" -- because their replica_id is learned during a sync and nothing else supplies it. The hub next door had all of it. The sync loop now publishes the estate to mesh_state.json in the per-palace server state directory after every round, alongside the token and serverinfo that already use that directory for exactly this "hub records something other local processes read" purpose. 0600, and written to a temp name then renamed so a reader in another process never observes a half-serialized estate. _mesh_peers_payload merges the published estate underneath any in-process state, per peer, so the process that actually syncs keeps reporting its own fresher observation and every other process reports the hub's instead of nothing. The new estate_source field says where the reading came from: in_process, published_at, and whether the publishing hub is still alive. A crashed hub leaves a last-known-good estate, which is worth showing -- "last seen as" beats a blank node -- but must not be read as live. peers.json tokens are not in the estate and never reach the file; both are asserted.
list_drawers(limit=20) still materialized every matching drawer (documents + embeddings). get(ids=...) scanned the whole collection. Search ran a second exact-cosine over 166k closets. Status/list_wings re-ran the 1s json_extract GROUP BY on every call. graph_stats had no sqlite_exact fast path and paged all metadata. - get() selects only requested columns; get(ids=) uses IN; equality where + LIMIT go to SQL - list_drawers walks metadata only, then hydrates the page - 5s taxonomy cache, dropped on writes - graph_stats uses json_extract GROUP BY on sqlite_exact - closet boost uses FTS lexical_search instead of cosine Isolated live palace (167k drawers): full-row fetch 806ms vs metadata 591ms vs LIMIT 20 / IN 20 at 0ms. Taxonomy GROUP BY ~1s, then cached.
mempalace-mcp is spawned once per agent session. Whenever a hub is running, every one of those processes is a pure proxy: the stdio dispatcher forwards each JSON-RPC request over HTTP and never touches local storage. Importing mempalace.mcp_server to do that cost ~77 MB anyway, because chromadb (~61 MB on its own), numpy, pydantic, grpc and opentelemetry are all imported at module scope. A fleet of 50 agents therefore spent ~3.9 GB holding proxies that do no work. Move the mempalace-mcp entry point to a new mempalace.mcp_proxy, which imports only the standard library plus config and server_registry (~5 MB each). A proxied session measures ~17 MB end to end, ~24 MB of import weight against ~80 MB for the full server. The full server is imported lazily, the first time this process has to answer a request itself. Anything that is not a plain stdio session -- another transport, a server-side flag, an argument the thin path does not recognise -- is handed to the full server untouched, since the heavy import was going to happen there regardless. The local fallback is kept as-is, including the rule that a mutating call which failed mid-flight is never replayed locally: the hub may still be executing it. What changes is that the fallback is no longer silent. The first tools/call served locally carries a notice in result.content, because the agent driving the session is the one that needs to know its memory backend just lost the hub and this process is now holding the whole index. Restoring stdout around that lazy import is load-bearing rather than cosmetic: importing the server installs its stdio protection with os.dup2(2, 1) and sys.stdout = sys.stderr, which moves fd 1 itself, so holding a reference to the old object is not enough. Without the restore, every response after a fallback goes to stderr and the client waits forever -- caught by an end-to-end test that killed a real hub mid-session.
fix(sqlite_exact): cut remaining palace-wide read paths
Status, graph_stats, and wing filters still json_extract-scanned the whole documents table. That is the opposite of structural retrieval: loci (wing/room/hall) should be first-class, not buried in JSON. Add VIRTUAL generated columns (no rewrite of embedding blobs) and a composite index on (collection_id, wing, room, hall). Existing palaces migrate on the next writable open. Taxonomy, graph_stats, facet_counts, and equality where use the columns when present. This is the sqlite slice of mempalace-structural's idea — search the place, not the haystack — without porting palace_v4 routing.
perf(mcp): make a proxied stdio session stop loading the storage stack
fix(sqlite_exact): index wing/room/hall for structured access
On the i7-8750H chroma palace, list_drawers(limit=20) took 36s and find_tunnels died at 30s because both paged col.get() and cold-loaded the vector index. chroma.sqlite3 already has embedding_metadata. - sqlite_list_id_metadata: ids + metadata + chroma:document without HNSW - sqlite_room_wing_hall_counts shared with graph_stats - build_graph uses that grouped read when no collection is injected - find_tunnels/traverse no longer open the collection first MCP tests assert the client paging path is not used.
… rule Coordination stalls are usually listening failures, not protocol failures: a task sits open because the agent it was addressed to was never watching, and the requester cannot tell "working on it" from "nobody is home". The load-bearing correction is the cursor. Events are ordered by append order (ORDER BY rowid), not wall clock, so a peer's event is appended whenever it syncs and can already be older than a timestamp high-water mark. Resuming with since_created_at therefore drops late-arriving cross-replica events permanently. Measured one such inversion in a single 50-event window: a windows-origin event created 09:10:48Z ingested after a mac-origin event created 09:13:21Z. list_events' docstring already said since_event_id is "the precise cursor ... regardless of timestamp ties" — that just never reached an agent. - coordination-protocol.md: new "Monitoring the stream" section — the cursor rule, four modes (inbox sweep / long-poll / SSE / declared-idle), the announce-your-watch convention, and declaring when you are NOT watching. Two new hard rules. - coordination-protocol.md: system-prompt snippet updated, since that block is copied verbatim into every agent's instructions. - agent-logstream.md: matching "Monitoring" concepts section. - mcp_server.py: event_list and event_wait descriptions now state the cursor rule at the point of use, and both since_created_at params are marked "NOT a resume cursor". The announce-your-watch shape is generalized from windows:grok:mempalace's issue-354-live-layer announcement, which named its filter, its cursor, and the work not to duplicate. Docs and tool descriptions only; no behavior change.
…an be woken by `logstream wait` is a primitive, not a watcher. It caps at MAX_WAIT_TIMEOUT_MS and reports a timeout, so every caller writes the same re-arm loop and each one has to remember to carry the cursor forward. Agents mostly did not, so coordinated tasks stalled on nobody listening rather than on the work. Two filters a watcher needs cannot be expressed by list_events, whose SQL is single-valued and positive-only: * "wake me for task.request OR patch.ready" * "everything addressed to me EXCEPT my own events" The second is not a nicety. to_agent=<me> deliberately also matches '*' broadcasts, and an agent's own broadcasts are broadcasts — so a hand-rolled watcher wakes itself every time it posts a status. Found by running one: mac-claude woke on its own fleet announcement within seconds. - logstream.watch_events(): owns the re-arm loop and the cursor. Yields ([], cursor) on idle polls so callers can time out, heartbeat, or checkpoint without a second clock. The cursor advances past every event examined, not only matches, so a restart never re-judges what it rejected. - event_matches_watch() / normalize_watch_values() / pushdown_watch_filters(): single-valued filters push down to SQL to keep the query selective; multi-valued and negative parts are re-checked client-side. Pushdown is an optimization, never a correctness dependency. - read/write_watch_cursor(): atomic temp+rename. A corrupt state file costs a replay; refusing to start costs every event after it, so reads degrade to None rather than raising. - CLI `logstream watch`: --agent (shorthand for --to-agent X --exclude-from-agent X), repeatable filters meaning "or", --state-file, --follow, --idle-exit-ms. Exit 0 on a match, 2 on idle — the same convention `wait` uses, so a harness can background the process and treat its exit as "you have mail". Docs updated to lead with it: coordination-protocol.md (including the system-prompt snippet every agent copies), the agent-logstream concepts page, and the CLI reference. 20 tests: self-exclusion at both the matcher and CLI level, multi-valued ORs, cursor advance-past-rejected, resume-without-replay, --since-event-id overriding the state file, and the exit-code contract.
…iew cursor docs Codex P2s on the watch PR, confirmed on Windows: - Cap each wait_events timeout to remaining --idle-exit-ms so idle 400ms does not wait out a 300s long-poll (measured 2687ms -> 572ms). - Persist the watch cursor after successful stdout on a match; unmatched advances still checkpoint immediately. A broken pipe no longer skips the event on resume. - preview=true docs told clients to re-fetch via since_event_id of the truncated event, but that cursor is strictly after it. Repeat the original filters with preview=false instead.
… the log windows:grok:mempalace flagged the first-run replay on PR #2315. Measured on the real shared brain: a fresh `logstream watch --agent mac-claude` with no cursor woke holding 41 events, the oldest 49 days old. Nothing in the payload marks them stale, so an agent reads July's task.requests as new work — the opposite of the problem this command exists to solve. `latest_event_id()` already documents the right behaviour for the SSE live-tail: capture the tip at connect time "so they receive only post-connect events". The CLI watcher not doing the same made one product with two first-run semantics. - default: a watch with no --since-event-id and no stored cursor starts at the tip. Backlog belongs to the inbox sweep (`logstream list`), which can page it deliberately; a watcher is for what arrives from now on. - never silent: it prints what it skipped and how to get it, on stderr, in --json mode too, so the note cannot corrupt a parsed payload. - --from-start opts back into the replay. Six existing tests turned out to depend on the replay, which is the evidence this mattered. Their intent is filtering, not first-run behaviour, so the shared args helper opts them into --from-start and three new tests cover the tip default explicitly — including that an explicit --since-event-id or a state-file cursor still wins over it. Note for review: test_agent_shorthand_does_not_wake_on_your_own_broadcast asserts exit 2, which the tip default would satisfy for the wrong reason. It opts into --from-start so it keeps testing the exclusion. 4395 passed, 31 skipped.
Seven inline P2s on 8663ecd. Two were stale — Codex anchored them to lines 0d3f216 had already fixed. The other five were real, and all five are mine. (a) read_watch_cursor: a state file holding valid JSON that is not an object (null, [], a bare string) made .get() raise AttributeError, which the except clause did not catch — so the watcher refused to start, the exact opposite of the recovery contract in its own docstring. (b) KeyboardInterrupt returned normally, exiting 0. Exit 0 is the documented "a match was printed" signal, so SIGINT told a supervisor it had mail that never arrived. Now 130. (c) --poll-timeout-ms 0 with no idle deadline made the callable return zero forever, so watch_events' expired-deadline branch yielded without ever polling and burned a core. Rejected at the CLI, so that branch can now only mean a deadline actually elapsed. I saw this shape while reviewing 0d3f216 and wrongly filed it as unreachable in-tree. (d) A regression I introduced in 8663ecd. read_watch_cursor returns None for both "no state file yet" and "state file present but unreadable"; the tip-default treated the second like the first, skipping everything since the last good checkpoint and then re-checkpointing at the tip, making the loss permanent. A failed read now replays and says so, which is what read_watch_cursor promises: "a corrupt state file costs a replay". (e) --follow --json printed repeated indented documents, which json.load and jq reject as trailing data — a machine-readable flag that was not, on the mode built for daemons. Follow now emits NDJSON, one compact record per line; single-shot keeps the one pretty document. Five regression tests, each verified to fail against 8663ecd. The --poll-timeout-ms one initially passed pre-fix for the wrong reason — an idle timeout also exits nonzero — so it now pins the validation error itself rather than "some nonzero exit". 4400 passed, 31 skipped.
…behind Third Codex round, one P2 on 57d8d24, and a third variant of the same family as the last two: "no cursor" is not one fact. Sequence: a stateful watch starts against an empty log, so latest_event_id() is None and write_watch_cursor ignored the falsey cursor, leaving no file. The watcher idles out. An event arrives while it is stopped. The next launch sees no state file, concludes first run, jumps to that event as the new tip, and skips it — then checkpoints past it, making the loss permanent. The root cause both this and the previous round shared is that a missing cursor was being read as a single condition when it is four: absent no state file — a genuine first run, may start at the tip ok a usable cursor, resume strictly after it empty file exists recording cursor: null — started against an empty log, has not seen an event yet. NOT a first run. corrupt unreadable, truncated, or valid JSON that is not an object Only `absent` may start at the tip. `empty` and `corrupt` replay. read_watch_state returns (cursor, condition) and the CLI branches on the condition; read_watch_cursor stays as a wrapper for callers that do not care. write_watch_cursor now persists a null cursor rather than skipping the write, and the starting position is recorded at startup instead of waiting for the first idle yield, so a kill in between cannot reproduce the same hole. Three regression tests, each verified to fail against 57d8d24, including the full arrives-while-stopped sequence end to end. 4403 passed, 31 skipped. CI green on 57d8d24 across all nine checks.
…one must not fail quietly Fourth Codex round, two P2s on b35bfc1. Same family as rounds two and three — both are ways a *present* checkpoint gets mistaken for a first run — so both are checked against the invariant rather than patched individually: a restart may cost a duplicate, never a missed delegation. read_watch_state no longer preflights with os.path.exists(). That call answers False both for "no such file" and for "cannot traverse the parent directory", so a checkpoint that exists but is momentarily unreachable was classified absent, started at the tip, and skipped every event since the stored cursor. Opening directly lets FileNotFoundError mean absent and every other OSError — permissions, a directory in the way, I/O error — mean corrupt, which replays. It also closes the exists()/open() race. write_watch_cursor gains required=. Best effort is the right default because a lost checkpoint costs a replay while a crashed watcher costs every event after it. That trade-off inverts for the first checkpoint of a fresh watch: if it never lands there is no state file, the next launch calls itself a first run, and it starts at the tip — so a swallowed failure there guarantees the exact skip the checkpoint existed to prevent. The CLI now refuses to start rather than continuing into that. Three regression tests, each verified to fail against b35bfc1. The unreachable-file case is exercised twice, via a directory standing where a file is expected (OSError on every platform) and via a simulated PermissionError, so it holds on the Windows runner too. 4406 passed, 31 skipped.
…ced it Fifth Codex round, one new P2 on dd9b123, and the sixth instance of one bug: a starting cursor that never reaches disk, so the next launch calls itself a first run and jumps to the tip. This time --since-event-id with no state file skipped the immediate checkpoint entirely, because that write lived inside the "resolve the tip" branch and an explicit cursor never enters it. Nothing was persisted until watch_events yielded, which by default is up to five minutes later; an interrupt inside that window leaves no file at all. Rather than add a third call site I hoisted it: one place resolves the starting position, one place records it. The position can arrive three ways — an explicit --since-event-id, the tip, or None from an empty log or --from-start — and all three now take the same required write whenever no state file exists yet. That removes the shape the last four rounds kept finding new instances of, instead of removing one more instance. Two test-quality notes, both the same lesson: The first version of this test passed against the unfixed code. The helper uses a 60ms poll timeout, so watch_events yielded immediately and the loop's own checkpoint wrote the file — the startup window the bug lives in never opened. It now stubs watch_events to interrupt before yielding, which is what actually pins the startup write, and covers all three entry paths. The hoist also broke test_match_does_not_checkpoint_if_output_fails, which asserted that no state file existed after a failed match write. That was pinning the symptom: a startup checkpoint now legitimately exists. The invariant is that its cursor must still be the pre-match position so the undelivered event replays, so it asserts the cursor value instead. 4407 passed, 31 skipped.
…tive idle as forever Sixth Codex round, two P2s on b5c4af4. A typo'd or stale --since-event-id was persisted by the required startup write before anything checked it. list_events raises on an unknown anchor and that ValueError was uncaught, so the run died with a traceback while the bad id sat in the state file — and every later run without the flag reloaded it and died the same way until someone deleted the file by hand. The cursor is now verified before it is written anywhere. That check distinguishes where the cursor came from, because the right answer differs. An explicit --since-event-id that does not resolve is user error: refuse, and leave nothing behind to reload. A *stored* cursor whose event has gone — log rebuilt, replica reset — is corrupt state, and refusing would strand the watcher exactly as an unreadable file would, so it replays with a notice instead. Same invariant as the rest of this branch: a duplicate, never a missed delegation. --idle-exit-ms now rejects negatives. Only 0 documents "wait forever", but a negative took that same branch silently, so a value arriving from config or from timeout arithmetic would leave a harness waiting on a watcher it believed would time out. Three regression tests. The negative-idle one cannot simply be run against the old code to prove it: the pre-fix behaviour is an infinite wait, so the tripwire check hangs rather than fails, and confirming it needed an external timeout. Worth knowing before someone "fixes" that test by shortening it. 4410 passed, 31 skipped.
… polling Seventh Codex round, two P2s on af9b7e2, and the same shape twice: input that was only rejected once the loop had already started. Filter validation depended on arity. list_events sanitizes what it is given, so a single-valued watch filter was checked for free by being pushed down, while a repeated one was not pushed down and got compared raw. `--type Task.Request` alone was an error; `--type Task.Request --type patch.ready` was accepted and then matched nothing, so the watcher waited forever for an event type that cannot exist. sanitize_watch_spec now runs every value through the same sanitizers list_events uses, whatever the arity, and normalizes them so a padded routing value still matches. --limit 0 was accepted by argparse and raised inside the first poll, where the only handler was for KeyboardInterrupt: a traceback, and under --json no error document at all. Validated up front. Both are argument checks, so they belong together and ahead of everything else: _watch_spec now validates and builds the whole spec before any polling, cursor resolution, or checkpoint write. That ordering is the point — round six's bug was a bad cursor being persisted before it was validated, and keeping validation in one place ahead of side effects is what stops the next one. Added a ValueError backstop around the loop as well. Every known bad input is rejected earlier now, but an escape would otherwise surface as a traceback, and under --json as nothing a machine consumer can parse. Four regression tests, each verified to fail against af9b7e2. 4414 passed, 31 skipped.
feat(logstream): background watcher agents can be woken by, plus the monitoring protocol
…hkova-566240 fix(embedding): never let CoreML return NaN vectors for embeddinggemma
…ntent fix(repair): check the FTS5 content table before rebuilding from it (#2278)
Version bumped across all seven sources and CHANGELOG [Unreleased] promoted to [3.8.0]. Note for whoever cuts the next one: `integrations/openclaw/SKILL.md` carries a `version:` in its frontmatter and is the seventh source. version-guard checks five, a test enforces the README badge, and nothing at all enforces that one — it has been bumped by hand in every release commit back to 3.3.0. docs/RELEASING.md still says "all five". uv.lock carries only the mempalace version line. Regenerating it locally also resolved onnxruntime-gpu for Python 3.10 from 1.24.3 down to 1.20.2 and dropped the 1.24.3 entry — an unrelated dependency change that has no place riding along in a release bump, so the lock was restored and the single line edited instead. Verified with --frozen.
chore(release): 3.8.0
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3e56979fb4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+961
to
+964
| cached = self._handle._vector_cache.get(collection_id) | ||
| if cached is None: | ||
| cached = self._load_all_vectors(cur, collection_id, expected) | ||
| self._handle._vector_cache[collection_id] = cached |
There was a problem hiding this comment.
When a long-lived sqlite_exact handle has completed one query and another process or backend handle appends or updates drawers, this handle never executes its own write path, so _vector_cache is not cleared and every subsequent search ranks the old IDs and matrix indefinitely. This makes newly filed memories unsearchable until the reader process restarts; reload the cache when the database changes externally, such as by tracking SQLite's data version or file freshness.
AGENTS.md reference: AGENTS.md:L7-L9
Useful? React with 👍 / 👎.
| # scan) and non-negative bounds: SQLite does not honor a negative LIMIT | ||
| # or OFFSET the way a Python slice does, so those keep the slice path. | ||
| # get(ids=...) must not scan the collection: look up by primary key. | ||
| if ids is not None and where is None and where_document is None: |
There was a problem hiding this comment.
When callers provide ids together with where or where_document, this condition bypasses the ID lookup, and the remaining path never applies ids at all. The call therefore returns every row matching the filter instead of the requested-ID intersection, unlike the previous implementation and the other backends; retain the ID restriction for combined-filter calls.
Useful? React with 👍 / 👎.
…2320) `sync --apply` deleted drawers whose source file it could not reach, not only drawers whose source file was removed. `_classify_drawer` drew that line with one `Path.exists()`, and `missing` feeds `removable_ids`, so every state that answered no became a deletion: a directory that is gone, a path component that is a regular file, a parent that is a dangling symlink, and a volume that is not mounted. Three further states did not delete but ended the whole run at the first such drawer, in dry run as well, because nothing between `_classify_drawer` and the CLI catches: a symlink loop, a directory the process may not enter, and a `source_file` the platform cannot encode. Two of those three turn into deletions on newer interpreters, as `pathlib` stopped raising in 3.13 and in 3.14. Removal now asks for corroboration rather than for a probe. A file that is not at its path is removable only while the palace can still see a source of its own in that same directory: a deletion leaves the file's neighbours where they were, and a volume that is not mounted takes every one of them away at once. Nothing about the file alone separates those, because no errno distinguishes "nothing is here" from "this cannot be reached right now", and the directory's own contents are no better a witness: a mount point inside a repository is never empty, since git cannot track an empty directory, so its `.gitkeep` outlives the unmount. `_uncopyable_reason` in `backups.py` already states the errno half of the rule, for an operation that only leaves a file out of a backup copy. Three things make that corroboration safe to lean on. A neighbour must stat as a regular file: `tool_add_drawer` stores its caller's string verbatim, and a `source_file` whose last component is empty, `.` or `..` is filed by `os.path.dirname` under the directory it names, or under that directory's parent, either of which outlives the unmount that empties it. Every neighbour the palace knows in a directory is asked rather than one remembered drawer, so the verdict does not turn on which drawer the pass reached first. And both halves are read again when the verdict is formed rather than trusted from earlier in the pass, because a volume can leave inside one pass and can come back inside one. Everything not corroborated lands in a new bucket, `unresolved`, never added to `removable_ids`. A count of kept drawers the operator cannot turn into paths is not a report, so it is reported the way removals are: counted beside the other buckets, its sources named in a new `unresolved_by_source` field, and the first five printed with the remainder stated. Both renderers also say why such a drawer is kept, since no command removes one. Measured through `sync_palace` across twelve source-file states: eight were deleted before, and only four of those eight were a file that really had gone. Two are deleted now and both really were deleted, and no state ends the run any more. `_auto_detect_project_roots` no longer ends it on an ancestor it cannot walk either, and its two markers are probed separately so an unreadable `.git` cannot hide a `.gitignore` beside it. The price is two real deletions it no longer prunes: a file deleted alone from a directory the palace knew nothing else in, and a directory's files deleted together. Both are indistinguishable from that directory's contents being away. Two shapes stay open. A neighbour proves the directory is reachable and not that it is the filesystem the missing file was mined from, so a mount point that also holds a mined file of its own still corroborates, and mounting a volume over a directory the palace knows a file in still removes that file's drawer; both were measured through a real mount, and `develop` loses the same drawers in both. Telling them apart needs the identity of the filesystem each source was mined from, which nothing records. And nothing spans two syscalls, so a volume that leaves and returns between the neighbour's reading and the file's own is not covered.
fix(sync): require an established absence before removing a drawer (#2320)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d1088c5b4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+1017
to
+1021
| if unkeyed: | ||
| progress( | ||
| f" {unkeyed} content row(s) sit at an id no embeddings row uses; " | ||
| "this table was not written by the current chromadb schema." | ||
| ) |
There was a problem hiding this comment.
When a palace was carried through Chroma's legacy migration that keyed FTS content by embedding_metadata.rowid, unkeyed indicates that this code cannot reliably associate content rows with drawers. Merely reporting the condition and continuing allows damaged, unverifiable content rows to be baked into the rebuilt index; quick_check then passes even though affected drawer terms are no longer searchable. Detect the legacy layout and decline the heal rather than committing a rebuild from content that cannot be verified.
AGENTS.md reference: AGENTS.md:L22-L23
Useful? React with 👍 / 👎.
Comment on lines
+72
to
+74
| if arg in allowed_flags: | ||
| i += 2 | ||
| continue |
There was a problem hiding this comment.
When a hub is live, an invocation ending in --palace, --backend, or --collection is classified as proxyable because this branch advances past the end of argv. The full server's argument parser would reject the missing value, but the proxy instead silently falls back to the configured palace and can serve requests against the wrong hub; verify that a value exists before accepting these flags.
Useful? React with 👍 / 👎.
#2322 merged into develop after the release commit, so it is inside the develop -> main promotion (#2317) and ships in the v3.8.0 tag. The entry was filed under [Unreleased], which would have shipped a changelog telling users the fix was not yet released while the code was in their install. Move the entry verbatim into the 3.8.0 Bug Fixes section. No wording changes. The [Unreleased] link definition still points at v3.7.1...HEAD; it was already stale before this change and belongs to the tag step, so it is left alone here.
docs(changelog): file the sync corroboration fix under 3.8.0
SgtSlummy
pushed a commit
to SgtSlummy/mempalace
that referenced
this pull request
Aug 22, 2026
MemPalace#2322 merged into develop after the release commit, so it is inside the develop -> main promotion (MemPalace#2317) and ships in the v3.8.0 tag. The entry was filed under [Unreleased], which would have shipped a changelog telling users the fix was not yet released while the code was in their install. Move the entry verbatim into the 3.8.0 Bug Fixes section. No wording changes. The [Unreleased] link definition still points at v3.7.1...HEAD; it was already stale before this change and belongs to the tag step, so it is left alone here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v3.8.0 — fast large palaces, lean MCP proxies, wakeable agents
Large palaces get fast and stay small in 3.8.0. Chroma metadata tools no longer cold-load HNSW,
sqlite_exactstops scanning and hydrating the whole palace for common reads, long-running Chroma servers stop rebuilding their client on their own writes, and proxied MCP sessions no longer import a storage stack they never use. Agents also gain a persistentlogstream watchprimitive so coordination can wake a harness instead of relying on repeated five-minute waits.Performance
list_drawers(limit=20)fell from 2.31 s and 1148 MB peak RSS to 0.01 s and 86 MB.list_drawersand tunnel reads now query Chroma's SQLite metadata directly, push filters into SQL, and hydrate only the requested previews. (fix(chroma): read list_drawers and tunnels from sqlite, skip HNSW #2314)sqlite_exactcommon reads no longer walk the whole palace. On a live 167k-drawer / 166k-closet palace,mempalace_statusfell from 6997 ms to 1045 ms and warmmempalace_searchfrom 6364 ms to 210–1600 ms. Vector ranking is batched and cached, top-k documents are hydrated afterward, equality filters paginate in SQL, taxonomy results are cached, andwing/room/hallgain indexed generated columns without rewriting the 1.6 GB embedding table. (fix(sqlite_exact): speed up query and status on large palaces #2308, fix(sqlite_exact): cut remaining palace-wide read paths #2311, fix(sqlite_exact): index wing/room/hall for structured access #2313)Agent coordination
mempalace logstream watchprovides a durable wake-up loop. It owns cursor advancement, supports repeated type filters, agent self-exclusion, persistent state files, idle exit, and NDJSON follow mode. Cursorless first runs start at the live tip; restarts may replay an event but do not silently miss one. (feat(logstream): background watcher agents can be woken by, plus the monitoring protocol #2315)since_event_id—not a wall-clock timestamp—is the resumable high-water mark. Tool descriptions and the shared coordination protocol now put that rule where agents use it. (feat(logstream): background watcher agents can be woken by, plus the monitoring protocol #2315)Integrity and operations
sync --applyno longer deletes drawers whose source file it could not reach. OnePath.exists()separated keep from remove, so an unreadable directory, a path component that is a regular file, and a volume that is not mounted all answered the same as a deletion — a project mined from a mounted volume could lose every drawer to onesync --applywhile the volume was away, and come back empty. Removal now asks for corroboration rather than a probe: a file that is not at its path is removable only while the palace can still see a source of its own, a regular file, in that same directory. Everything uncorroborated lands in a newunresolvedbucket, never pruned and reported the way removals are. (bug: sync --apply deletes drawers whose source file it could not reach #2320)mempalace_mesh_peerscan report reachability and profiles from stdio as well as HTTP. (fix(mesh): publish the estate so every transport can report it #2309)sweepreports failed stats,inithandles unreadable manifest paths, andsplitrefuses FIFO or symlink output targets instead of blocking or writing through them. (fix(ingest): finish the non-regular-file guards left out of 3.7.1 (#2221) #2244)A big thank-you to everyone who contributed to this release. Contributors and first-time contributors are credited below.
What's Changed
New Contributors
Full Changelog: v3.7.1...v3.8.0
Install:
pip install -U mempalace==3.8.0