Bind tool calls to the active engine session - #6
Merged
stephenschoettler merged 1 commit intoApr 12, 2026
Merged
Conversation
Collaborator
Author
|
Quick heads up: I also have a Hermes side PR here: NousResearch/hermes-agent#8416 That PR is only about making Hermes work cleanly with external context engine plugins. It does not bundle or vendor hermes-lcm into Hermes. I am linking it here because this session scoping fix matters for longer lived Hermes runtimes where more than one session can exist over time and tool state needs to stay bound to the active engine instance. |
stephenschoettler
approved these changes
Apr 12, 2026
stephenschoettler
left a comment
stephenschoettler
left a comment
Owner
There was a problem hiding this comment.
Review: Approve
Solid fix that eliminates the global _engine singleton in tools.py, replacing it with explicit engine=self parameter passing. This prevents cross-engine contamination in multi-session/multi-engine processes.
Highlights
- Session-scoping added to describe/expand —
_get_session_node()helper correctly prevents cross-session data leaks - Sparse depth fix — Old overview used
range(10)with earlybreakon empty depth, so d0+d2 (no d1) would miss d2. Now iterates actual depth set. Good catch. - Clean code improvements —
elifafterreturn→if, descriptive variable names, consistent formatting - Good test coverage — 3 new tests cover per-engine isolation, cross-session rejection, and sparse depth
Minor notes
-
_require_enginenaming — ReturnsNoneinstead of raising, which is misleading for a "require" function._get_enginewould be more accurate. Not a blocker. -
Overview memory usage —
get_session_nodes(session_id)now fetches all nodes into memory (old code usedlimit=100per depth). Fine for current DAG sizes but worth revisiting if they grow large.
LGTM — ready to merge.
ruangraung
pushed a commit
to ruangraung/hermes-lcm
that referenced
this pull request
Jul 20, 2026
stephenschoettler#413) * feat: add dry-run historical tool-output externalization backfill * feat: add bounded search over externalized payloads * feat: cap the protected fresh tail by tokens * feat: stub externalized tool results in active replay * fix: restore cooldown-first plain preflight * test: cover flag-off cooldown preflight paths * fix: report partial rollback failures * fix: harden externalized payload search * feat: add bounded atomic threshold full sweeps * feat: add temporal rollup storage substrate * feat: add embedding vector storage substrate * fix(vector-store): KNN correctness * fix(rollups): store-level correctness * fix(rollups): lazy feature tables + generation/lease + per-scope cursor (#387) Addresses maintainer Tosko4's #387 blockers 1-3 (and the #390 schema mirror). - Blocker 1 (schema versioning): revert core SCHEMA_VERSION to 5 and stop creating the opt-in rollup tables in run_versioned_migrations. RollupStore now creates them lazily+idempotently on the enabled path, recorded via the NAMED migration step "temporal_rollups_v1" (independent of the numeric counter). A disabled install creates no tables and stays base-readable. - Blocker 3 / #388 blockers 2-3 (build race): add a generation column + lease. upsert_building returns a RollupBuildToken and stamps lease_expires_at; mark_ready is a compare-and-set on generation (returns bool, superseded builds discard); mark_stale_for_day advances generation on every invalidation; reclaim_stale_building reclaims an expired-lease 'building' row to 'stale'. - Blocker 2 (scope identity): no stable conversation-family key is reachable from summary_nodes (they are session-keyed with no conversation column), so keep session-scoped rollups and make cursor state per-(period_kind, scope) instead of global-per-kind. Adds upsert_stale / stale_aggregates_for_day / record_incomplete_aggregate consumed by the builder and rebuild command. * fix(vector-store): revert core schema_version to 5 + canonical profile identity + durable data-version + chunked id lookup Maintainer #390 architecture-gate blockers (Tosko4): - Blocker 1 (schema versioning): stop bumping the core schema_version for opt-in embedding tables. SCHEMA_VERSION back to 5; create embedding tables lazily and idempotently from VectorStore init, recorded via the named 'embeddings_v1' migration marker. Disabled install stays v5 with no tables (no v6 collision with the temporal train). - Blocker 2 (profile identity): key profiles + vectors + meta on a canonical identity hash of (provider, model, revision, dim, dtype, byteorder, task). Re-registering a model under a new provider is a new profile (no clobber); switching config back reactivates the prior identity with its vectors intact. - Blocker 3 (numpy cache staleness + 33k-id lookup): durable per-identity data_version counter bumped in the same transaction as every vector write/delete and folded into the matrix cache key; autocommit connection so a cross-process write is observed. Replace the giant WHERE id IN (...) resolve with a chunked temp-table JOIN that scales past the SQLite variable limit. - KNN filter signature extended with until (time_to) + source, enforced before the top-k cap (wired from #394). * fix(vector-store): legacy-db compat, PK-index binds, per-call scratch, filter-before-bound, identity read path, purge wiring Address codex-bot round-2 findings on #390's VectorStore surface (fixes surfaced on #392/#394/#395 that live in vector_store.py land here too): - Legacy-DB compat: PRAGMA-fallback COALESCE(latest_at,created_at)->created_at and skip the source filter when messages.source is absent, so a VectorStore-only worker DB predating the DAG/source migrations does not crash time-/source-scoped KNN. - Bind the integer node_id (record_embedding + candidate JOINs) so the summary_nodes INTEGER PRIMARY KEY index is used instead of a CAST(node_id AS TEXT) scan. - Order the no-numpy candidate scan from the indexed embedded_at (no temp B-tree) and filter BEFORE the recency bound so a filtered match outside the recent window is not lost. - Unique per-call scratch temp-table name (dropped in finally) so concurrent candidate sets cannot clobber each other. - Resolve the active profile inside the write transaction in record_embedding. - Resolve KNN by the full provider identity, not model_name alone, so a provider A->B switch scores against the intended vectors (else degrades). - Wire purge_embeddings_for_nodes into on_session_reset and doctor clean apply (behind embeddings_enabled); orphaned embeddings were already kept out of ranking by the summary_nodes inner join. * fix(rollups): complete the generation/lease model + perf indexes + purge re-stale (#387) - upsert_building advances generation on re-claim so racing claims get distinct leases (only the latest claimant can publish); a fresh row stays 0 - mark_failed takes an optional generation compare-and-set so a superseded builder's late exception can't flip a newer ready/stale row to failed - replace record_incomplete_aggregate with token-guarded defer_incomplete and add resolve_no_source (CAS-delete) so a claimed contentless period is cleared rather than lingering stale and consuming a build slot - purge_rollups_for_sources re-stales affected windows (generation+1, sources cleared) instead of hard-deleting, so a window at/before the cursor is not orphaned and never rebuilt - indexes: lead ready/pending partial indexes with scope; add idx_lcm_rollup_sources_node for purge's node_id lookup * fix(rollups): status-guard terminal transitions + preserve rebuild lineage + verify feature schema (#387) A1: every generation-guarded terminal transition now also requires status='building' (mark_ready, guarded mark_failed, resolve_no_source DELETE), so once a row leaves 'building' no token-holder can transition it — closes the late mark_failed-after- mark_ready flip (mark_ready never advances generation). A2: upsert_building no longer clears lcm_rollup_sources at claim; the last-known-good lineage stays queryable until mark_ready atomically swaps it, so a concurrent purge-by-node still re-stales the rollup mid-rebuild instead of missing it and letting deleted-node content publish. A3: RollupStore init no longer trusts the temporal_rollups_v1 marker alone — new db_bootstrap.verify_temporal_rollup_schema verifies the required tables+indexes exist and re-ensures before recording the step, repairing a marker whose table was dropped. Also adds RollupStore.upsert_stale_many for the transactional rebuild seed consumed by #391. * fix(vector-store): durable identity through write CAS, fail-closed source filter, SQL-bounded enumeration, schema verify A1: record_embedding accepts a captured identity and publishes under THAT identity within the write transaction, never re-resolving to the active profile. A provider switch A->B mid-flight can no longer rebind an A-vector onto B; an unregistered captured identity is rejected, not silently rebound. A2: _source_allowed_ids fails CLOSED when messages.source is absent (provenance unverifiable) -- returns no allowed ids instead of treating "cannot check" as "all allowed", so a source-filtered query yields no false-positive legacy hit. A3: the dependency-free bounded path enumerates candidates via one SQL query with column filters in WHERE + ORDER BY embedded_at DESC + LIMIT, so neither the result set nor host memory materializes the whole corpus; the source-lineage walk runs on the already-bounded set. Removes unbounded _candidate_ids_by_recency. A4: _ensure_embedding_schema re-ensures AND verifies required tables/indexes every init (db_bootstrap.embedding_schema_missing), repairing a marker set over a dropped table rather than trusting the embeddings_v1 marker alone. * fix(rollups): enforce temporal store invariants (#387) * feat: build temporal rollups from the DAG in bounded maintenance passes Daily rollups keyed by each node's newest covered source-message timestamp (UTC, COALESCE(latest_at, created_at) per dag.py's source-window fallback); week/month aggregates built from ready dailies only; stable source fingerprints; escalation-backed token caps; failure isolation (a rollup failure never affects the turn). Two flag-gated engine hooks: post-ingest staleness marking and bounded catch-up (rollup_builds_per_pass, dailies before aggregates, oldest first) at the existing lifecycle-bind maintenance opportunity, sharing the engine's summary circuit breaker and spend guard. * fix(rollups): builder + engine safety * fix(rollups): staleness keyed to summary publication + aggregate completeness (#388) Addresses maintainer Tosko4's #388 blockers 1-5. - Blocker 1 (staleness signal): rollups consume PUBLISHED summary nodes, so the load-bearing invalidation now fires on summary-node publication (both _dag.add_node call sites on the engine — leaf compaction and condensation) via the new engine hook _invalidate_rollups_for_published_node, using the node's covered-message newest timestamp for the day. The raw-ingest hook is retained as an additional signal. - Blocker 2 (crashed build): run_rollup_maintenance calls reclaim_stale_building at the top of each pass so a crashed 'building' row is retried. - Blocker 3 (superseded build): builders capture the generation token at upsert_building and pass it to mark_ready. - Blocker 4 (source dedup): daily sources attribute each node to exactly one day (its newest covered timestamp), and aggregates draw source ids from the deduped ready-daily source rows, so a parent+child never double-count a span. - Blocker 5 (aggregate completeness): a week/month publishes ready only when every day WITH content in the window has a ready daily (else it stays stale with a recorded reason); a daily (re)build stales its containing week and month. * fix(rollups): capture-token-first builds, publication-only staleness, source dedup, scope gating (#388) - build_day/_build_aggregate claim the build token BEFORE reading their source snapshot, so an invalidation between snapshot and claim is superseded by the generation CAS; no-source periods resolve via resolve_no_source - mark_stale_for_deleted_nodes advances generation so an in-flight build can't publish deleted-node content - P1: remove the premature raw-ingest staleness hook; staleness is driven solely by summary-node publication (wired at every add_node site), so a rebuild can't publish 'ready' from old sources and omit a not-yet-published leaf - _daily_sources excludes condensed children when their parent is present the same day (no double-counting) - gate bind-time rollup maintenance off for bypassed/stateless sessions - capture the COMPLETE deleted-node id set (unbounded) instead of the first 1000 get_session_nodes returns - document rotation as the rollup scope boundary * fix(rollups): interval-aware coverage helper for daily frontier + publication staleness (#388) Introduce one shared interval-aware coverage helper in rollup_periods.py: covered_days() (UTC days a [covered_start, covered_end] interval intersects) and canonical_frontier()/CoverageNode (drop a child condensed by a higher-depth node in the candidate set). B1: _daily_sources/_days_with_content now derive from _scope_frontier — canonical_frontier over the WHOLE scope, each survivor assigned to one representative (latest_at) day — so a child covered by a parent that spans onto an adjacent day is suppressed and the parent feeds only its own day; adjacent dailies no longer duplicate the same covered leaf lineage. B2: mark_stale_for_published_summary now stales EVERY UTC day the node's [earliest_at, latest_at] coverage intersects (each day + its week/month) via covered_days, not only the latest_at day; engine passes node.earliest_at through. * fix(vector-store): enforce canonical bounded write contract * feat: add embedding provider warmup * fix(embedding-provider): resilience + safety * fix(embedding-provider): Voyage item cap + absolute retry deadline; Ollama truncate:false; FastEmbed query API Maintainer #392 architecture-gate blockers (Tosko4): 1. Voyage batches were bounded by tokens but not by the 1000-item per-request cap; embed_documents now splits on the item cap as well (configurable via config.embedding_max_batch_items, default 1000). 2. Retry timeout was per-attempt; replace with ONE monotonic deadline across all attempts and backoff so a 0.02s budget returns in ~0.02s, not ~1.5s. 3. Ollama requests set truncate:false so oversized input fails loudly instead of being silently truncated. 4. FastEmbed query embeddings use the query-specific API (query_embed) while documents use embed(), preserving query/passage task asymmetry. * fix(embedding-provider): one absolute deadline on normal ops; clamp Voyage item cap at 1000 B1: normal Voyage/Ollama embed_query/embed_documents pass deadline_budget_s= self.timeout, so a tiny configured timeout bounds every attempt AND its backoff (the interactive path already did this). A 0.02s budget with retryable failures now returns in ~0.02s instead of stacking 0.5s+1.0s backoffs to ~1.5s. B2: VoyageProvider clamps max_batch_items down to Voyage's hard 1,000-input cap regardless of config, so configuring 2000 can no longer emit a 1,001-input request. The provider's own hard limit is authoritative. * fix(rollups): enforce builder and invalidation invariants (#388) * feat: add natural-time rollup retrieval * fix(rollups): drop unreachable global scope * fix(rollups): lcm_recent requires full-window coverage + broader fallback (#389) Addresses maintainer Tosko4's #389 blockers 1-2. - Blocker 1 (completeness): lcm_recent serves rollup mode only when ready rollups cover the ENTIRE requested window. _recent_has_unready_rollups now detects MISSING days (no row), not only existing non-ready rows, and _recent_ready_rollups falls back for the whole window unless every expected period is ready. A 2-day window with one ready day falls back instead of serving a partial rollup. - Blocker 2 (fallback breadth): the leaf-summary fallback no longer restricts to depth-0 current-session leaves; it includes retained higher-depth/carry-forward summaries and spans the conversation session family (current + finalized) when lifecycle state is available. * fix(rollups): lcm_recent spans the fallback session set + overlap-based leaf window (#389) - rollup mode falls back when another session in the conversation family (current + last-finalized) holds overlapping window content, so it never drops a finalized session's content the leaf fallback would show - the leaf fallback filter includes any summary whose covered span INTERSECTS the window (earliest < end AND latest >= start), not only summaries whose newest timestamp lands inside it * fix(rollups): canonical frontier for lcm_recent fallback + provenance bound to returned sections (#389) C1: _recent_leaf_sections now selects all window-overlapping candidates and applies the shared canonical_frontier (reused from #388) BEFORE the limit, so a child contained by an overlapping selected parent is suppressed and does not consume a public limit slot ahead of independent summaries. C2: provenance.rollups is built by _bounded_recent_json from the sections actually RETURNED (in lockstep with the char-budget check) instead of every candidate rollup, and a bounded O(1) rollups_covered scalar carries the "N covered, showing M" aggregate — 1000 ready rollups + limit=1 now respects the response char cap. * fix(rollups): bound and fail-close recent retrieval (#389) * feat: add temporal rollup operator introspection * fix(rollups): rebuild durably seeds unattempted targets + docs (#391) Addresses maintainer Tosko4's #391 blocker. - Rebuild: `/lcm rollups rebuild <all|kind> [date]` now durably seeds a stale row (via the #387 store upsert_stale) for EVERY target in range BEFORE applying the per-pass build budget, so targets beyond the budget remain durable 'stale' rows (not absent) and get built by later maintenance. Unattempted targets report 'stale (bounded; not attempted)'. - Introspection: the rollup-status cursor read is filtered per scope, matching the #387 per-(period_kind, scope) cursor table. - Docs: reconcile the operator guide with actual behavior (lazy tables, publication-keyed staleness, aggregate completeness, honest scope/rotation boundary, per-scope cursor, durable-seed rebuild) and the retrieval-tools note (full-window coverage + retained-summary fallback). * fix(rollups): rebuild status reflects attempted failures + transactional multi-target seed (#391) D1: /lcm rollups rebuild no longer reports top-level 'status: complete' when an attempted target ended 'failed'; a failed attempted target downgrades to 'partial'. 'complete' is reserved for successful attempted targets plus intentionally-bounded not-attempted debt. D2: the multi-target stale-seed is now a single store.upsert_stale_many(...) transaction (all-or-nothing) instead of a per-target loop, so a mid-batch seed failure rolls back cleanly and cannot leave some targets seeded and others absent (no missing month with no row). * fix(rollups): make operator outcomes and output bounds honest (#391) * fix(embedding-provider): bound full operations and yield accepted batches * fix(embedding-provider): preserve authentication failures * feat: add embedding backfill command * fix(embedding-backfill): correct apply-mode — claim-before-discovery, heartbeat lease, truthful status, crash-safe in_flight, op budget Maintainer #393 architecture-gate blockers (Tosko4) — implement apply-mode correctly instead of retreating to dry-run-only: 1. Claim BEFORE discovery: acquire the lease first, then re-query pending rows so the batch is not stale. 2. Renewable heartbeat lease with owner-CAS refresh; only a truly-expired lease is stealable, and a stolen lease fails renewal and aborts the run. 3. Truthful status: never start at 'complete'; report partial/failed when any batch fails, complete only when all discovered work embedded (the probe's complete/embedded=0/failed=1 now reads failed). 4. Crash-safe exactly-once posture: mark rows in_flight before the provider call, clear on record_embedding; NOT-EXISTS discovery re-attempts crashed rows without re-billing already-recorded ones. 5. Operation-wide runtime budget + lease refresh cadence (LCM_EMBEDDING_BACKFILL_*). Also adapts the backfill discovery/profile queries to the identity-keyed schema. * fix(embedding-provider): backfill bypasses the interactive per-minute spend guard A large `lcm embed backfill --apply` embeds thousands of documents (~1920 docs at batch 32 -> ~60 provider calls) and tripped the 60/min interactive spend guard mid-run, stalling the bulk job. resolve_provider(config, for_backfill=True) now builds the provider with a disabled spend guard (max_calls=0); the backfill worker already has its own op budget + lease. The circuit breaker is retained, and interactive query embedding keeps the guard. * fix(embedding-backfill): re-check lease ownership after provider return; publish under captured identity C1: after the blocking provider call returns and before ANY publication (vector writes or in_flight clears), the backfill worker re-checks ownership via the lease's owner CAS (lease.renew(force=True)). A worker whose lease was stolen (TTL lapsed, successor took over) discards its result and exits cleanly instead of publishing under the successor's claim -- renewal-before-batch could not stop a stale owner writing AFTER the call. The lease id is a fresh uuid per acquire, so the owner CAS subsumes the generation check. A1 wiring: record_embedding is now called with identity=identity (the profile captured at lease-claim time), so an active-provider switch mid-run cannot rebind a vector onto the newly-active identity. * fix(embedding-backfill): atomically publish and quarantine uncertainty * fix(embedding-backfill): serialize schema and purge leases * feat: add semantic and hybrid embedding search * fix(embedding-search): degrade + latency * fix(embedding-search): enforce conversation/role/source/time filters before ranking + one absolute deadline Maintainer #394 architecture-gate blockers (Tosko4): 1. Filters were parsed but not enforced. Pass conversation_ids, source, time_from AND time_to into the store KNN filter so ineligible vectors are excluded before the top-k cap (no wrong-conversation hits; no eligible lower vector dropped for an ineligible top hit). conversation_id resolves to its sessions; role has no summary dimension so it degrades to full_text (which enforces role) rather than being silently ignored. 2. Replace the embed-only timeout with ONE absolute deadline across query embedding, vector-store construction and KNN; exceeding it degrades to FTS. Bounded worker ownership (BoundedSemaphore) so repeated timeouts do not accumulate live workers; the one-time numpy import is warmed outside the deadline. full_text output stays byte-identical. * fix(embedding-search): honor the raw-only lcm_grep contract in the semantic arm The advertised lcm_grep contract (schemas.LCM_GREP) returns raw-message hits only for broader scopes (all/session) and for role/time_from/time_to, and full_text treats conversation_id as a raw-only message-lane filter. A summary node has no single role/lane and is cross-session, so the semantic arm now degrades to the raw full_text path (which filters at the message-row level and reports degraded_to_fts) whenever session_scope != current, a time bound is set, or conversation_id is set — instead of resolving conversation->sessions and returning cross-session / wrong-lane summary hits. This aligns semantic with full_text and the #394 review. Also thread the provider identity into knn() so reads match the configured provider, not model_name alone. * fix(embedding-search): document the semantic-arm-only latency budget (D1) D1 decision: NARROW rather than enforce end-to-end. The full_text fallback is a synchronous, uncancellable SQLite path on the SHARED store connection (unlike KNN, which runs on a throwaway VectorStore connection), so it cannot be safely preempted by a wall-clock cutoff; and "degrade within budget" is self- contradictory because degradation is itself triggered by budget exhaustion. The enforceable contract is therefore semantic-arm-only: the deadline bounds the semantic attempt (query-embed + KNN); on exhaustion the tool degrades to full_text which runs to completion. Corrects the overpromising in-code comment; no behavior change (the semantic attempt was already deadline-bounded). * fix(embedding-search): enforce one whole-request deadline * fix(embedding-search): preserve bounded fallback paths * docs: free and local embeddings setup guide Voyage free tier (verified 2026-07, pricing page linked as source of truth), fastembed as the pip-only no-signup local default with an honest download-size note, Ollama recipe, per-provider config walkthroughs, cost table, performance/footprint numbers from the design benchmark, and provider-switch/re-backfill semantics. * docs(embeddings): align guide to corrected behavior Maintainer #395 architecture-gate notes (Tosko4): make every doc claim match the post-fix behavior. - Remove the lcm_status/doctor corpus-model-mismatch claim (no such surface was added). - Remove the external-reranker implication; retrieval is RRF-only. - Keep the 'slow operations degrade within the latency budget' claim, now true and accurate (the absolute deadline covers KNN, not just embed). - Replace the unverified 50k-vector figure with the validated fact: id/metadata resolution scales past the ~32k SQLite variable limit (validated to 40k). - Keep provider switching/reactivation (now true after the identity fix). - Re-verify Voyage free-tier numbers against the pricing page (2026-07). * docs(retrieval-tools): semantic arm degrades to raw full_text under the raw-only contract Update the retrieval-tools contract note: only source is enforced inside the semantic KNN (by descendant lineage); role/time/conversation and broader scopes now degrade the semantic arm to the raw full_text path. Record the FIXSPEC3 codex-bot dispositions in FIX-OUTCOME.md. * docs(embeddings): remove sqlite-vec claim; document semantic-arm budget, fail-closed source, never-mixed E1: remove the "sqlite-vec can be enabled" claim -- this train has no supported loader/extension-loading path for it. E2: keep "vectors from separate identities are never mixed" (now TRUE via A1) and make it explicit that every vector is published under the exact identity that produced it. Reconcile the latency-budget wording to the semantic-arm-only contract (D1: the fallback runs to completion) and document the fail-closed source filter (A2). * docs(embeddings): align setup and retrieval contracts * docs(embeddings): clarify bounded hybrid fallback * chore: drop internal dispatch artifact OUTCOME.md (not for upstream) * chore: drop internal dispatch artifact OUTCOME.md (not for upstream) * chore: drop internal dispatch artifact OUTCOME.md (not for upstream) * chore: drop internal dispatch artifact OUTCOME.md (not for upstream) * chore: drop internal dispatch artifact OUTCOME.md (not for upstream) * chore: drop internal dispatch artifact OUTCOME.md (not for upstream) * chore: drop internal dispatch artifact OUTCOME.md (not for upstream) * chore: drop internal dispatch artifact FIX-OUTCOME.md (not for upstream) * fix(embedding-search): degrade semantic/hybrid to full_text for non-history content_scope content_scope is a payload-search dimension owned by the full-text arm (externalized payloads are never embedded; embedded_kind='summary'), so a semantic request scoped to externalized content has no vector corpus to search. Previously the semantic arm silently ignored content_scope and returned history-only semantic hits — a contract violation on a combined head with the externalized-payload-search train. It now degrades to full_text with an explicit degraded_reason, matching the arm's existing role/time/conversation guards; in hybrid mode this surfaces as the full-text-arm result (which honors content_scope) plus the degraded marker. Regression: test_semantic_content_scope_degrades_to_full_text, test_hybrid_content_scope_degrades_to_full_text_arm. * fix(vector-store): bound candidates by source recency * Add LongMemEval retrieval-quality benchmark harness Offline-first harness scoring recall@k / NDCG@10 on LongMemEval_S (Wu et al., ICLR 2025) for the LCM retrieval arms: fts, summary_vectors, hybrid_rrf, and a deterministic embedding-cosine hybrid_rerank (cross-encoder placeholder). No LLM judge — the dataset labels evidence sessions, so recall is computable offline. - benchmarking/longmemeval.py: ingest each question history into a fresh temp LCM store (store/dag/vector_store APIs directly, no live host), deterministic per-session summaries, optional embedding backfill, per-arm scoring + aggregate-only report (no transcript content, session ids, or local paths). - scripts/lcm_longmemeval.py: fetch (pinned HF revision 2ec2a557f339b6c0369619b1ed5793734cc87533) + run subcommands; stub provider is deterministic/offline, fastembed is CI-grade, voyage gated. - tests: evidence scorer, metric math (R@k / NDCG / percentiles), CLI arg validation, end-to-end stub plumbing run. - benchmarks/README.md: usage + MemDelta (arXiv:2606.29914) honest caveat that benchmark rankings flip with embedding/base-model choice. * Extract shared retrieval core from tools.py (no behavior change) Factor the retrieval/fusion plumbing shared by lcm_grep into a new retrieval_core.py so the forthcoming lcm_recall tool reuses one ranking engine instead of duplicating logic. Pure move — every tool contract, JSON shape, degrade path, and error string is byte-identical. Moved: RRF fusion (rrf_fuse), the KNN invocation (run_knn) and node hydration, confidence scoring, deadline-error shaping, semantic conversation-scope resolution, and the FTS message/summary hit shapers. Guards (role/time/conversation/scope/content_scope), the bounded-worker deadline machinery, and provider-resolution/query-embedding stay in tools.py: their module-level names are monkeypatched through the tools namespace and the worker-slot default binding is lexical to that module. VectorStore is injected into run_knn so callers keep resolving it through tools.VectorStore (preserves test monkeypatch semantics). * test(embedding-search): enable externalization in content_scope degrade regressions On a combined head with the externalized-payload-search train, the degraded full-text arm actually executes the payload scan, which returns an honest error when externalization is disabled instead of degrade markers. Enable the flag in the fixtures so the tests assert the degrade contract identically on the standalone and combined heads. * fix(fts): run startup FTS integrity-check off the bind thread (#6) The throttled FTS5 deep integrity-check still blocked cold session binds on every cache-miss (first bind + each 24h expiry): ~10s on a 347k-row DB, up to ~2min cold (issue #235). On the startup path (`throttle=True`) we now run only the cheap structural check synchronously and dispatch the O(index) deep scan to a daemon thread that opens its OWN sqlite connection (never the store's, which is unsafe to drive cross-thread). The background scan does not rebuild: on corruption it records a `fts_integrity_failed:<table>` metadata flag that `/lcm doctor` surfaces with guidance to run the explicit repair path. - One scan at a time per (db, table): in-process registry + a persisted `fts_integrity_scan_started_at` stamp with a 15m staleness window so a crashed scan cannot wedge future binds. - Kill-switch `LCM_FTS_INTEGRITY_BACKGROUND=false` restores the exact old synchronous behavior; explicit `/lcm doctor repair` (throttle=False) is unchanged and fully synchronous. - `join_background_integrity_scans()` exposes the handles for deterministic tests; existing synchronous-behavior tests pin the kill-switch off. Live proof (347k-row real DB copy, markers cleared to force due): sync bind 10.26s -> async bind 1.72s (6x), background scan stamps the marker afterward. * test(embedding-search): give the fixture engine a home for combined-head payload scans The content_scope degrade regressions exercise the full-text payload scan on a combined head with the externalized-payload-search train, which resolves its directory from the engine home. The lightweight SimpleNamespace fixture engine lacks one; real engines always have it. * fix(vector-store): per-row created_at fallback in the source-recency bound Builds on the source-recency ordering fix (f9a240a): the DAG migration adds latest_at without backfilling legacy rows, and a bare latest_at DESC sorts those NULLs last — silently dropping legacy summaries out of the bounded candidate window on upgraded databases. COALESCE to created_at per-row so upgraded archives keep chronological ordering. Regression: test_bounded_scan_keeps_null_latest_at_legacy_rows_by_created_at. * feat(chunking): content-aware turn-aligned message chunker Add the chunk-corpus chunker: conversational/heads/full content policies, ~600-token turn-aligned windows with one-sentence overlap, error-signature extraction for tool results, and (store_id, chunk_index, char_span) records that map each chunk back to lcm_expand. * feat(db): lazy chunk_vectors_v1 schema (meta + vectors, verified) Add ensure_chunk_tables / verify_chunk_schema / chunk_schema_missing mirroring the embeddings_v1 discipline: lazy additive tables (never created on a stock install), structural verification over the marker, keyed (chunk_id, identity_hash) with chunk profiles sharing lcm_embedding_profile under task='chunk'. * feat(vector-store): chunk-corpus profiles, write, KNN, purge-archiving - Allow task='chunk' identities; task-scope profile activation so summary and chunk profiles coexist (each with its own active profile). - Extract corpus-agnostic _publish_under_lease from the summary publish CAS and reuse it for chunks (no forked concurrency logic). - Add lazy chunk_vectors_v1 schema init, _write_chunk_row, record/publish chunk embeddings, bounded-candidate knn_chunks with the full|bounded|none coverage contract (message-keyed: direct source column, no lineage walk), and archive_chunks_for_messages soft-archiving on message purge. * fix(db): detect interim-build schema-stamp and add guided remediation Databases touched by interim development builds carry a numeric schema_version ahead of this build's ladder while their actual schema is the v5 shape plus named feature markers. The generic 'upgrade the plugin' refusal was wrong for this case (no newer plugin exists). - classify_version_mismatch(): read-only v5-shape vs genuinely-newer classification, reusing verify_embedding/temporal_rollup helpers; errs toward genuinely_newer so an unrecognised shape is never downgraded. - refuse_schema_version_too_new(): interim-stamp refusal now names the remediation command; genuinely-newer keeps the restore-backup guidance. - remediate_interim_schema_stamp(): dry-run-by-default, backup-first apply (caller-owned backup), refuses genuinely_newer, never auto-downgrades. - /lcm doctor repair schema-stamp [apply]: opens the DB independently (read-only for preview), backup-first on apply, matching doctor UX. - Tests for classify, refusal guidance, dry-run/apply, genuinely-newer refusal, and the command path. Kept clear of db_bootstrap.py's FTS integrity-check region (owned by fix/async-fts-integrity) to keep the assembly merge clean. * feat(provider): chunk model mapping + contextualized-embed helper - default_chunk_model: voyage -> voyage-context-4; local providers reuse the configured model (local-first posture unchanged). - embed_contextualized dispatcher: uses a provider's embed_contextualized when present, else flattens to the plain embed_documents path and regroups by doc. (Live voyage-context-4 wire-shape is a follow-up; the plain fallback yields correct per-chunk vectors and is what the chunk backfill uses today.) * feat(backfill): /lcm embed backfill --corpus chunks|both --policy Extend the backfill command with a chunk corpus that REUSES the shipped lease/inflight/uncertain machinery unchanged (mark_inflight/mark_dispatched/ owned_inflight_transition key on (embedded_id, identity_hash) and are corpus- agnostic). Chunk discovery streams policy-chunked messages; apply publishes via publish_chunk_embedding_under_lease with the same CAS/crash-safety semantics. Default --corpus summary is byte-identical (summary path untouched). Adds LCM_EMBED_CONTENT_POLICY config. Dry-run works without a registered chunk profile (estimates over default_chunk_model). * fix(db): classify interim stamp by core shape; drop early feature caches Real-data acceptance against the actual interim-stamped operator DB showed the reused final-shape verifiers rejected EARLY feature-table variants (lcm_rollups without generation/lease_nonce/failed_at and no lcm_rollup_invalidations; lcm_embedding_profile keyed on model_name without identity_hash/data_version), misclassifying a genuine interim stamp as genuinely_newer. - classify_version_mismatch: the numeric stamp only certifies CORE shape. interim_stamp = core v5 matches exactly AND every extra table is a known family prefix (lcm_rollup/lcm_embedding/lcm_chunk) or FTS shadow — regardless of the feature tables' internal shape (owned by each feature's own marker-gated init/verify). genuinely_newer stays for core mismatch or any non-family extra table. - remediate apply now drops each family whose final-shape verifier fails (derived caches: rollups rebuild from the DAG, vectors re-backfill), including family-owned triggers, before resetting the stamp. Dry-run lists what would be dropped with rebuild hints; passing families are never touched. - /lcm doctor repair schema-stamp surfaces would_drop/dropped lines + hints. - End-to-end test: early-variant fixture -> interim_stamp, dry-run lists drops, apply drops + resets, then refuse passes and RollupStore + VectorStore reconstruct the final shape clean. Hard acceptance on a fresh copy of the real operator DB: classify=interim, apply drops the 6 early tables + resets to v5, refuse passes, and both feature stores construct clean (verifiers return []). * feat(purge): archive chunks when messages are deleted or GC-rewritten Add engine._archive_chunks_for_messages mirroring _purge_embeddings_for_nodes (best-effort, embeddings-gated). Wire it into the retention session-scope message delete (same transaction, via archive_chunks_for_messages_on_connection) and into transcript-GC tool-result rewrites (stale-content chunks archived). * feat(recall): chunk-KNN + rerank plumbing for lcm_recall - retrieval_core: run_chunk_knn (chunk-corpus KNN, mirrors run_knn) and hydrate_chunk_hits (chunk_id -> store_id/span/excerpt message-excerpt hits, keyed by store_id so RRF fuses against FTS raw hits). - embedding_provider: VoyageProvider.rerank (rerank-2.5-lite, one API call, single absolute deadline; raise=skip). - config: rerank_enabled (LCM_RERANK_ENABLED, default off). * feat(recall): lcm_recall tool + fused cross-conversation pipeline Forever-memory surface: one tool searching the entire local database by meaning across all conversations. Three arms via retrieval_core (no duplicated plumbing) — FTS raw (all sessions) + summary KNN (no filter) + chunk KNN (no filter) — RRF-fused (chunk hits dedupe against FTS by store_id), optional voyage rerank-2.5-lite (default off, skips silently on failure), then a soft scope_bias + 30d-half-life recency prior (boosts, never filters). Bounded, char-capped, honest degrade matrix (embeddings-off => FTS-only). Adds the LCM_RECALL schema and the lcm_grep forward-pointer sentence. * feat(recall): register lcm_recall across engine, plugin + docs surfaces Wire lcm_recall into engine.get_tool_schemas (after lcm_grep) and the handle_tool_call dispatch map, the __init__ Path-A _TOOLS registry list, the plugin.yaml provides_tools manifest, and the README + docs/retrieval-tools.md tool tables (keeps the tool-contract sync + documentation tests green). * test(recall): fused-pipeline coverage for lcm_recall Seeds summaries + chunks + raw messages across three synthetic sessions and asserts cross-session recall without a filter, scope_bias + recency boosts (not filters), chunk/FTS store_id dedupe, include filtering, rerank apply/ skip/failure fallback, and the embeddings-off degrade to the FTS arm. * fix(recall): scan the full corpus, not grep's 2000-recent window lcm_recall promised 'all conversations, all time' but the summary/chunk KNN arms inherited lcm_grep's embedding_bounded_scan_rows (2000), which enumerates only the 2000 MOST-RECENT vectors — structurally hiding the oldest memories. On the 3,683-summary real DB the canonical DASHBOARD SPRINT v1.5.1 / Fleet v1.0 archives (recency rank ~3.4k) were unreachable. Thread an optional scan_rows override through run_knn/run_chunk_knn (default None => unchanged, lcm_grep stays byte-identical) and give recall its own config bound (recall_scan_rows, LCM_RECALL_SCAN_ROWS, default 25000, still deadline-guarded). Live smoke now returns coverage=full and recalls both archives cross-session. * test(recall): guard that both vector arms scan the full recall bound Locks in the 'all time' contract: recall's summary + chunk KNN arms must pass recall_scan_rows (not grep's embedding_bounded_scan_rows) to the VectorStore, so the oldest memories stay reachable. * fix(doctor): surface background integrity-scan failure flags in the lcm_doctor tool Mirrors the /lcm doctor text path: a prior non-blocking background integrity scan persists fts_integrity_failed:<table> when it finds corruption without rebuilding; the JSON doctor tool now reports it as a fail-class check with the explicit repair guidance (deferred follow-up from fix/async-fts-integrity, applied at train assembly once tools.py was free of concurrent agent ownership). * test: lcm_recall joins the declared tool set The registration/host-capability suites enumerate the exact declared tool set; lcm_recall (new in the recall train) registers correctly and belongs in the expectation. * feat(benchmark): longmemeval chunk_vectors arm + hybrid_rrf3 fusion Add a raw-chunk KNN retrieval arm to the LongMemEval harness (predates the chunk substrate): ingest now runs the conversational chunker + records chunk embeddings into the temp store, and chunk_vectors scores evidence-hit@k by mapping chunk hits (store_id:chunk_index) back to their sessions. hybrid_rrf3 fuses it as a third arm alongside FTS + summary vectors. Existing arms keep byte-identical outputs; stub/fastembed providers work offline. * fix(embed): route Voyage context models to the contextualized endpoint Context models (voyage-context-3/-4) exist ONLY on /v1/contextualizedembeddings; posting them to the flat /v1/embeddings endpoint returns HTTP 400 (live-proven: chunk backfill wrote zero rows). Add VoyageProvider.embed_contextualized plus a _contextualized_request that reuses the flat path's retry/deadline/cap discipline with an injected nested inputs payload and order-preserving nested-response parser (usage token accounting from the response). The document-batch and query paths now route context models through the contextualized endpoint (each document = one single-chunk input list; query = single-item list, input_type=query per the docs), and the flat payload path structurally refuses a context model so the 400 can never be re-emitted. Non-context Voyage models and fastembed/ollama are unchanged. Mocked wire-shape tests cover request body shape, nested response parsing, order preservation across mixed batch sizes, and the flat-endpoint regression guard. * fix(embed): honor an explicit context model for the chunk corpus default_chunk_model mapped Voyage to voyage-context-4 unconditionally, so with LCM_EMBEDDING_MODEL=voyage-context-3 the chunk-backfill dry-run displayed voyage-context-4 while apply used context-3. An explicit context model is the operator's stated chunk-model intent and now wins; the voyage-context-4 mapping applies ONLY when the configured model is a plain (non-context) voyage model. Dry-run display and apply resolve through this single path, so they agree. Tests lock the resolver (explicit context wins, plain maps) and the dry-run report model line for both cases. * fix(recall): correct chunk/FTS ranking bugs (RRF-1, F1-chunk-dedupe, DEDUPE-1, RERANK-1) - rrf_fuse collapses a repeated identity within one arm to its best rank, counting a single 1/(k+rank) term (a multi-chunk message no longer double-counts and out-scores a genuine higher-rank match) [RRF-1]. - chunk-vs-FTS dedupe keeps the best-ranked chunk per store_id via setdefault over the best-first list, not the worst (last) entry [F1-chunk-dedupe]. - the merged hit adopts the better-ranked arm's snippet AND offsets together so preview and expand handle describe the same span [DEDUPE-1]. - rerank window is selected AFTER the scope/recency prior and rerank is a pure rank-reorder within the window; voyage scores are never spliced onto the RRF scale; ordering semantics documented in response provenance [RERANK-1]. - scope boost eligibility spans the conversation-scope session set so a session rotation keeps the boost [SCOPE-1]. * fix(db): never re-stamp/drop a DB whose feature table has an extra column F2-schema-stamp-drops-newer-data: the interim schema-stamp classifier only checked the core-table column contract and family-table *names*, never a family table's internal shape. A future release that adds a column to a family table (e.g. lcm_rollups) was misclassified as an interim stamp, so remediate_interim_schema_stamp(apply=True) DROPPED the table and its siblings, destroying real data (reproduced empirically). classify_version_mismatch now runs each present family's final-shape verifier and returns genuinely_newer when it reports an unexpected (extra) column — the unambiguous newer-build signature. Early variants only ever OMIT later-added pieces (distinct missing findings) and stay interim, so drop-and-rebuild remediation is unchanged. _interim_family_drops gains the same guard as defense in depth. Renamed-away early columns collapse to a malformed-table finding on the embedding/chunk verifiers (not unexpected-column), so the real interim operator DB still classifies interim and remediates clean. * fix(db): verify and remediate the lcm_chunk feature family on schema-stamp reset F2-schema-stamp-chunk-family-missing / F4-chunk-family-verifier-missing: _KNOWN_FEATURE_TABLE_PREFIXES allowlisted lcm_chunk but _INTERIM_FEATURE_FAMILIES and _family_verifier had no entry for it, so remediate_interim_schema_stamp reported status: ok / dropped_tables: [] while leaving a broken lcm_chunk_meta / lcm_chunk_vectors untouched — the next chunk backfill then hit a hard OperationalError with no doctor warning. Register the chunk family wired to the existing verify_chunk_schema so an early/broken chunk schema is dropped and rebuilt like the rollup and embedding families. * fix(db): clear the FTS integrity-failed flag on any successful repair F1: repair_external_content_fts never cleared fts_integrity_failed:<table>, so after `/lcm doctor repair apply` succeeded `/lcm doctor` kept reporting issues-found forever and the next self-healing scan was pushed out a full LCM_FTS_INTEGRITY_CHECK_INTERVAL_HOURS. Clear the flag in the same transaction that commits the rebuild (and on the low-disk degrade-to-LIKE path, whose corrupt index is removed outright). * perf(recall): pool VectorStore + bounded-LRU matrix caches (F2-matrix-cache, sprint-opt-6) - retrieval_core keeps a small LRU pool (cap 2, keyed by db_path+scan_rows) of long-lived VectorStore instances so their matrix caches survive across calls instead of being cleared by a build+close per call; data_version keys still invalidate on any committed write, and a per-store lock serializes the shared connection. Pooling is opt-in via VectorStore._supports_pooling so injected test doubles stay transient [F2-matrix-cache-never-persists]. - _matrix_cache / _chunk_matrix_cache are bounded LRUs (move_to_end + evict oldest) instead of clear-on-every-miss, so distinct candidate sets coexist and a warm identical query hits [sprint-opt-6]. Note: kept the bounded-candidate load rather than the suggested full per-identity matrix cache -- a full 261k-chunk float32 matrix (~1GB) conflicts with SCAN-1's bounded-scan memory guard, and pooling already delivers the survive-across-calls win. Synthetic 5k-vector back-to-back recall: cold 44.4ms median -> warm 20.6ms median (~2.2x). * fix(db): join background scans before repair; only fail on real corruption F3: _doctor_repair_apply_text now calls join_background_integrity_scans() before repairing, so a scan still mid-flight cannot error out afterward and re-write a fresh fts_integrity_failed marker over a just-successful repair (F1's stuck false-positive via a race). check_external_content_fts_integrity narrows its DatabaseError handling: only corruption signatures (malformed / disk image / not-a-database) classify as 'fail'; a transient lock/busy/timeout now returns 'unchecked' instead of recording a false corruption flag. * fix(db): dispatcher stamps scan_started_at before starting the scan thread F6: thread.start() returns before the spawned scan thread commits its own scan_started_at stamp. A second process racing ensure_external_content_fts in that window read no stamp and dispatched a duplicate concurrent deep scan (doubling I/O, raising the odds of the lock-timeout misclassification F3 addresses). The dispatching process now writes the stamp itself under a quick BEGIN IMMEDIATE (best-effort) before thread.start(), durably claiming the scan cross-process. * fix(recall): surface bounded coverage + degrade/expand-handle docs (SCAN-1, F4-degrade, F3-expand-hint) - KNNResult carries scanned/total; a coverage='bounded' arm now adds a degraded_reasons entry naming the arm and the scanned/total ratio (a cheap single-table COUNT once per bounded arm) so recency-truncated corpus coverage is visible instead of silent [SCAN-1]. - lcm_recall runs the FTS arm whenever embeddings are disabled regardless of include, so include='summaries' + embeddings-off degrades to full-text hits instead of returning nothing [F4-degrade-to-fts-false-for-summaries-only]. - schemas.py + docs/retrieval-tools.md qualify the expand handle: verbatim/ current-session hits get lcm_expand, cross-session summary hits get lcm_load_session [F3-expand-hint-not-lcm-expand-cross-session]. * fix(engine): archive a message's chunks atomically with its GC content rewrite F2: _maybe_gc_compacted_tool_results committed each tool result's content rewrite immediately (store.gc_externalized_tool_result) but archived chunks only once for the whole batch after the loop. A recall landing in that gap sliced the new (short) placeholder content at the OLD chunk char offsets, silently returning '' or a garbled fragment. gc_externalized_tool_result now takes a before_commit hook that runs after the rewrite and before its single commit; the engine passes a hook that archives that row's chunks on the same connection, so the rewrite and the chunk archive land in one transaction (per-message, not post-batch). * perf(recall): dedicated LCM_RECALL_QUERY_TIMEOUT_S budget (sprint-opt-2) lcm_recall fans out three arms + fusion/hydration/rerank, so it gets its own recall_query_timeout_s (default 8.0s, env LCM_RECALL_QUERY_TIMEOUT_S) instead of sharing lcm_grep's single-arm embedding_query_timeout_s (left at 3.0s). * perf(recall): batch chunk hydration into one JOIN (F4-chunk-hydrate-n-plus-1) hydrate_chunk_hits collects the rank-ordered chunk ids (bounded to knn_limit) and resolves them with a single IN(...) JOIN per bounded batch instead of one SELECT per ranked hit, then re-emits hits in KNN rank order. * fix(embed): persist real chunk char spans on --retry-uncertain, reject zero spans F3-uncertain-retry-zero-span: _chunk_authorized_uncertain_rows hardcoded (store_id, chunk_index, 0, 0) for chunks recovered via --retry-uncertain, discarding the real span _rebuild_chunk_document had already computed. Those zeros flowed into lcm_chunk_meta, so later hydration did content[0:0] (empty snippet) and handed callers lcm_expand(content_offset=0) — the start of the whole message, not the matched span. _rebuild_chunk_document now returns the real char_start/char_end and the uncertain-retry path persists them. Both chunk row-selection seams reject char_end<=char_start as defense in depth so a degenerate span can never reach lcm_chunk_meta. * perf(vector_store): skip no-op migration-marker re-writes (sprint-opt-1) _ensure_embedding_schema / ensure_chunk_schema read the migration-state marker before writing: when the step is already stamped and the schema verifies clean, skip the marker re-write + commit (an otherwise per-construction write-txn contending with real writers). CREATE-IF-NOT-EXISTS + verify stay read-only no-ops on a materialized schema. Note: the db_bootstrap.py run_versioned_migrations / mark_migration_step_complete portion of sprint-opt-1 is owned by the sibling agent and left untouched here. * test(recall): cover voyage rerank provider + JSON doctor bg-flag (F2-voyage-rerank, F1-json-doctor) - FakeTransport tests for VoyageProvider.rerank: happy-path ordering + single call, out-of-range index dropped, non-2xx raises VoyageError, empty-documents short-circuits with no transport call [F2-voyage-rerank-provider-untested]. - direct test of the JSON lcm_doctor MCP tool surfacing a pre-recorded background FTS-integrity flag in its checks list [F1-json-doctor-background-flag-untested]. * fix(embed): gate chunk-corpus raw-text sends behind --confirm-raw-text; document corpora F1-chunk-consent-gap: the chunk corpus embeds RAW, VERBATIM message text (tool-result + error/traceback content — exactly what tends to carry secrets), unlike the summary corpus, yet it was gated only by the same embeddings_enabled bool and was undocumented. --corpus chunks|both --apply now refuses on a CLOUD provider unless --confirm-raw-text is passed; local providers (fastembed/ollama) are exempt. The refusal explains the raw-text exposure and that LCM_SENSITIVE_PATTERNS_ENABLED redaction runs at ingest (already-stored text is not retro-redacted). Also documents --corpus/--policy/--confirm-raw-text in the operator-guide command reference and a new chunk-corpus section in both docs, incl. the note that the chunk corpus needs its own backfill run (F5-backfill-corpus-policy-undocumented). * fix(embed): emit one coherent next-hint for --corpus both dry-run F6-backfill-both-next-hint-incoherent: a --corpus both dry-run concatenated the summary and chunk reports, each with its own 'next' hint — '/lcm embed backfill --apply' and '/lcm embed backfill --corpus chunks --apply' — neither of which is the --corpus both --apply command that reproduces the preview, so an operator copying the first prominent hint silently ran a summary-only apply. The per-corpus reports now suppress their next-hint in the both path and the both branch emits a single hint naming the actual --corpus both --apply invocation (with the --confirm-raw-text note for the chunk corpus). * fix(db): treat fts5 checksum-mismatch as corruption in the integrity classifier Follow-up to F3: the FTS5 integrity-check reports same-row-count stale drift as 'fts5: checksum mismatch for table ...', which the initial corruption-signature list (malformed/disk image/not-a-database) missed — flipping a genuine drift 'fail' to 'unchecked'. Add 'checksum mismatch' so drift is still classified as corruption while transient lock/busy errors stay 'unchecked'. * feat(retrieval): rrf_fuse gains optional per-arm weights (default 1.0 ≡ old) * feat(config): LCM_RECALL_ARM_WEIGHTS lenient parse (default fts=0.5,summary=1,chunk=1) * feat(recall): weight lcm_recall RRF arms from config, echo weights in provenance * docs(retrieval): document LCM_RECALL_ARM_WEIGHTS format + −21pt rationale * test(recall): lock weighted-RRF property, byte-identical defaults, lenient env, provenance echo * batch-publish: one transaction per accepted network batch (F5) Wrap each accepted network batch's per-row publications in a single BEGIN IMMEDIATE (one fsync) instead of one write-transaction per row (93k-261k txns per backfill; 32x amplification). Each row still runs the full ownership/identity CAS under its own SAVEPOINT via the unchanged single-row publish_*_under_lease, so a mid-batch failure rolls back only its row and quarantines its request while committed siblings survive. _write_transaction is now re-entrant: the outer entry owns BEGIN/COMMIT, a nested entry rides it under a savepoint. command.py summary + chunk apply paths consume publish_*_batch_under_lease per network batch with byte-identical accounting. Ownership is verified per-row via CAS inside one serialized transaction; a takeover is now observed at batch boundaries, not mid-transaction (the external steal serializes after the batch commit). Rewrites the row-interleaved takeover test accordingly (contract preserved and strengthened). * tests: batch-publish crash-atomicity, single-transaction, mixed-superseded - single-transaction: trace-callback proves one BEGIN IMMEDIATE + five SAVEPOINTs for a 5-row network batch (the fsync win) - crash-mid-batch: a commit that fails between provider return and COMMIT rolls the whole batch back; nothing half-published, all rows stay in-flight and recoverable - mixed-superseded: a row superseded mid-batch stops the batch without aborting the rows already published in the same transaction * harness v2: turn-level scoring, real Voyage rerank, ingest batching (B4) Three SPEC-M upgrades to the LongMemEval retrieval harness. (Interwoven in evaluate_question/run_harness, so landed as one cohesive commit rather than a split that would break the shared refactor.) 1. Turn-level scoring: every arm now reports recall@k/NDCG@10 at turn granularity alongside session level. Raw FTS + chunk hits localize to a turn via store_id->(session,turn); summary hits score at session granularity, flagged with `session_granularity`/`*` so the coarse number is never mistaken for exact localization. New turn_recall_at_k/turn_ndcg_at_k coverage scorers. 2. Real rerank arm: `--provider voyage --rerank` reranks the top-20 fused sessions with VoyageProvider.rerank (rerank-2.5-lite) in one call under a 10s budget, falling back to the labeled placeholder-cosine reranker on any error or offline. Mode recorded in JSON (`rerank.mode`) + markdown header. 3. Ingest batching (F7): a question's session summaries embed in one batched call instead of one-per-session; a pre-migrated DB template is cloned per question instead of re-running schema bootstrap. Per-question ingest timing reported. Raw chunks stay per-item: local ONNX pads a batch to its longest text, making chunk batching slower (~0.4x on bge-small); the summary-call collapse is the >=3x win on the network/live-provider path. Tests: +9 (turn scorers, chunk turn-localization, voyage rerank reorder/fallback + real-mode label, db-template==from-scratch parity); existing harness tests green (25 passed). ruff clean. Full-suite FAILED set unchanged. Only touches benchmarking/longmemeval.py, scripts/lcm_longmemeval.py, tests/test_longmemeval_harness.py, benchmarks/README.md. * fix(vector-store): task-scope _resolve_profile so chunk registration cannot hijack summary knn Registering the chunk-corpus profile for the same (model, provider) made the shared _resolve_profile — which orders by registered_at with no task filter — return the chunk identity to the SUMMARY knn path, yielding zero summary candidates and coverage 'none'. Every harness run since the chunk arm landed silently lost its summary_vectors arm (0.84 → 0.00) while unit fixtures stayed green (single-profile setups). Both branches of the resolver now filter task='summary', matching the already task-scoped _current_profile/_resolve_chunk_profile. Regression: test_summary_knn_survives_chunk_profile_registration. * fix(benchmark): total-order tie-break for fused turn keys A summary turn key (session, None) tying a localized (session, int) key on both RRF score and best rank crashed the fusion sort (None < int). Map None turns to -1 in the tie-break only; regression locks the tie. Surfaced at assembly — the authoring environment's tie patterns never triggered it. * benchmark(results): v2 LongMemEval_S 500q matrix (fastembed bge-small) Committed the config-exact retrieval-quality numbers behind the v2 default decisions: chunk arm session R@5 0.96 (0.99 knowledge-update, 0.91 temporal) beats summary 0.87 across every category; rerank shows no lift over summary at this scale (stays default-off); the temporal fact-layer is a measured NO-GO (the categories that need it in the industry are already top-scoring here on lossless+chunk). Aggregate metrics only — no transcript content (MemDelta caveat noted). * fix(retrieval): reject negative arm weights that invert RRF config._parse_arm_weights now drops a negative env weight (keeping the arm's default, with a logged warning); retrieval_core.rrf_fuse clamps a negative weight to 0.0 so the arm drops out cleanly instead of making a rank-1 hit score negative and inverting rank-monotonicity. Weight 0.0 stays legal (drops the arm). Adds regression tests for both paths. * fix(benchmark): treat empty/degenerate voyage rerank response as fallback rerank_sessions_voyage now validates the provider response covers every candidate index; an empty (data: []) or partial-coverage response is routed to the placeholder fallback (same path as a provider error) instead of being silently accepted and counted as a real voyage rerank. Adds empty-response and partial-coverage regression tests. * fix(benchmark): aggregate harness rerank mode across questions run_harness previously reported rerank.mode from only the final question (last-question-wins), so a run where some questions silently fell back to the placeholder while others used real voyage rerank was mislabeled. Now per-question modes are counted and collapsed: real only if ALL questions used voyage, else mixed (with real/placeholder counts surfaced in the metrics JSON), else placeholder. Adds mixed + all-real run tests. * fix(backfill): label + count batch-publish local failures as local_error When the batch publish CALL itself fails (SQLITE_BUSY on BEGIN IMMEDIATE or a commit I/O error), the generic except (a) mislabeled it provider_error though it is a LOCAL storage/lock failure and (b) under-counted it -- failed_indexes = request_indexes - accepted_indexes was empty because accepted_indexes was populated pre-publish, so the unpublished rows were omitted from the run's failed count. Both the summary and chunk paths now wrap the publish call in its own try/except: every accepted-batch row is appended to failed with a local_error reason, a _LocalPublishError carries that reason to the outer handler so the inflight quarantine text is routed correctly, and rows still go uncertain via the inflight fallback. Adds summary + chunk crash-on-commit tests. * feat(vectors): int8/binary storage schema + config surface Add lcm_embedding_binary/lcm_chunk_binary sign-bit tables (additive; empty for float32 identities) and config LCM_EMBEDDING_STORAGE_DTYPE / LCM_EMBEDDING_STORE_DIM / LCM_KNN_PRESCREEN_MULTIPLIER. float32 vectors stay byte-identical; dtype/store_dim are already profile-identity components. * feat(vectors): int8 dtype-aware storage + two-stage binary-prescreen KNN _encode/_decode_int8_vector + _pack_sign_bits codecs; dtype-aware write path (int8 -> quantized vec blob + sign-bit prescreen row, float32 unchanged); full-corpus two-stage KNN (Hamming top-M prescreen -> int8 cosine rescore) in knn/knn_chunks, coverage='full'; LCM_KNN_PRESCREEN_MULTIPLIER; binary purge cleanup; new test suite (round-trip, identity isolation, stage-1 recall@4k>=0.98 on synthetic 5k, coverage, Matryoshka store_dim). * feat(backfill): honor LCM_EMBEDDING_STORAGE_DTYPE/STORE_DIM + --dtype guard Warmup registers the profile with the configured storage dtype + Matryoshka store dim (reported in output); chunk backfill reconstructs the captured identity from the registered profile dtype instead of hardcoding float32 (so an int8 chunk identity backfills correctly); add `--dtype float32|int8` backfill guard that refuses a run mismatching the registered profile. * feat(vectors): float32 prescreen opt-in + binary-matrix caching LCM_EMBEDDING_BINARY_PRESCREEN writes the sign-bit prescreen for a float32 identity too (distinct identity via revision), giving the full-corpus two-stage path with EXACT float rescore of survivors -- the high-recall, low-RAM config (real-data recall@10=0.96 vs int8-rescore 0.84 on a duplicate-dense corpus). Cache the unfiltered full-corpus binary matrix per (identity, data_version) so a pooled store's back-to-back recalls skip reloading the sign-bit corpus (warm p95 ~70ms vs ~340ms float full-scan). Adds float32+binary test. * feat(chunking): expose per-document chunk grouping (group_by_store_id) Group a flat chunk sequence into per-message documents (contiguous runs by store_id), preserving store and within-message order. This is the per-document grouping the contextualized (cross-chunk) embedding path consumes so a message's chunks are embedded together. * feat(embedding): contextualized chunk grouping with cap-aware request planning voyage-context-* now groups a document's chunks into one inputs inner-list so the context model contextualizes them together. Adds: - _plan_contextualized_requests: splits oversize documents (>120k/doc) into contiguous sub-documents and packs sub-documents into requests bounded by the official caps (120k tokens, 16k chunks, 1000 inputs per request). - embed_contextualized: now split/pack-aware, reassembling per-chunk vectors. - embed_chunk_group_batches + supports_contextualized_grouping: a batches-style API for the backfill that keeps the before_dispatch/lease crash-safety contract and yields per-chunk flat indexes+vectors (each chunk still published per-row); oversize single chunks are skipped via last_skipped_documents. Non-context / plain-voyage / local providers are unchanged (flat per-chunk). * feat(backfill): route chunk backfill through grouped contextualized path The chunk backfill now groups each message's chunks (by store_id) into one voyage-context-* inputs inner-list so the context model actually contextualizes them, instead of sending each chunk as its own single-item list (zero cross-chunk benefit). The batch-publish + lease/inflight path is preserved: every chunk is still an independently-published row, before_dispatch still marks each netwo…
stephenschoettler
pushed a commit
that referenced
this pull request
Aug 3, 2026
The throttled FTS5 deep integrity-check still blocked cold session binds on every cache-miss (first bind + each 24h expiry): ~10s on a 347k-row DB, up to ~2min cold (issue #235). On the startup path (`throttle=True`) we now run only the cheap structural check synchronously and dispatch the O(index) deep scan to a daemon thread that opens its OWN sqlite connection (never the store's, which is unsafe to drive cross-thread). The background scan does not rebuild: on corruption it records a `fts_integrity_failed:<table>` metadata flag that `/lcm doctor` surfaces with guidance to run the explicit repair path. - One scan at a time per (db, table): in-process registry + a persisted `fts_integrity_scan_started_at` stamp with a 15m staleness window so a crashed scan cannot wedge future binds. - Kill-switch `LCM_FTS_INTEGRITY_BACKGROUND=false` restores the exact old synchronous behavior; explicit `/lcm doctor repair` (throttle=False) is unchanged and fully synchronous. - `join_background_integrity_scans()` exposes the handles for deterministic tests; existing synchronous-behavior tests pin the kill-switch off. Live proof (347k-row real DB copy, markers cleared to force due): sync bind 10.26s -> async bind 1.72s (6x), background scan stamps the marker afterward.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This removes the global tool-engine singleton and routes tool calls through the active engine instance instead.
What changed:
handle_tool_call()now passesengine=selfinto the tool handlerslcm_describeandlcm_expandnow reject nodes outside the current sessionWhy this matters:
Tests:
python3 -m pytest tests/test_lcm_engine.py tests/test_lcm_core.py -qThis should make the plugin behave much better inside Hermes gateway-style runtimes.