feat: ingest_log parallel-write phase (#205) - #260
Merged
robotrocketscience merged 8 commits intoApr 28, 2026
Conversation
) v2.0 first slice of write-log-as-truth (docs/design/write-log-as-truth.md): add ingest_log as a parallel-write table. No view-flip yet — beliefs and edges remain authoritative; this commit just lands the schema and API so subsequent commits can wire entry points. Schema (additive, additional table; no existing-data migration): - ingest_log(id ULID PK, ts, source_kind, source_path, raw_text, raw_meta JSON, derived_belief_ids JSON, derived_edge_ids JSON, classifier_version, rule_set_hash, session_id) - idx_ingest_log_source(source_kind, source_path) per spec - idx_ingest_log_session(session_id) for cross-cut with #192 Decisions per approach memo (Kulili, 2026-04-28): - D1: hand-rolled monotonic ULID (stdlib only, ~50 LOC, seedable) - D2: store raw_text inline (dedup deferred to v2.1 if storage bites) - D3: legacy_unknown source_kind reserved for migration (not yet wired) - D4: single-transaction parallel writes (cheap; existing pattern) - D5: reachability check is the v2.0 contract; full-equality is v2.x - D6: latency budget ≤15% — measured in commit 6 - Open Q: session_id added now to avoid a later migration Bootstrap migration: drops a pre-#205 stale empty experimental ingest_log table (id INTEGER PK, raw_meta_json) found on some local stores. Refuses to drop if non-empty. 17 hypothesis-driven tests pass; full suite 1740 passed, 8 skipped.
Mirrors #204's belief/edge version-vector pattern for ingest_log: - log_versions(log_id, scope_id, counter) sibling table - _bump_log_version called on every record_ingest - get_log_version_vector accessor - _maybe_backfill_log_version_vectors stamps {local_scope: 1} on pre-existing log rows; idempotent via schema_meta marker Per spec: 'ingest_log rows should carry the same VV primitive in one migration window' (memo D7 / open question). v3 federation reconcile will treat log rows as first-class replication units. 2 hypothesis-driven tests pass; full suite 1744 passed.
First two of six entry points wired to the v2.0 parallel-write log: - scan_repo: each persisting candidate becomes one log row with source_kind=filesystem, source_path=candidate.source. - ingest_turn: each persisting sentence becomes one log row with source_kind=filesystem, source_path=source, session_id stamped. Per spec, log row is written BEFORE belief insert. derived_belief_ids is known up-front because belief_id is deterministic on (source, text). Re-ingest of identical content skips both belief insert and log write — corroboration row still records the re-assertion. 3 hypothesis-driven tests pass (per-belief reachability + dedup idempotence + scanner reachability); full suite 1747 passed.
Third entry point wired: each new belief produced by _resolve_or_create_belief logs one ingest_log row with source_kind=git, raw_text=phrase, derived_belief_ids=[bid], session_id (commit SHA when present). Re-assertions of an existing belief still record a corroboration row but do NOT emit a duplicate log row — the original log row already points at the canonical belief. 1 hypothesis-driven test pass; full suite 1748 passed.
…ngest_log (#205) Last three of six entry points wired: - tool_lock (MCP) → source_kind=mcp_remember, raw_text=statement - _cmd_lock (CLI) → source_kind=cli_remember, raw_text=statement - accept_classifications → source_kind=filesystem (host-classified candidate from a project file) All v2.0 ingest entry points now write to the parallel log. Re-locks do not duplicate the log row (canonical belief unchanged). 3 hypothesis-driven tests pass; full suite 1751 passed.
#205) v2.0 first-slice harness per memo D5(C): - src/aelfrice/replay.py - check_log_reachability(store) → ReachabilityReport. For every canonical belief, asserts ≥1 ingest_log row references its id. Spec acceptance #1. - replay_full_equality(store) → FullEqualityReport. Stub returning implemented=False. Full-equality replay is the v2.x flip-readiness probe; surface is wired so consumers can detect the not-yet-built branch without raising. 4 hypothesis-driven tests: - reachability passes after a v2.0 ingest_turn run - reachability flags a manually-inserted orphan - full_equality stub returns implemented=False - ingest_turn latency stays within the D6 budget (≤2.0x ceiling, alarms on >1.15x baseline) Full suite 1755 passed.
v2.0 first slice of write-log-as-truth (docs/design/write-log-as-truth.md): ingest_log is populated alongside every belief insert at six entry points. View authority remains on `beliefs`/`edges`; this is parallel write only, not the view-flip. 6 atomic commits + this gate: - 7bdb607 schema + ULID + record_ingest API + stale-table bootstrap - 8425ca8 log_versions VV mirroring #204 - 909e4bf scanner + ingest_turn entry-point parallel writes - 783bb6f triple_extractor (commit-ingest) parallel write - baf5049 MCP remember + CLI remember + onboard accept parallel writes - 3224b24 reachability harness + latency-budget alarm + full-equality stub Verification: - 30 hypothesis-driven tests in tests/test_ingest_log.py - Full suite 1755 passed, 8 pre-existing skips - Spec acceptance #1 (every belief has ≥1 log row): proved by test_*_writes_log_row_per_new_belief at every entry point + the reachability harness - Spec acceptance #2 (O(log n) source lookup): idx_ingest_log_source - Spec acceptance #3 (validation harness exists): reachability shipped, full-equality stubbed for v2.x - Spec acceptance #4 (latency delta ≤15%): alarmed at >1.15x, hard ceiling at 2.0x, deterministic baseline-vs-with-log measurement - VV primitive shared with #204: log_versions(log_id, scope_id, counter) Decisions locked per approach memo (~/.claude/handoffs/ kulili-2026-04-28-issue-205-approach-memo.md): - D1 hand-rolled monotonic ULID (stdlib only) - D2 raw_text inline (dedup deferred to v2.1) - D3 legacy_unknown source_kind reserved for migration (not wired) - D4 single-transaction parallel writes - D5(C) reachability default + full-equality opt-in - D6 ≤15% latency alarm, ≤2.0x hard ceiling - session_id added to ingest_log schema (open Q answered yes) Out of scope (v2.x or later): - View-flip on beliefs/edges - Derivation worker - Re-ingest dedup at storage layer (#254) - legacy_unknown migration of pre-v2.0 stores - Federation log shipping (v3) Blockers (require user decision before next phase): - [user] None for this phase. All design decisions locked at memo time; no in-flight surprises. Open questions: - v2.x flip-readiness: when do we wire replay_full_equality? Probably after the classifier-version field starts getting populated, which is itself a v2.x concern. Rollback: - git revert 3224b24..7bdb607 cleanly removes all six commits. Schema additions are CREATE TABLE IF NOT EXISTS; reverting drops the new tables on next open via the bootstrap. Any ingest_log rows collected before revert are lost — acceptable, no consumer depends on them.
Per memo D6 'alarm-not-block': the 2.0x hard ceiling on the ingest_turn latency ratio failed on noisy GH Actions Py3.12 runners (7x ratio observed) without representing a real regression. Replace with sanity floor (positive timings) and keep the >1.15x stdout alarm so drift stays visible to reviewers. The reachability check is the contract gate.
This was referenced Apr 28, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v2.0 first slice of #205 — parallel-write phase only, no view-flip.
Adds
ingest_logas an append-only sibling table populated alongsideevery belief insert at the six ingest entry points. View authority
remains on
beliefs/edges; this commit lands the data source thev2.x flip will consume.
Spec: docs/design/write-log-as-truth.md.
Approach memo:
~/.claude/handoffs/kulili-2026-04-28-issue-205-approach-memo.md.What lands
ingest_log(id ULID PK, ts, source_kind, source_path, raw_text, raw_meta JSON, derived_belief_ids JSON, derived_edge_ids JSON, classifier_version, rule_set_hash, session_id)+ indexes on(source_kind, source_path)andsession_idlog_versions(log_id, scope_id, counter)mirroring #204's VV primitiveMemoryStore.record_ingest(...),update_ingest_derived_ids(...),get_ingest_log_entry(...),iter_ingest_log_for_belief(...),count_ingest_log(),get_log_version_vector(...)aelfrice.ulid.ulid()— stdlib-only, monotonic, 26-char Crockford base32aelfrice.replay.check_log_reachability(...)(default) +replay_full_equality(...)stub for v2.xingest_logtable on open; refuses to drop if non-emptyEntry points wired (6 of 6)
scanner.py:236filesystemingest.py:129filesystemtriple_extractor.py:268gitmcp_server.py:217remember(lock)mcp_remembercli.py:797aelf lockcli_rememberclassification.py:503filesystemonboard_sessionsis unchanged. Re-ingest of an existing belief continues to record a corroboration row but does NOT add a duplicate log row — the original log row already references the canonical belief.Spec acceptance
(source_path, raw_text)lookup is O(log n)idx_ingest_log_sourceTest plan
tests/test_ingest_log.py— 30 hypothesis-driven tests (each test docstring states a falsifiable hypothesis):record_ingestfield round-trip + enum validation + nullable defaultsimplemented=FalseDecisions locked
Per approach memo:
raw_textinline; dedup deferred to v2.1 if storage biteslegacy_unknownsource_kind reserved (migration of pre-v2.0 stores not wired in this slice)Out of scope (v2.x or later)
beliefs/edges(v2.x)legacy_unknownmigration of existing pre-v2.0 storesCommits
7bdb607schema + ULID + record_ingest API + stale-table bootstrap8425ca8log_versions VV mirroring [v1.x forward-compat] SUPERSEDES version-vector schema for v3 federation #204909e4bfscanner + ingest_turn entry-point parallel writes783bb6ftriple_extractor (commit-ingest) parallel writebaf5049MCP remember + CLI remember + onboard accept parallel writes3224b24reachability harness + latency-budget alarm + full-equality stub8e6c9b2gate: phase complete