Skip to content

perf(bm25): update the index incrementally instead of rebuilding it (#1199) - #1223

Merged
github-actions[bot] merged 3 commits into
mainfrom
perf/issue-1199-incremental-bm25f
Jul 30, 2026
Merged

perf(bm25): update the index incrementally instead of rebuilding it (#1199)#1223
github-actions[bot] merged 3 commits into
mainfrom
perf/issue-1199-incremental-bm25f

Conversation

@robotrocketscience

@robotrocketscience robotrocketscience commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Closes #1199.

AC1 was measured and posted on the issue; the operator ruled option 2,
incremental — target build(), not serialisation
(2026-07-30). This is that.

What the measurement said

597 real user_prompt_submit fires, 60 sessions, 2026-06-29 → 2026-07-30, on
the 45,931-belief ambient store:

  • 86.2% of the fires that actually reach retrieval rebuilt the whole index
    (163/189); the median session rebuilt on every such prompt.
  • The all-fires figure is 27.3%, but 408 of 597 are shape-gated before
    retrieval and never ask for an index — that denominator credits the sidecar
    for work it never did.
  • Cold p50 1839 ms / p95 2499 ms vs 246 ms for a sidecar hit, and the two modes
    do not overlap (warm tops out at 411 ms, cold starts at ≥1000 ms, zero fires
    between).
  • 96% of the cold path is BM25Index.build() — 1113 ms of tokenising and
    stemming against 44 ms serialise, 2 ms disk read, 52 ms deserialise.
  • Cause: ~37 belief writes per retrieval-running prompt, and every mutation
    bumps store_generation in the same transaction.

Result

On that store, update_from against a stale sidecar:

change set full build incremental
nothing indexed changed 1562 ms 392 ms 4.0x
+1 belief 1500 ms 429 ms 3.5x
+20 beliefs 1544 ms 390 ms 4.0x
+200 beliefs 1460 ms 401 ms 3.6x

Flat in the size of the change set, because what is left is the fixed
change-detection scan, not the tokenisation it replaces. Profiled, that
remainder is ~178 ms of fetchall and ~120 ms of fingerprinting — 4x, not
the 20x the 96% figure might suggest
, and the honest reason is that
detecting which documents changed still costs a read of every document.
Going below that needs a trusted per-row change signal, which the schema does
not currently have; see the note at the end.

Two design decisions worth reviewing against intent

Change detection does not use beliefs.content_hash. That column is
written by callers (classification._content_hash, derivation._content_hash)
and is not enforced by the store. Keying index validity on it would mean one
writer's wrong hash silently serves stale retrieval results — a correctness
bug with no symptom. The sidecar carries a digest of the exact text that was
tokenised instead. It costs the SELECT id, content the build already does and
owes nothing to a convention the store does not guarantee.

Anchor fingerprints sort before hashing. Anchor order feeds per-term counts
and a total length, both commutative, so hashing in iteration order would
invalidate documents whose index is provably identical.

AC3 — identity, not approximation

The invariant from #1135 is that retrieval output is a deterministic function
of store content, so "almost the same index" would be a correctness regression
wearing a performance improvement's clothes. Every value is derived the way
build() derives it rather than patched forward:

  • dl is copied for reused rows, so no float re-rounding can creep in.
  • df is counted off the assembled sparsity pattern, not carried and
    adjusted, so a mis-tracked increment cannot accumulate across generations.
  • idf and avgdl are computed from those at the end, by the same
    expressions.
  • Deletions prune vocabulary terms they orphan — otherwise n_terms and
    every column index after them drift from a fresh build.

Verified on the production store, field for field:

full build 1765 ms   vs   update_from 519 ms
identical belief_ids : True      identical dl    : True
identical vocabulary : True      identical idf   : True
identical tf.indices : True      identical avgdl : True
identical tf.data    : True

test_incremental_matches_a_full_rebuild_under_random_churn is the
distinguishing assert: 60 random mutations — inserts, content edits,
soft-deletes, hard-deletes, anchor-edge churn, and posterior bumps that change
no indexed text — asserting exact equality after each, in both scoring modes.

The test suite was checked for vacuity rather than assumed. The walk
exercises the incremental path 60/60 steps in both modes (pinned by an
assertion, since a version that declined every step would otherwise compare a
rebuild against itself). Two mutations were applied to the shipped source and
reverted:

mutation caught by
carry the base vocabulary forward without pruning orphans churn test, both modes
recompute dl for reused rows instead of copying churn test + 2 others

The first initially escaped the walk, because a 16-word shared vocabulary can
never orphan a term — the corpus generator now emits a document-unique rare
token half the time, and the mutation is caught. That is recorded in the
generator's docstring so it does not get "simplified" away.

test_cache_serves_a_stale_sidecar_incrementally pins the wiring separately by
replacing BM25Index.build with a bomb, so the only way get() can return is
the incremental path; it has a negative control asserting a build still happens
when there is no sidecar to update from.

Where it declines

Returns None and the caller builds exactly as before: no fingerprints on the
base, a different anchor_weight or per_field (its rows describe different
documents), an empty side, or more than a 50% change ratio, where the
bookkeeping costs more than the tokenisation it saves.

Serialisation goes to v5 for the fingerprints — one rebuild per store, the same
posture as v3 and v4. A blob without them still round-trips and scores; it just
cannot seed an update.

Not in scope

The ~300 ms floor is the change-detection read. A per-row generation or
indexed_at column would let the query skip unchanged rows entirely and take
this toward the ~53 ms sidecar-hit cost — but that is a schema migration on the
hot table, and #1161 is a live reminder of what a bad one-shot migration does
to this store. Worth its own issue with its own gate rather than riding along
here.

Full suite: 6487 passed, 69 skipped, 71 xfailed.

Summary by Sourcery

Introduce incremental BM25F index updates that reuse unchanged rows from a stale sidecar instead of rebuilding the full index on most retrievals, while preserving bit-for-bit equality with a full rebuild.

New Features:

  • Add BM25Index.update_from to rebuild the BM25F index incrementally by re-tokenising only documents whose indexed text changed.
  • Persist per-document source and anchor fingerprints in the BM25F index and its serialization format to enable change detection.
  • Allow the BM25 index cache to load stale sidecars and refresh them incrementally when safe, falling back to full rebuilds when necessary.

Enhancements:

  • Define a configurable maximum change ratio threshold beyond which incremental BM25 index updates decline in favor of a full rebuild.
  • Optimize CSR row gathering and anchor fingerprinting to reduce overhead in the incremental update path.

Documentation:

  • Document the new incremental BM25F index update behavior and performance characteristics in the v4 changelog.

Tests:

  • Add a dedicated test suite validating that incremental BM25 index updates are identical to full rebuilds under random store churn and various mutation types.
  • Add tests covering fingerprint behavior, serialization round-trips, decline conditions, and cache integration for incremental updates.

@robotrocketscience robotrocketscience added the author-Setr PR coordination mutex label Jul 30, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@robotrocketscience, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 38 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: d695bf12-19e5-472f-958e-40c451379966

📥 Commits

Reviewing files that changed from the base of the PR and between 64f698e and bed0f6f.

📒 Files selected for processing (3)
  • CHANGELOG/v4.md
  • src/aelfrice/bm25.py
  • tests/test_bm25_incremental_1199.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

PR-size soft cap

This PR is over the advisory size threshold:

  • 827 changed lines (limit: 200)
  • 3 changed files (limit: 3)

Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated attn:merge-conflict cycles (see #602). When practical, split into smaller PRs that each touch a focused surface.

This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the size:override label and this comment will be removed on the next push.

@sourcery-ai

sourcery-ai Bot commented Jul 30, 2026

Copy link
Copy Markdown

Reviewer's Guide

Implements incremental BM25 index updates that reuse unchanged document rows based on source fingerprints instead of rebuilding the full index on every mutation, updates the sidecar cache to leverage stale indexes for incremental rebuilds, and extends serialization to carry the necessary fingerprints, with a comprehensive test suite validating bit‑for‑bit identity with full rebuilds and correct cache wiring.

Sequence diagram for BM25 sidecar incremental update vs full rebuild

sequenceDiagram
    actor RetrievalCaller
    participant BM25IndexCache as BM25IndexCache
    participant MemoryStore as MemoryStore
    participant BM25Index as BM25Index

    RetrievalCaller->>BM25IndexCache: get()
    alt cached index present
        BM25IndexCache-->>RetrievalCaller: return _index
    else no cached index
        BM25IndexCache->>MemoryStore: store_generation()
        BM25IndexCache->>BM25IndexCache: _load_sidecar(require_fresh=False)
        BM25IndexCache-->>BM25Index: stale_sidecar
        BM25Index->>BM25Index: update_from(base, store,...)
        alt update_from returns index
            BM25Index-->>BM25IndexCache: updated_index
        else update_from returns None
            BM25IndexCache->>BM25Index: build(store,...)
            BM25Index-->>BM25IndexCache: rebuilt_index
        end
        BM25IndexCache->>BM25IndexCache: _write_sidecar(index, generation)
        BM25IndexCache-->>RetrievalCaller: return index
    end
Loading

File-Level Changes

Change Details Files
Add per-document source and anchor fingerprints and incremental update capability to BM25Index so that only documents with changed indexed text are re-tokenised while preserving exact index identity with a full rebuild.
  • Introduce DEFAULT_MAX_CHANGE_RATIO threshold and bump serialization version to include fingerprints for incremental updates.
  • Add fingerprint helpers for document content and incoming anchors, including order-independent anchor hashing with length-prefixing and a precomputed empty-anchor digest.
  • Extend BM25Index dataclass with content_fp and anchor_fp fields and populate them during build using the new fingerprint helpers.
  • Implement BM25Index.update_from() to detect changed vs reusable rows via fingerprints, gather CSR slices for reused rows, retokenise only changed documents, rebuild the vocabulary with orphan term pruning, and recompute df/idf/avgdl to match a full build bit-for-bit.
src/aelfrice/bm25.py
Wire the BM25IndexCache to use incremental updates from a stale sidecar when possible, falling back to full builds otherwise, and relax sidecar freshness checks to support this behavior.
  • Update BM25IndexCache.get() to try loading a stale sidecar, call BM25Index.update_from() when available, and only perform a full BM25Index.build() when incremental update declines.
  • Modify _load_sidecar() to accept a require_fresh flag that bypasses the generation-stamp check when using a stale sidecar as an incremental base while still enforcing scope and parameter compatibility, and only update _generation when a fresh sidecar is used.
src/aelfrice/bm25.py
Extend BM25 index serialization format to version 5 to persist fingerprints while preserving backward compatibility for indexes without fingerprints.
  • Increase _SERIALIZE_VERSION from 4 to 5 and document that v5 adds per-document source fingerprints.
  • Update BM25Index.serialize() to write a presence flag followed by content_fp and anchor_fp arrays when available.
  • Update BM25Index.deserialize() to read the fingerprints conditionally based on the presence flag and attach them to the reconstructed BM25Index, leaving them None for older blobs.
src/aelfrice/bm25.py
Document the incremental BM25F update behavior and performance characteristics in the changelog, including the conditions under which incremental updates are used or declined.
  • Add a detailed changelog entry describing the move to incremental BM25F index updates, performance measurements, correctness requirements, and decline conditions such as missing fingerprints, parameter mismatches, empty sides, or high change ratios.
CHANGELOG/v4.md
Add a dedicated test suite to validate fingerprint behavior, incremental update correctness under various mutation scenarios, decline conditions, and BM25 index cache wiring.
  • Create tests for _source_fingerprint and _anchor_fingerprint properties (content changes, order independence, length-prefixing) and for fingerprint propagation through BM25Index.build() and serialization round-trips.
  • Add tests verifying that editing a belief affects only its own fingerprint and that indexes without fingerprints still round-trip and score correctly but cannot be used for incremental updates.
  • Implement a randomized churn test that applies diverse store mutations (insert/edit/delete, soft-delete, anchor-edge changes, posterior-only bumps) and asserts that BM25Index.update_from() produces an index identical to BM25Index.build() whenever it takes the incremental path, with a guard to ensure the path is exercised.
  • Add targeted tests for orphaned vocabulary term pruning on deletion, decline-on-parameter mismatch and change-ratio thresholds, and cache behavior asserting that a stale sidecar triggers incremental update (with BM25Index.build patched to fail) while absence of a sidecar still forces a full build.
tests/test_bm25_incremental_1199.py

Assessment against linked issues

Issue Objective Addressed Explanation
#1199 Benchmark BM25F sidecar behavior and cold-path latency in normal usage, and document the results.
#1199 Change BM25F index maintenance so that it performs incremental updates instead of full rebuilds on each write, while limiting when incremental applies (e.g., via a change-ratio threshold).
#1199 Ensure that any incremental BM25F index maintenance produces retrieval indices that are byte-identical to a full rebuild on a fixed corpus, or clearly document acceptable divergences.

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@robotrocketscience

Copy link
Copy Markdown
Owner Author

[claim:review:garsecg:2026-07-30T22:13:11Z]

@robotrocketscience

Copy link
Copy Markdown
Owner Author

Reviewed by re-deriving the correctness claim independently rather than reading the equality table. No blocking findings. Approving via ready-to-merge.

The claim that matters here is not the 4x — it is that update_from returns an index identical to build(), because #1135's contract makes "almost the same index" a correctness regression wearing a performance improvement's clothes. That is the claim I went after.

Independent identity probe — 32/32

I wrote my own harness rather than trusting the shipped churn walk, comparing update_from against build() field for field (belief_ids, vocabulary, CSR indptr/indices/data, dl, idf, avgdl, plus tf_anchor/dl_anchor/avgdl_anchor under per-field) across 16 mutation shapes × both scoring modes:

shape
delete the only anchor edge on a doc · delete every anchor edge in the store · add the first-ever anchor edge OK
edit anchor_text in place · anchor_text"" · anchor_textNone OK
soft-delete · hard-delete · delete the sole holder of a rare token · insert a doc bearing a brand-new rare token OK
content edit at identical length · content → "" · content → punctuation-only · content → whitespace-only OK
posterior bump with no text change · no-op generation OK

All 32 took the incremental path (my harness reports a decline separately, so an "OK" produced by falling back to a rebuild would have been visible). The empty-string and punctuation-only cases were the ones I expected to break something — a document that tokenises to nothing still has to hold a row and a dl of 0.

The shipped tests have teeth — checked by mutation, not assumed

Two mutations applied to the shipped source and reverted:

mutation result
carry the base vocabulary forward without pruning orphans (terms = set(base_terms)) 3 failed — churn walk in both modes, plus the dedicated orphan test
never decline on the change ratio 1 failedtest_update_declines_past_the_change_ratio

And the churn walk chains generations (idx = got) rather than re-seeding from a fresh build each step, so drift across 60 successive incremental updates is what it actually measures. The exercised >= 50 guard against a version that declines everything is the right instinct.

Points I checked specifically because they could fail silently

  • Stale-sidecar parameter safety. _load_sidecar(require_fresh=False) applies the anchor_weight / k1 / b / k3 / per_field / b_anchor rejections on the stale path too, not just the fresh one — so a sidecar written under a different scoring config cannot seed an update. Worth stating because the rejections sit after the require_fresh branch and it would be easy to read them as fresh-only.
  • Generation stamping. The stamp is still read before the incremental update and written after, so a mutation landing mid-update leaves the sidecar already-stale and the next reader rebuilds. Same conservative posture as the existing build path; the incremental branch does not weaken it.
  • Per-field anchor-only terms. used_old concatenates the content and anchor column sets under per_field, so a term that appears only in an anchor stream survives the vocabulary rebuild. Under single-field it correctly uses the content columns alone, since there is one stream.
  • Not keying on content_hash. Right call, and the reasoning in the docstring is the reasoning: that column is caller-written and store-unenforced, so keying index validity on it converts one writer's bug into silently stale retrieval with no symptom. The blake2b-64 digest of the text actually tokenised owes nothing to that convention, and the length-prefixed sorted anchor digest closes the ["ab","c"] / ["a","bc"] collision.
  • Full suite on the branch: 6485 passed, 71 skipped, 71 xfailed. Six test_uninstall_* failures on my first pass were the known missing---extra archive local environment issue, not this branch — they pass with the extra synced, which is what CI does.
  • Discretion grep on added lines vs main: clean. CI: green, mergeable.

One thing worth keeping visible, not a change request

The honest framing of the 4x — that the remaining ~300 ms is the change-detection read, and going below it needs a per-row change signal the schema does not have — is the most useful paragraph in the PR body, and it belongs in an issue rather than only here. Deferring that schema change on the strength of #1161 is the right read of the risk. If it is not already filed, it should be, so the ceiling is recorded where the next person profiling this will look.

Adding ready-to-merge.

@robotrocketscience robotrocketscience added ready-to-merge Trigger merge-train: FF main to this PR's head and removed attn:review Needs review (PR open, awaiting reviewer) labels Jul 30, 2026
@robotrocketscience

Copy link
Copy Markdown
Owner Author

[release:review:garsecg:2026-07-30T22:23:06Z]

@github-actions

Copy link
Copy Markdown

merge-train: blocked

branch is not fast-forward on main (branch base 1f86537d720a3b21c1b2e463bbe76c00d2ec2b0d, current main d514499f59f2ff1091c1a76ecd74fbecc9dd1429). Rebase locally (git rebase github/main), force-push, and re-add the label.

The ready-to-merge label has been removed. Address the issue above and re-add the label when you're ready for another attempt.

@github-actions github-actions Bot removed the ready-to-merge Trigger merge-train: FF main to this PR's head label Jul 30, 2026
Prerequisite for the incremental update (#1199): without a per-document
record of what was indexed, a loaded sidecar cannot say which rows are
still valid, and the only safe move is to re-tokenise everything.

Fingerprints the text that was actually tokenised rather than reusing
`beliefs.content_hash`. That column is written by callers and is not
enforced by the store, so keying index validity on it would let one
writer's wrong hash silently serve stale retrieval results.

Anchor fingerprints sort before hashing: anchor order feeds per-term
counts and a total length, both commutative, so hashing in iteration
order would invalidate documents whose index is provably identical.

Serialisation goes to v5, flagged rather than mandatory so a hand-built
index still round-trips and scores. No scoring path reads these.
#1199 AC1 measured 86.2% of retrieval-running prompts rebuilding the
index from scratch, 96% of that cost in build() re-tokenising ~45k
documents to absorb a change ratio under 0.1%.

update_from() reuses the rows whose fingerprint is unchanged and
tokenises only the rest. On the real 45,931-belief store: 1.5s -> 0.4s,
about 4x, flat in the size of the change set because the remaining cost
is the fixed change-detection scan (178ms SQL + fingerprints), not the
tokenisation it replaces.

Identity, not approximation. Every value is derived the way build()
derives it rather than patched forward: dl is copied for reused rows so
no float re-rounding creeps in, df is counted off the assembled
sparsity pattern rather than carried and adjusted, and idf/avgdl are
computed from those at the end. Verified field-for-field on the
production store and over randomised churn in both scoring modes.

Declines to None -- caller builds as before -- when the base has no
fingerprints, when anchor_weight or per_field differ so its rows
describe different documents, when either side is empty, or past a 50%
change ratio where bookkeeping costs more than it saves.
@robotrocketscience robotrocketscience added the attn:merge-conflict PR branch needs rebase label Jul 30, 2026
@robotrocketscience

Copy link
Copy Markdown
Owner Author

Merge-train bounced this on a stale base, not on the review: main moved from 1f86537d to d514499f between the label and the run, and the branch is now CONFLICTING. My approval above stands — nothing in the diff changed. Rebase on github/main, force-push, and re-add ready-to-merge; no re-review needed unless the rebase touches bm25.py.

Carries the AC1 measurement that motivated it, including why the
headline is 86.2% rather than the 27.3% an all-fires denominator would
give, and states the identity guarantee and the conditions under which
the path declines.
@robotrocketscience

Copy link
Copy Markdown
Owner Author

Rebased onto current main#1220, #1224 and #1221 all landed underneath while this sat in review. The only conflict was the [Unreleased] CHANGELOG hunk, resolved insert-only: main's heat-kernel entry kept in place, mine appended after it, nothing reordered. Verified both entries survive and the diff deletes no line (git diff main...HEAD -- CHANGELOG/v4.md | grep '^-' is empty apart from the --- header).

The src/ + tests/ diff is untouched by the rebase, and the full suite is green on the new base: 6505 passed, 69 skipped, 71 xfailed. Discretion grep on added lines clean.

Still needs a reviewer — I authored it, so I can't self-review or merge.

@robotrocketscience
robotrocketscience force-pushed the perf/issue-1199-incremental-bm25f branch from 6bdb7fd to bed0f6f Compare July 30, 2026 22:49
@robotrocketscience robotrocketscience added attn:review Needs review (PR open, awaiting reviewer) and removed attn:merge-conflict PR branch needs rebase labels Jul 30, 2026
@robotrocketscience

Copy link
Copy Markdown
Owner Author

[claim:review:Toug:2026-07-30T22:57:22Z]

@robotrocketscience

Copy link
Copy Markdown
Owner Author

Approving. I verified the bit-exactness claim independently against the live store rather than reading the table, added two mutations the PR did not try, and checked the one interaction I thought most likely to be missed. Everything held.

Bit-exactness, re-measured on the live store (44,584 docs, 21,147 terms)

Both scoring modes, four scenarios each, comparing update_from against a fresh build field by field — belief_ids, vocabulary, tf.indices / indptr / data, dl, idf, avgdl, plus tf_anchor / dl_anchor / avgdl_anchor in per-field:

scenario single-field per-field
nothing changed ALL EQUAL ALL EQUAL
+1 belief with globally-unique rare tokens ALL EQUAL ALL EQUAL
+anchor-only terms (never present in any content) ALL EQUAL ALL EQUAL
delete that orphans a vocabulary term ALL EQUAL ALL EQUAL

I constructed the third case deliberately. df is counted with np.bincount over (tf + tf_anchor), and that is only equal to build's per-document set-union count if neither stream contributes an explicitly-stored zero. It holds — both streams carry non-negative counts, so CSR addition can never produce one — and the anchor-only-term case is the sharp end of it, since those terms enter the shared vocabulary through a stream the content side never sees.

Worth a note for whoever touches it next: that correctness rests on an invariant nothing asserts. If a future change ever stores an explicit zero in either stream, df inflates silently and every idf moves. Non-blocking, but a one-line assert or a comment at the bincount would pin it.

Mutations, mine rather than the PR's

mutation result
reuse a row on content_fp alone, ignoring anchor_fp 2 failed — churn test, both modes
hash anchors in iteration order instead of sorted(anchors) 1 failedtest_anchor_fingerprint_ignores_anchor_order
(control) 54 passed, 1 skipped

The first is the one that matters: an anchor-text edit that leaves content untouched is exactly the change a content-keyed fingerprint would miss, and it would surface as a stale index with no symptom. The churn walk catches it in both modes.

The interaction I went looking for

b_anchor landed in #1219 as a scoring-time scalar carried on the index. An incremental path that took it from base instead of the call would produce a correct-looking index scored under the wrong parameter — silent, and invisible to any structural equality check.

It is taken from the call, and the comment says why. Verified behaviourally rather than by reading:

base built at b_anchor=0.75, updated at b_anchor=0.25
  b_anchor carried from call : 0.25
  top-10 scores identical to a fresh build at 0.25 : True
  k1 likewise : True

Also confirmed: all four decline paths return None (no fingerprints, per_field mismatch, anchor_weight mismatch, past the 0.5 change ratio); the v5 round-trip preserves fingerprints and scores; a fingerprint-less blob still loads and scores, it just cannot seed an update; and generation is read before the update, so a mutation landing mid-update makes the stamp stale and the next reader rebuilds — the conservative direction.

Perf, independently

Best of 3 on my copy of the live store, single-field: 882 ms full build → 197 ms update_from, 4.5×. Consistent with the table, and with the PR's honest framing that the floor is the change-detection read rather than the tokenisation it removes.

Deferring the per-row indexed_at column to its own issue is the right call — that is a schema migration on the hot table, and #1161 is the reason not to ride it along here.

Also checked

  • Discretion grep on added lines vs main: clean.
  • The change-ratio decline, the orphaned-vocabulary drop, and the no-op generation bump each have their own named test; the churn test carries the "incremental path actually ran 60/60" assertion, so it cannot degrade into comparing a rebuild against itself.

Nothing blocking. The df invariant note above is the only thing I would consider following up, and it is a comment rather than a change.

@robotrocketscience robotrocketscience added the ready-to-merge Trigger merge-train: FF main to this PR's head label Jul 30, 2026
@robotrocketscience

Copy link
Copy Markdown
Owner Author

[release:review:Toug:2026-07-30T23:03:45Z]

@robotrocketscience robotrocketscience removed the attn:review Needs review (PR open, awaiting reviewer) label Jul 30, 2026
@robotrocketscience

Copy link
Copy Markdown
Owner Author

Rebase verified: FF on main, all three commits signed (G), and git diff of src/aelfrice/bm25.py + tests/test_bm25_incremental_1199.py against the revision I reviewed is empty — the rebase moved the base, not the content. Approval stands; re-adding ready-to-merge.

@github-actions github-actions Bot removed the ready-to-merge Trigger merge-train: FF main to this PR's head label Jul 30, 2026
@github-actions
github-actions Bot merged commit bed0f6f into main Jul 30, 2026
32 of 34 checks passed
@github-actions

Copy link
Copy Markdown

merge-train: merged bed0f6fmain via FF push.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author-Setr PR coordination mutex

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(bm25): the BM25F sidecar invalidates on every write, so the first prompt after an ingest rebuilds the whole index

1 participant