feat(bm25): score content and anchor as two normalised BM25F fields (#1180)

[robotrocketscience]

Closes #1180.

Implements the per-field BM25F scorer behind [retrieval] bm25f_per_field
(default off), plus bm25_b_anchor as the anchor stream's b.

What was wrong

The lane concatenates each belief's incoming anchor text into its own
document and normalises by the combined length. Because the replicas
land in the same dl, a belief's own content terms are length-penalised
in proportion to how much text its citers wrote about it.

Measured on a corpus where two beliefs are identical in everything BM25
can see except that one is cited, and the anchor text never mentions the
query term:

corpus-wide anchor density	legacy cited/uncited	per-field
1%	0.27x	1.0000x
10%	0.35x	1.0000x
100%	0.56x	1.0000x

A cited belief is demoted to roughly a quarter of an identical uncited
one, hardest at the sparse densities production actually runs at, for
something no part of the belief itself did.

What changed

tf~ = Σ_f w_f·tf_f/B_f with per-stream B_f = (1-b_f) + b_f·dl_f/avgdl_f,
saturated as (k1+1)·tf~/(k1+tf~). anchor_weight becomes a field weight
rather than a replication count; df counts a term once across the union
of the two streams; serialisation goes to v4.

Default off because this replaces the functional form rather than
re-parameterising it — the saturation denominator becomes the constant
k1 instead of tf + k1·B, so no choice of constants makes on and off
agree once an anchor stream exists. There is no parity test that could
gate the flip, only a bench. The legacy path is untouched and verified
byte-identical to main across a query/weight grid.

Two corrections to the issue

The stated formula breaks the issue's own acceptance criterion. It
gives score_t = idf·tf~/(k1+tf~), dropping the (k1+1) numerator as
Robertson's rank-equivalent presentation does. That leaves every score a
factor of 2.5 below the current lane at the shipped k1 = 1.5, so
w_anchor = 0 would not "recover standard BM25 byte-exact". Keeping
the numerator makes it exact — |delta| = 0 on every probe, and the
test asserts full score equality, not just ordering.

"Unbounded" is false. The issue rejects the simpler content-only-dl
fix on the grounds that it "trades a bounded penalty for an unbounded
boost". BM25 saturation caps every variant at idf·(k1+1); that fix's
boost converges to exactly 1.75x, measured out to a term frequency of
3e9. It is still the wrong instrument — the boost is unearned, paying
no length normalisation at all — but the conclusion needed a different
argument, and the acceptance criteria as written encode the wrong reason.

What the gating bench must control for

The anchor stream is sparse to absent in practice. The one real store
available for measurement holds 16,454 beliefs and no edges at all;
60 realistic turns through the production ingest_turn path produced
1 anchored belief in 15, with a one-token anchor.

Per-field's boost is strongly coupled to corpus-wide anchor density
(1.08x at 1% anchored, 1.91x at 100%), because avgdl_anchor averages
over every document including the many with no anchor text. So a
benchmark over a corpus with little anchor text will report a near-zero
delta that says nothing about the scoring change — the #1160 failure
mode. Any bench of this flag should publish, alongside the delta:

• share of indexed beliefs with ≥1 incoming anchor,
• anchor-stream lengths among those,
• share of eval queries whose gold belief is anchored.

A neutral result on a corpus with no anchors is a no-measurement, not
a refutation. The demotion fix, unlike the boost, is density-independent
— which is the argument for the change surviving a weak bench.

Acceptance criteria

• Separate tf_content / tf_anchor CSR matrices, dl_*, avgdl_*
• Per-field tf~; w_anchor takes over anchor_weight's role
• w_anchor = 0 recovers standard BM25 over content alone, exactly
• b_anchor tunable; default 0.75, justified in bm25.py and CONFIG
• Serialisation v4 (two tf matrices, two length vectors)
• Behind a flag, default off, legacy parity preserved when off
• Bench before any flip — not run here, see above for what it needs

b_anchor defaults to the content stream's b deliberately: the field
split already changes the functional form, so giving the anchor stream a
different b at the same time would confound "fields were separated"
with "the anchor stream is normalised differently". 0.75 also reproduces
the issue's own worked figures (0.5714 / 0.7835) exactly.

Tests

tests/test_bm25_per_field.py, 17 cases.
test_legacy_demotes_cited_belief_below_uncited is the distinguishing
assert for the file — it pins the defect on the legacy path, so the
per-field assertions cannot pass vacuously if the branch is later
short-circuited or the flag stops reaching build().

Full suite: 6448 passed, 69 skipped, 71 xfailed.

Summary by Sourcery

Introduce a bench-gated per-field BM25F mode that scores content and anchor text as two separately normalised fields with configurable anchor length-normalisation, updating index storage, retrieval configuration, and tests while preserving legacy behaviour by default.

New Features:

• Add optional per-field BM25F scoring that treats content and anchor text as separately normalised fields, controlled by retrieval config and environment flags.
• Introduce a configurable anchor-stream length-normalisation parameter for BM25F scoring.

Bug Fixes:

• Prevent cited beliefs from being unfairly demoted relative to uncited ones when anchor text does not mention the query term by correcting how anchor text contributes to BM25 scores.

Enhancements:

• Extend BM25 index construction, scoring, and caching to support dual content/anchor streams and maintain exact legacy behaviour when the new mode is disabled.
• Update BM25 index serialisation to a new version that persists per-field metadata and anchor-stream matrices while remaining backward-compatible with legacy blobs.

Documentation:

• Document the new per-field BM25F and anchor-normalisation options in the user configuration reference and changelog, including rationale and bench expectations.

Tests:

• Add a dedicated test suite for per-field BM25F covering scoring behaviour, boundedness, IDF computation, serialisation round-trips, and regression coverage for the legacy demotion defect.

[sourcery-ai]

Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Warning

Review limit reached

@robotrocketscience, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 52 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 19a30ceb-71d4-42f5-8916-a542099b5a6a

📥 Commits

Reviewing files that changed from the base of the PR and between b90cab7 and 1f86537.

📒 Files selected for processing (5)

• CHANGELOG/v4.md
• docs/user/CONFIG.md
• src/aelfrice/bm25.py
• src/aelfrice/retrieval.py
• tests/test_bm25_per_field.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sourcery-ai]

Reviewer's Guide

Implements a per-field BM25F scorer that separates content and anchor streams, adds configuration/flags and serialization support for the new mode, and introduces tests and docs to verify behavior and explain gating/bench expectations while preserving legacy behavior by default.

Sequence diagram for per-field BM25F scoring path resolution

sequenceDiagram
    participant Retrieval_l1_hits as _l1_hits
    participant Config as resolve_bm25f_per_field / resolve_bm25_b_anchor
    participant CacheFactory as _store_scoped_bm25f_cache
    participant Cache as BM25IndexCache
    participant Index as BM25Index

    Retrieval_l1_hits->>Config: resolve_bm25f_per_field()
    Config-->>Retrieval_l1_hits: per_field
    Retrieval_l1_hits->>Config: resolve_bm25_b_anchor()
    Config-->>Retrieval_l1_hits: b_anchor

    Retrieval_l1_hits->>CacheFactory: _store_scoped_bm25f_cache(anchor_weight,k3,per_field,b_anchor)
    CacheFactory->>Cache: BM25IndexCache(store,anchor_weight,k3,per_field,b_anchor)
    CacheFactory->>Cache: get()
    Cache->>Index: build(store,anchor_weight,k1,b,k3,per_field,b_anchor)
    Retrieval_l1_hits->>Index: score(query,top_k)

Loading

File-Level Changes

Change	Details	Files
Add per-field BM25F indexing and scoring with separate content and anchor fields, including new parameters and internal helpers.	Introduce DEFAULT_B_ANCHOR and extend BM25Index with per_field, anchor tf/dl/avgdl, and b_anchor attributes. Refactor stream construction into a reusable _build_stream helper that returns CSR tf, dl, and per-document term sets. Extend BM25Index.build to optionally construct separate content and anchor streams, compute df over their union, and validate b_anchor. Implement per-field BM25F saturation via _saturated_per_field, including per-field length-normalisation and anchor_weight as a field weight. Refactor legacy single-field scoring to share row index and length-normalisation helpers and branch on per_field. Bump serialization format to v4, writing/reading per_field flag, anchor-length stats, and anchor CSR, while keeping legacy blobs small and compatible.	src/aelfrice/bm25.py
Add retrieval-layer configuration, env overrides, and cache wiring for the per-field BM25F mode and anchor b parameter.	Introduce BM25F_PER_FIELD_FLAG and BM25_B_ANCHOR_FLAG plus corresponding env var names for per-field mode and anchor b. Add env parsing helpers and resolvers resolve_bm25f_per_field and resolve_bm25_b_anchor with precedence (env, explicit, TOML, default) and clamping for negatives. Extend BM25IndexCache and _store_scoped_bm25f_cache to track per_field and b_anchor, invalidating cached indices when these change and respecting per_field-only relevance of b_anchor. Wire resolve_bm25f_per_field and resolve_bm25_b_anchor into _l1_hits so the live retrieval path can toggle the new scoring mode.	src/aelfrice/retrieval.py
Document the new per-field BM25F configuration knobs and changelog entry for the #1180 feature.	Document bm25f_per_field and bm25_b_anchor under the [retrieval] CONFIG section, including defaults, semantics, and env overrides. Add a detailed changelog entry explaining the defect in legacy concatenation, the new BM25F formulation, tuning implications, and bench requirements.	docs/user/CONFIG.md CHANGELOG/v4.md
Add a focused test suite to validate per-field BM25F behavior, legacy regression, serialization, and boundedness properties.	Construct synthetic corpora with controlled content/anchor distributions to expose the legacy demotion bug and per-field fixes. Assert that per-field mode avoids demoting cited beliefs on irrelevant anchors and rewards anchor evidence monotonically while respecting BM25 saturation. Verify that anchor_weight=0 with per_field reproduces legacy content-only BM25 exactly and avoids touching anchor text, protecting df/idf. Test b_anchor behavior (including b_anchor=0 ablation and negative rejection), df over the union of streams, and serialize/deserialize determinism and version handling.	tests/test_bm25_per_field.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#1180	Implement per-field BM25F with separate content and anchor streams: store tf_content/tf_anchor with their own dl/avgdl, compute tf~ as sum over fields with per-field length normalisation, treat anchor_weight as a field weight (not replication), and ensure that w_anchor = 0 recovers standard BM25 over content alone byte-exactly.	✅
#1180	Introduce a tunable b_anchor with a justified default, extend serialization (v4) to carry per-field data (per_field flag, b_anchor, avgdl_anchor, dl_anchor, tf_anchor), and expose the new per-field BM25F scorer behind a retrieval flag that defaults off while preserving legacy behaviour when the flag is off; document these configuration knobs.	✅

Possibly linked issues

• fix(bm25): BM25F normalises content+anchor as one field — supersedes #1166 AC3 #1180: PR delivers the per-field BM25F scorer, separate tf/dl for content/anchor, new b_anchor, flag, and v4 serialization exactly as requested.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[github-actions]

PR-size soft cap

This PR is over the advisory size threshold:

• 973 changed lines (limit: 200)
• 5 changed files (limit: 3)

Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated attn:merge-conflict cycles (see #602). When practical, split into smaller PRs that each touch a focused surface.

This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the size:override label and this comment will be removed on the next push.

[robotrocketscience]

[claim:review:garsecg:2026-07-30T21:05:21Z]

[robotrocketscience]

Review: approved, with one claim that needs weakening

CI green. Commits atomic and signed. Discretion grep on added lines clean.

The claim that ships is verified

The default path is what production runs, and "legacy is untouched" holds.
Built the same 300-belief corpus (60% anchored) on github/main and on this
branch, and compared full score() output at anchor_weight ∈ {0, 1, 3, 5},
plus idf[:50] and avgdl:

legacy path identical to main: True

Byte-identical, not approximately. The df refactor into _build_stream is
equivalent to the old inline df_counts[j] += 1 — both count one increment per
(doc, term-with-nonzero-count) — and that shows up here rather than being
argued.

The per-field math checks out against Robertson/Zaragoza/Taylor. Applying
B_f to each stream's raw tf before summing, with k1 alone in the
saturation denominator, is the right form; a mixed B in the denominator
would let one stream's length set the other's penalty. Keeping the (k1+1)
numerator is correct and the reasoning in _saturated_per_field's docstring
is right: with one stream, (k1+1)(tf/B) / (k1 + tf/B) is algebraically
tf(k1+1) / (tf + k1·B), which is the legacy expression.

w_anchor = 0 is not exact, and the test that says so is a latent flake

Algebraically identical, yes. Numerically it is not, because the two
paths evaluate that identity in a different order in float32 — legacy divides
by (tf + k1·B), per-field divides by B first and then saturates.

test_per_field_at_w0_is_identical_to_legacy_at_w0 asserts legacy.score(q) == per_field.score(q), exact float equality. It passes on the ~8-belief
fixture. On a 400-belief random corpus (seeded, 50% anchored, 200 queries of
1–8 terms):

queries=200  exact=0  mismatch=200  max_abs_delta=1.907e-06
max_relative_delta=2.192e-07
queries_with_different_order=1/200
different_topk_membership=0

All 200 mismatch. The relative delta is 2.19e-07, i.e. float32 epsilon —
so this is rounding, not a defect, and I am not asking for the arithmetic to
change. Top-k membership is unaffected. But two things follow:

1. The test is pinned to its fixture, not to the property. It passes
   because that corpus happens to round identically. Any change to the fixture,
   or a platform with different float behaviour, turns it red for a reason
   unrelated to correctness — and the docstring tells the next reader the
   (k1+1) numerator is what makes it pass, so they will look in the wrong
   place. pytest.approx(rel=1e-6) on the score, with exact assertion on the
   id ordering, pins the real invariant.
2. The AC wording overstates it. "recovers standard BM25 byte-exact" and
   "|delta| = 0 on every probe" are true only of the probes run. Worth saying
   "exact to float32 rounding (rel ~2e-7)" in the PR body and the changelog
   entry, since the next person to bench this will otherwise treat any observed
   delta at w=0 as a bug.

One-in-two-hundred queries also comes out in a different order at w=0:
two beliefs that tie exactly on one path differ by one ulp on the other. Not a
determinism-contract violation — each mode is deterministic in itself — but it
does mean per-field at w=0 is not a drop-in for legacy at tie boundaries,
which is worth a line given the AC claims equivalence.

Neither of these blocks the merge: the flag is default-off, the shipped path is
verified unchanged, and the finding is about how the equivalence is asserted
rather than whether the scorer is right.

Two things for whoever runs the gating bench

The PR's own warning about anchor density is the right one and I have nothing
to add to it. Two more:

• _saturated_per_field runs per score() call and does two .copy()
  plus a sparse addition over the union sparsity pattern, where legacy does one
  in-place transform. On a 16k-belief store that is a real per-query cost, and
  it is not on the flag's measured axis. Publish latency alongside the quality
  delta, or the flip trades an unknown amount of it for the boost.
• b_anchor = 0.75 reproducing the issue's worked figures is a good
  sanity check but is not evidence for the value. The docstring already says
  tuning it is separately benched — agreed, and the b_anchor > 0 boundedness
  argument is the load-bearing part, which test_b_anchor_zero_removes_the_ anchor_length_penalty pins.

Adding ready-to-merge. The two items above are wording and test-robustness,
not correctness; file or fold them as you prefer.

[robotrocketscience]

Correcting my own last line: holding ready-to-merge rather than adding it,
for one round trip.

I said the exactness finding was wording-and-test-robustness and did not block.
That was right about the code and wrong about where the claim lands. The
changelog entry says:

w_anchor = 0 reproduces the legacy lane's scores with zero delta

Unqualified, in a permanent user-facing record, and I have a direct
counterexample — 200/200 queries differ on a 400-belief corpus, max abs delta
1.9e-06. The scorer is fine; the sentence is not, and a changelog is the one
artifact here that does not get quietly corrected later. The next person to
bench this reads that line and treats a 1e-6 delta at w=0 as a regression.

What unblocks it, both small:

1. Changelog + PR body: "reproduces the legacy lane's scores exactly to float32
   rounding (rel ~2e-7)" — or state the probe it was measured on. The
   (k1+1)-numerator argument is unaffected and stays; without it the gap is
   2.5x, not 1e-7, which is the point worth keeping.
2. test_per_field_at_w0_is_identical_to_legacy_at_w0: pytest.approx(rel=1e-6)
   on scores, exact on the id ordering. As written it passes because of its
   fixture's size rather than the property, and its docstring points the next
   reader at the numerator when it eventually reddens.

Everything else in my review stands — legacy path verified byte-identical to
main, math checks out, approve on the substance. Re-add the label once the
wording lands and I will not need to look again.

[robotrocketscience]

[release:review:garsecg:2026-07-30T21:09:49Z]

[robotrocketscience]

[claim:review:Setr:2026-07-30T21:13:14Z]

[robotrocketscience]

Review: approved. Prior round's two asks are now landed on the branch.

The previous review approved the substance and held ready-to-merge on a
changelog claim plus a fixture-dependent assert. Both were still open and the
branch had gone quiet, so I verified them independently and pushed the fixes
rather than leaving the PR parked.

Independently reproduced, both findings

The legacy path really is untouched. Built the same 300-belief corpus
(60% anchored) on github/main and on this branch and dumped avgdl,
idf[:50], dl[:20], tf.nnz and full score() output over 40 random
queries at anchor_weight ∈ {0, 1, 3, 5}, at repr() float precision:

diff main.json pr.json  ->  IDENTICAL
serialize/deserialize round-trip: True on both trees

That is the whole safety argument for a default-off flag, so it is worth
having it re-derived from a different corpus and a different seed than the
first review used.

w_anchor = 0 is not exact, on a corpus I generated independently —
400 beliefs, 50% anchored, seed 17, 200 random queries:

vocab equal: True   idf equal exactly: True
queries=200  exact=0  mismatch=200
max_abs_delta=1.907e-06  max_rel_delta=2.229e-07
order_diff=1  topk_membership_diff=0

Same numbers as the first round to three digits, from a different corpus. It
is float32 rounding — legacy divides by (tf + k1·B), per-field divides by
B first and then saturates — not a defect, and I am not asking for the
arithmetic to change.

I did check the one way this could have been a real difference rather than
rounding: under per_field the vocabulary and df are built over the union
of both streams, so anchor-only terms would move idf for every term. They
do not, because from_store guards the anchor read with if anchor_weight > 0 — at w = 0 no anchor text is tokenised in either mode. idf equal exactly: True above confirms it, and
test_per_field_at_w0_does_not_read_anchor_text_at_all already pins it.

Pushed — 6073b076, 5ccd428c

1. test_per_field_at_w0_is_identical_to_legacy_at_w0 now asserts the
   same id set plus pytest.approx(rel=1e-6) per belief, with the measured
   figures in the docstring. Mutation-checked rather than assumed: dropping
   the (k1+1) numerator from _saturated_per_field still reddens it
   (1 failed, 16 passed), so the relaxation kept the property the test
   exists for — rel=1e-6 has six orders of magnitude of headroom over a
   2.5x gap.

2. Changelog now says "exactly up to float32 rounding" and carries the
   measured delta, so the next person to bench this reads 1e-6 at w = 0 as
   rounding rather than a regression.

One correction to the previous round's prescription. It suggested
approx on the score with an exact assertion on the id ordering. Ordering
is the one part that is not safe to pin: 1 of 200 queries orders an exact
tie differently by a single ulp, so an exact-ordering assert would be the
same latent flake one layer over. Top-k membership was stable at 0/200
differences, so the test asserts the id set and per-id scores and documents
the tie caveat instead.

Non-blocking, for whoever runs the gating bench

Both carried forward from the first round and still worth having in one
place: publish per-query latency alongside the quality delta —
_saturated_per_field does two .copy() plus a sparse union-add per
score() where legacy does one in-place transform — and treat a neutral
result on a corpus with no anchor text as a no-measurement, per the PR
body's own warning.

Full suite on the pushed head: 6448 passed, 69 skipped, 71 xfailed.
Discretion grep on added lines clean. Adding ready-to-merge.

[github-actions]

merge-train: merged 1f86537 → main via FF push.

[robotrocketscience]

[release:review:Setr:2026-07-30T21:28:53Z]