feat(bench-gate): #433 vocab_bridge NDCG@k uplift driver — evidence: bridge regresses on labelled corpus

[robotrocketscience]

Adds the strict NDCG@k uplift driver for #433 Phase 2 (the operator-side ship gate). Three atomic commits:

1. feat(test): run_vocab_bridge_uplift driver in tests/retrieve_uplift_runner.py — mirrors run_doc_linker_uplift / run_query_strategy_uplift / run_clustering_uplift shape. Result type VocabBridgeUplift{n_rows, mean_ndcg_off, mean_ndcg_on} with .uplift property.
2. test(uplift): three unit tests (empty input, skip-without-expected_top_k, synthetic-row smoke).
3. test(bench-gate): test_vocab_bridge_ship_gate_runner_present in tests/bench_gate/test_vocab_bridge_uplift.py — same skip-if-runner-absent + skip-if-no-expected_top_k pattern as [v2.0] Intentional clustering — co-locate related beliefs for multi-fact coherence #436. Module docstring updated for the extended row schema (expected_top_k joins expected_canonicals).

Bench evidence

I authored a 20-row labelled corpus on the lab side (~/projects/aelfrice-lab/tests/corpus/v2_0/vocab_bridge/{seed,v0_1,v0_2_charitable}.jsonl, gitea origin only per directory-of-origin rule, lab commit 0f0f2b3) and ran both gates against the substrate at github/main HEAD 24134b6:

Gate	Result
Precondition (appends ≥1 expected canonical, threshold ≥50%)	0/20 rows
Strict NDCG@k uplift (this PR's runner)	ON=0.6623 OFF=0.6863 uplift=-0.0240 (n=20)

Both gates fail. The bridge regresses retrieval slightly (~2-3 NDCG points) across all 20 rows.

Substrate root cause

Per-row debug shows the bridge appends HRR-noise tokens (across, config, run) instead of the designed canonicals (vim, emacs, sqlite, compression, ruff, etc.) — even on rows designed maximally favorably (canonicals strictly NOT in the raw query, anchored on multiple beliefs).

Inspection of _harvest() (src/aelfrice/vocab_bridge.py:203-263):

def _ingest(text: str) -> None:
    for ent in extract_entities(text):
        ...
        vocab.add(low, low)          # canonical = surface
    for tok in bm25_tokenize(text):
        ...
        vocab.add(tok, tok)          # canonical = surface

Every surface form gets added as its own canonical. There is no cross-form mapping anywhere in the harvest. So bridge_vec ≈ sum_t bind(t, t) (every token bound to itself) plus HRR superposition noise. The cleanup-memory query for any unbind(query_token, bridge_vec) recovers the same token (self-similarity ≈ 1) — but the rewriter then drops it because it's already in the query (vocab_bridge.py:310-311). What gets through is HRR noise: random near-neighbors in the cleanup matrix that exceed the 1/sqrt(dim) ≈ 0.022 noise floor.

The "vocabulary-gap-recovery" claim from the spec presupposes the harvest step distinguishes surface forms from canonicals. The shipped harvest doesn't — it treats every observed token as both. There's no surface→canonical structure for HRR to recover.

What this PR ships

The PR is correct and lands the harness regardless of substrate verdict:

• The runner produces deterministic byte-identical results across runs (verified manually on 3 invocations of the same row set).
• The unit tests pass standalone (the strict gate skips on public CI per the existing bench_gated autouse marker).
• The bench-gate scaffold gives operator-side a one-step pytest -k vocab_bridge_ship_gate AELFRICE_CORPUS_ROOT=... that fires only when both runner and labelled rows are present.

Recommended next step (operator decision)

Two options:

1. Close-as-not-pursued (mirror Deduplication module (dedup) — v2.0 evaluation #197 dedup R2 outcome). The bench cleared no threshold on plausible corpora; the substrate's "vocab gap recovery" claim isn't supported by _harvest's self-mapping behavior. use_vocab_bridge stays default-OFF; the substrate stays in tree as a lane other modules can extend; the umbrella row for [v2.0] HRR vocabulary bridge — close vocabulary-gap-recovery claim #433 flips to "evaluated, declined."

2. Re-spec the bridge. Rework _harvest to extract surface↔canonical pairs (e.g. via lemmatization, abbreviation expansion, anchor-text aliasing rules, or a pre-trained subword tokeniser) so HRR cleanup has actual structure to recover. That's a fresh design loop and a fresh PR — out of scope for [v2.0] HRR vocabulary bridge — close vocabulary-gap-recovery claim #433 as filed.

Memory belief lock policy on #197 says "we don't ship a v2.x retrieval feature without positive bench delta" — option 1 is the consistent pick.

Companion to issue #474 umbrella refresh (separate small PR).

Summary by Sourcery

Add an NDCG@k-based uplift runner and ship gate for the vocabulary bridge feature flag.

New Features:

• Introduce a VocabBridge uplift driver that runs retrieve_v2 with and without the vocabulary bridge and reports mean NDCG@k uplift.

Tests:

• Add unit tests covering empty input, skipping rows without expected_top_k, and a synthetic end-to-end row for the VocabBridge uplift driver.
• Add a bench-gated test that wires the vocab bridge corpus into the uplift runner and enforces strictly positive NDCG@k uplift before enabling the feature by default.

Summary by CodeRabbit

• Tests
  • Added comprehensive test coverage for Vocabulary Bridge feature validation, including uplift measurement and schema validation.
  • Introduced new gated test that measures NDCG@k uplift performance when Vocabulary Bridge is toggled on versus off.
  • Added unit tests verifying correct handling of edge cases and end-to-end functionality.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR introduces a new bench-gate uplift runner for Vocabulary Bridge (#433). It adds a VocabBridgeUplift result type, a run_vocab_bridge_uplift function that evaluates NDCG@k improvements when vocab bridge is enabled, a benchmark-gated corpus test, and three unit tests covering edge cases and end-to-end behavior.

Changes

Vocab Bridge Uplift Runner

Layer / File(s)	Summary
Data Contracts tests/bench_gate/test_vocab_bridge_uplift.py, tests/retrieve_uplift_runner.py	VocabBridgeUplift dataclass with mean_ndcg_off, mean_ndcg_on, and computed uplift property. Row schema extended with optional k and expected_top_k fields for ground-truth ranking.
Core Runner Implementation tests/retrieve_uplift_runner.py	Module documentation for vocab bridge bench gate. _seed_store_for_vocab_bridge helper function seeds MemoryStore with beliefs and synthesizes citing beliefs linked via CITES edges. run_vocab_bridge_uplift executes OFF/ON retrieval evaluation per row (skipping rows without expected_top_k), computes NDCG@k, and returns aggregated mean NDCG and uplift.
Benchmark-Gate Test tests/bench_gate/test_vocab_bridge_uplift.py	New pytest test loads vocab_bridge corpus, requires uplift runner module, skips if no rows have expected_top_k, runs uplift evaluation, and asserts strictly positive uplift with diagnostic metrics on failure.
Unit Tests tests/test_retrieve_uplift_runner.py	Empty-input test returns n_rows=0 and uplift=0.0. Rows-without-ranking test confirms rows lacking expected_top_k are excluded from scoring. End-to-end test exercises OFF/ON arms on synthetic row with anchors, expected_canonicals, and expected_top_k, asserting bounded NDCG outputs.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• robotrocketscience/aelfrice#425: Modifies the same test harness (tests/retrieve_uplift_runner.py) to add uplift runner functions and dataclasses; this PR extends similar patterns for vocab-bridge-specific NDCG@k evaluation.

Suggested labels

author-Leibniz

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding a vocab_bridge NDCG@k uplift driver for the bench-gate and includes the key finding that the bridge regresses performance.
Description check	✅ Passed	The description includes a comprehensive summary, linked issue (#433), proper type-of-change classification (feat), detailed test plan, and substantial notes for reviewers covering the root cause analysis and recommended next steps.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/issue-433-vocab-bridge-uplift

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sourcery-ai]

Reviewer's Guide

Implements a strict NDCG@k uplift driver for the vocab bridge bench gate (#433), including a new runner that evaluates retrieval with and without the vocab bridge, supporting seeding helpers and result typing, plus unit tests and a bench-gated ship‑gate test that asserts strictly positive uplift when labelled corpus rows are present.

File-Level Changes

Change	Details	Files
Add a vocab-bridge NDCG@k uplift runner that evaluates retrieve_v2 with and without use_vocab_bridge against labelled corpus rows and reports mean NDCG and uplift.	Introduce VocabBridgeUplift dataclass with n_rows, mean_ndcg_off, mean_ndcg_on, and computed uplift property. Implement _seed_store_for_vocab_bridge to populate a MemoryStore from vocab_bridge row schema, including beliefs and CITES edges with anchor_text from anchors. Add run_vocab_bridge_uplift to run retrieve_v2 twice per row (bridge off/on) on fresh stores, compute NDCG@k vs expected_top_k, skip unrankable rows, and aggregate mean scores.	tests/retrieve_uplift_runner.py
Add unit tests covering the vocab-bridge uplift runner behavior and edge cases.	Test empty input returns zeroed metrics and zero rows. Test rows lacking expected_top_k are skipped rather than counted with zero score, keeping uplift undiluted. Test a synthetic row runs end-to-end with n_rows==1 and NDCG metrics bounded between 0 and 1.	tests/test_retrieve_uplift_runner.py
Add a bench-gated ship-gate test that wires the vocab-bridge uplift runner into the bench corpus and enforces strictly positive NDCG@k uplift when a labelled corpus is present.	Load vocab_bridge corpus rows and assert non-empty corpus for the gate. Conditionally import the uplift runner module, skipping if absent with an operator-focused reason message. Skip when no row carries expected_top_k and otherwise run the uplift driver and assert uplift > 0 with a detailed failure message summarizing ON/OFF NDCG and n_rows.	tests/bench_gate/test_vocab_bridge_uplift.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've left some high level feedback:

• _seed_store_for_vocab_bridge duplicates much of the belief/edge seeding logic from the existing vocab-bridge precondition seeder; consider factoring this into a shared helper so the store shape stays consistent if the schema or defaults change.
• _BENCH_TS is hard-coded to a specific future date; if this is only meant to be a stable, non-semantic timestamp you might prefer a more neutral constant (e.g. an epoch-like value or clearly fake date) and a short comment so it’s not confused with real chronology.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- _seed_store_for_vocab_bridge duplicates much of the belief/edge seeding logic from the existing vocab-bridge precondition seeder; consider factoring this into a shared helper so the store shape stays consistent if the schema or defaults change.
- _BENCH_TS is hard-coded to a specific future date; if this is only meant to be a stable, non-semantic timestamp you might prefer a more neutral constant (e.g. an epoch-like value or clearly fake date) and a short comment so it’s not confused with real chronology.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[github-actions]

This PR is now behind main. Rebase locally so your commit signatures stay intact:

git fetch origin && git checkout 'feat/issue-433-vocab-bridge-uplift' && git rebase origin/main
# resolve conflicts if any, then
git push --force-with-lease

Auto-rebase was removed because the bot has no signing key; rebasing as the bot strips author signatures and the required_signatures rule on main then blocks the merge. See #341.

[robotrocketscience]

[claim:review:Noether:2026-05-10T03:29:06Z]

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/bench_gate/test_vocab_bridge_uplift.py`:
- Around line 155-157: The test currently hard-fails when
load_corpus_module(aelfrice_corpus_root, "vocab_bridge") returns no rows; change
that behavior to skip the test instead. Replace the assert rows check with a
conditional that calls pytest.skip(...) with a clear reason when rows is empty
or falsy (and add an import for pytest if not already present). Keep the rest of
the test using the loaded rows when present.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 0418c668-e1e8-449c-bba2-8d0edb842a8f

📥 Commits

Reviewing files that changed from the base of the PR and between deb59cd and 37e0764.

📒 Files selected for processing (3)

• tests/bench_gate/test_vocab_bridge_uplift.py
• tests/retrieve_uplift_runner.py
• tests/test_retrieve_uplift_runner.py

[coderabbitai]

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Skip on empty corpus instead of failing the bench gate

Line 156 hard-fails when the corpus is absent/empty, which makes this gate brittle in environments without lab resources. This should skip like the other absent-resource paths.

Proposed fix

     rows = load_corpus_module(aelfrice_corpus_root, "vocab_bridge")
-    assert rows, "vocab_bridge corpus produced zero rows"
+    if not rows:
+        pytest.skip(
+            "vocab_bridge corpus produced zero rows; "
+            "add rows under tests/corpus/v2_0/vocab_bridge/*.jsonl "
+            "before enabling this ship gate"
+        )

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	rows = load_corpus_module(aelfrice_corpus_root, "vocab_bridge")
	assert rows, "vocab_bridge corpus produced zero rows"
	rows = load_corpus_module(aelfrice_corpus_root, "vocab_bridge")
	if not rows:
	pytest.skip(
	"vocab_bridge corpus produced zero rows; "
	"add rows under tests/corpus/v2_0/vocab_bridge/*.jsonl "
	"before enabling this ship gate"
	)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/bench_gate/test_vocab_bridge_uplift.py` around lines 155 - 157, The
test currently hard-fails when load_corpus_module(aelfrice_corpus_root,
"vocab_bridge") returns no rows; change that behavior to skip the test instead.
Replace the assert rows check with a conditional that calls pytest.skip(...)
with a clear reason when rows is empty or falsy (and add an import for pytest if
not already present). Keep the rest of the test using the loaded rows when
present.

[robotrocketscience]

LGTM — rebased and FF-merging.

Verified

• Rebased onto current github/main (3 atomic commits re-signed, all G).
• Local pytest on rebased tree: tests/test_retrieve_uplift_runner.py tests/bench_gate/test_vocab_bridge_uplift.py → 20 passed, 2 skipped (the bench-gate skips on public CI without AELFRICE_CORPUS_ROOT, as designed — same skip-policy contract as #291 query-strategy and #436 clustering gates).
• Discretion grep on diff: clean.
• Runner shape mirrors run_doc_linker_uplift / run_query_strategy_uplift — mean_ndcg_off / mean_ndcg_on / uplift field names preserve the branch-free formatter.
• expected_top_k row schema is a clean superset of the precondition row schema (expected_canonicals stays); rows can carry one or both, and the runner skips ones without ranking ground truth so they don't dilute the mean.
• Bench-gate test correctly assertion-pairs: pytest.importorskip on the runner module + skip when no row carries expected_top_k + assert results.uplift > 0 for the strict gate.

Substrate diagnosis spot-checked

Walked src/aelfrice/vocab_bridge.py:_harvest() on github/main HEAD. The PR body's claim is exactly what's there: every observed surface form is added as its own canonical (vocab.add(low, low) at line 223 for entities, vocab.add(tok, tok) at line 243 for BM25 tokens). There is no surface→canonical mapping logic in the harvest. The HRR superposition built downstream is therefore bridge_vec ≈ Σ_t bind(t, t) — self-bound tokens plus noise — and the rewriter's deduplication step (vocab_bridge.py:310-311) drops any token that's already in the query, so the rewrite path emits noise rather than canonicals. The "vocabulary-gap-recovery" claim presupposes a structure the shipped harvest doesn't build.

Per-row "appends ≥1 expected canonical" failing 0/20 plus strict NDCG@k uplift = -0.0240 is consistent with that diagnosis: the bridge isn't merely a no-op, it actively perturbs the query with low-amplitude noise, dropping retrieval slightly.

Recommendation

Merge as evidence + harness. Keep #433 open; #536 (re-spec _harvest) is the right follow-up. Do not flip use_vocab_bridge to default-on at any point on the current substrate — the bench gate scaffolded here will fire correctly the moment a future substrate change re-enables the lab corpus runner, so the gate is its own forward-protection.

(Cross-session same-account auth blocks formal --approve; merging by local FF push per protocol.)

[robotrocketscience]

[release:review:Noether:2026-05-10T03:35:47Z]