Skip to content

feat(vocab_bridge): pure HRR query-side vocabulary bridge + retrieve_v2 integration (#433) - #495

Merged
robotrocketscience merged 5 commits into
mainfrom
feat/issue-433-vocab-bridge
May 8, 2026
Merged

feat(vocab_bridge): pure HRR query-side vocabulary bridge + retrieve_v2 integration (#433)#495
robotrocketscience merged 5 commits into
mainfrom
feat/issue-433-vocab-bridge

Conversation

@robotrocketscience

@robotrocketscience robotrocketscience commented May 8, 2026

Copy link
Copy Markdown
Owner

Closes-progress on #433 (HRR vocabulary bridge). Lands the mechanically-correct, default-OFF form per the spec's bench-gate / ship-or-defer policy (docs/feature-hrr-vocab-bridge.md § Bench-gate).

What landed

1. src/aelfrice/vocab_bridge.py (commit 1)

VocabBridge builds a single (dim,) HRR composite from sum_{c, s in surface_forms(c)} bind(token_vec[s], canonical_vec[c]) over the corpus's surface-form universe. Pure linear algebra — no learned components, no LLM call, no embedding model. Surface-form harvest combines:

  1. Incoming anchor text (store.iter_incoming_anchor_text, [retrieval] Augmented BM25F (incoming-edge anchor text) + vectorized BM25 sparse matvec #148 substrate) — same field BM25F consumes
  2. Belief content (entity-extractor lane fallback per spec open-question 1; the beliefs.entity_id column does not yet exist on the v2.0.0 schema)
  3. Lock-asserted statements — included via source Add CI workflows, scan config, and align README #2 in this MVP

Surface-form filter: single-token canonicals only (multi-word noun phrases are dropped — the bridge rewrites query tokens, not query spans), tokens ≥ 3 chars (one- and two-letter words bind generically and bloat the bridge with no recoverable signal).

rewrite() unbinds each query token from the composite, cleanup-queries against the canonical-vec set, and appends canonicals above the noise floor (1/sqrt(dim)). Original-query tokens are preserved verbatim — bridged candidates are appended, never substituted, so flag-OFF / unseen-token paths are byte-identical to the pre-bridge call.

Determinism mirrors HRRStructIndex: dual-Generator stream split (canonical_rngseed, token_rngseed ^ _TOKEN_SALT), path-derived MD5-based 64-bit seed.

VocabBridgeCache mirrors BM25IndexCache: subscribes to the store's invalidation hook, rebuilds on next get() after any belief / edge mutation.

2. retrieve_v2() integration (commit 2)

  • New flag use_vocab_bridge with the established 4-stage precedence (env AELFRICE_VOCAB_BRIDGE > kwarg > TOML [retrieval] use_vocab_bridge > default-OFF).
  • New kwarg vocab_bridge_cache: VocabBridgeCache | None = None for amortised builds across long-running consumers.
  • use_hrr becomes a deprecated alias for use_vocab_bridge (per spec §Configuration). Lab v2.0.0 adapters that pass use_hrr=True route to the bridge automatically; use_vocab_bridge=False wins when both are passed.

When the flag resolves True, the bridge runs before lane fan-out:

query
  -> [bridge.rewrite(query) if use_vocab_bridge else query]
  -> retrieve() lane fan-out

3. Tests (commits 3 + 4)

  • tests/test_vocab_bridge.py — 18 unit tests: build determinism, algebraic self-recovery cosine, token-universe filter, empty-store no-op, anchor-text harvest, distinct stream verification, etc.
  • tests/test_vocab_bridge_integration.py — 14 integration tests: flag precedence, byte-identical default-OFF path, deprecated use_hrr alias semantics, VocabBridgeCache injection + invalidation.
  • tests/bench_gate/test_vocab_bridge_uplift.py — A2 precondition. Skips on public CI; with AELFRICE_CORPUS_ROOT populated, asserts ≥50% row-level coverage of expected_canonicals (loose threshold; A2's strict NDCG@k is the lab-side follow-up).

4. docs/CONFIG.md (commit 5)

use_vocab_bridge added to the [retrieval] knob list, the .aelfrice.toml example, and a dedicated Keys subsection with the rewrite-pipeline diagram, deprecated-alias note, and precedence chain.

Acceptance status

Test plan

  • uv run pytest tests/test_vocab_bridge.py tests/test_vocab_bridge_integration.py -q — 32 passed
  • uv run pytest tests/bench_gate/test_vocab_bridge_uplift.py -q — skipped (public CI, corpus absent)
  • uv run pytest tests/ -q --ignore=tests/bench_gate — 2870 passed, 24 skipped, no regressions vs main
  • AELFRICE_CORPUS_ROOT=~/projects/aelfrice-lab/tests/corpus/v2_0 uv run pytest tests/bench_gate/test_vocab_bridge_uplift.py — operator/lab-side; ratifies A2 precondition

Follow-up issues

Refs #433, #154 (composition tracker), #148 (BM25F anchor substrate), #152 (HRR struct index — sibling lane).

Summary by Sourcery

Introduce an HRR-based query-side vocabulary bridge and integrate it into the retrieval pipeline behind a feature flag.

New Features:

  • Add an in-memory HRR-based VocabBridge that harvests surface-form tokens from anchor text and belief content to append canonical entity tokens to queries.
  • Provide a VocabBridgeCache that lazily builds and invalidates the vocabulary bridge in response to store mutations for reuse across queries.
  • Expose a new use_vocab_bridge flag and optional vocab_bridge_cache parameter on retrieve_v2 to enable query rewrite before lane fan-out, with use_hrr as a deprecated alias.

Enhancements:

  • Extend configuration and environment handling to support the use_vocab_bridge flag, including env var, TOML, and kwarg precedence resolution, defaulting to off.

Documentation:

  • Document the use_vocab_bridge retrieval setting, its behavior, precedence, and relationship to the deprecated use_hrr alias in CONFIG.md.

Tests:

  • Add unit tests for VocabBridge covering determinism, algebraic properties, surface-form harvesting, and boundary conditions.
  • Add integration tests for retrieve_v2 covering flag precedence, deprecated use_hrr alias behavior, cache injection, and default-off byte-identity guarantees.
  • Add a bench-gate test harness for measuring vocabulary-bridge uplift on a labeled corpus, gated for lab-side execution.

Closes the vocabulary-gap-recovery claim on the query side. A query
token whose surface form does not appear verbatim in the corpus's
canonical vocabulary gets bridged to one or more canonical-entity
tokens before any retrieval lane fires.

Pure linear algebra over the surface-form universe the corpus already
exposes — no learned components, no LLM call, no embedding model.
Builds a single (dim,) HRR composite from sum_{c, s in surface_forms(c)}
bind(token_vec[s], canonical_vec[c]); rewrite step unbinds the query
token and cleanup-memory-queries against the canonical vec set.

Surface-form harvest combines incoming-anchor-text edges (#148) and
belief content (entity-extractor lane fallback per spec open-question
1; the beliefs.entity_id column does not yet exist on the v2.0.0
schema). Lock-asserted statements are included via source #2.

Determinism mirrors HRRStructIndex: dual-Generator pattern with a
salted seed for the token stream, path-derived seed convention.

In-memory only at v2.0.0; no schema impact, no new dependency
(numpy already runtime-required at v1.5.0). Module is self-contained
— not yet wired into retrieve_v2 (commit 2 lands the flag).

Spec: docs/feature-hrr-vocab-bridge.md
Adds the standard 4-stage flag-resolution surface (env > kwarg > TOML
> default-OFF) for the HRR vocabulary bridge, plus a VocabBridgeCache
mirror of BM25IndexCache so long-running consumers can amortise the
build cost across queries.

When the flag resolves True, retrieve_v2 builds (or fetches a cached)
VocabBridge against the store and rewrites the query before the lane
fan-out. The bridge appends canonical-entity rewrites; original tokens
are preserved verbatim, so a flag-OFF call is byte-identical to the
pre-bridge behaviour.

`use_hrr` becomes a deprecated alias for `use_vocab_bridge` per spec
§Configuration. Lab v2.0.0 adapters that pass `use_hrr=True` route to
the bridge automatically; new callers should use `use_vocab_bridge`
directly. Default for both is None (falls through to the resolution
chain).

Phase-1 ship: mechanically-correct, default-OFF. The bench gate (A2:
NDCG@k uplift on the labeled vocab_bridge fixture) is deferred to a
follow-up PR after the lab-side bench cut. Spec ship-or-defer policy
allows this — see docs/feature-hrr-vocab-bridge.md § Bench-gate.
Unit (test_vocab_bridge.py — 18 tests):
- Build determinism: same store + same path → byte-identical bridge_vec;
  different paths → different vectors; explicit seed overrides path.
- Algebraic invariant: unbind(token_vec[c], bridge_vec) recovers
  canonical_vec[c] above noise_floor for every canonical.
- Token-universe filter: unseen query tokens short-circuit; tokens
  shorter than 3 chars drop at harvest.
- Empty-store no-op; high min_score drops all appendees; top_k=0
  short-circuits.
- Anchor-text harvest: tokens visible only via incoming anchors land
  in the canonical set.
- Stream split: token_vec and canonical_vec for the same canonical
  are distinct draws (mirrors HRRStructIndex's id/role split).

Integration (test_vocab_bridge_integration.py — 14 tests):
- Flag precedence: kwarg/env/TOML/default-OFF, env-garbage falls
  through, TOML resolves when kwarg+env are unset.
- Default-OFF byte-identity: pre-bridge call shape is preserved when
  the flag does not resolve True.
- Deprecated `use_hrr` alias: True routes to bridge; explicit
  use_vocab_bridge wins when both are passed; None falls through.
- VocabBridgeCache: explicit cache injection works; invalidation
  hook fires on store mutation and rebuilds on next get().

Cross-talk magnitudes are corpus-size-dependent (HRR fundamentals);
small-fixture tests verify structural correctness only. The bench
gate (A2 NDCG@k uplift on a labeled corpus) is the empirical-uplift
gate and is deferred to the lab-side bench cut.
Lab-mounted bench gate. Skips on public CI via the autouse `bench_gated`
marker. When AELFRICE_CORPUS_ROOT points at a populated
tests/corpus/v2_0/vocab_bridge/ directory, asserts the upstream
invariant A2 depends on: per labeled (query, expected_canonicals)
row, the bridge appends at least one expected canonical above the
noise floor.

Threshold is loose (≥ 50% row-level coverage) — a regression in
surface-form harvest or rewrite that drops ALL bridging trips
immediately, but per-row tuning is left to A2-strict NDCG@k once
the budget-rewrite-style follow-up runs the full retrieve_v2 path
against the corpus.

Row schema documented in the gate's module docstring; same shape
convention as compression_uplift (#434).
Adds use_vocab_bridge to:
- The [retrieval] section header line listing v1.3+ knobs
- The .aelfrice.toml example block (alongside use_type_aware_compression)
- A dedicated Keys subsection with the rewrite-pipeline diagram, the
  use_hrr deprecation note, and the precedence chain

Per spec §A4 (acceptance: documentation entry). The bench-gate
follow-up will be linked once the lab-side cut lands.
@sourcery-ai

sourcery-ai Bot commented May 8, 2026

Copy link
Copy Markdown

Reviewer's Guide

Implements a deterministic, in-memory HRR-based query-side vocabulary bridge over the store, wires it into retrieve_v2 behind a new use_vocab_bridge feature flag (with env/TOML/kwarg precedence and a deprecated use_hrr alias), adds a cache layer to amortise bridge build cost, and documents plus bench-gates the feature via new tests and CONFIG docs updates.

Class diagram for VocabBridge and VocabBridgeCache

classDiagram
    class _Vocabulary {
        +dict~str, set~str~~ canonical_to_surfaces
        +add(canonical: str, surface: str) void
        +canonicals() list~str~
        +surface_forms(canonical: str) list~str~
        +all_surfaces() list~str~
    }

    class VocabBridge {
        +int dim
        +int seed
        +list~str~ canonicals
        +list~str~ surfaces
        +dict~str, Vector~ canonical_vecs
        +dict~str, Vector~ token_vecs
        +Vector bridge_vec
        -Vector _cleanup_matrix
        -Vector _cleanup_norms
        +build(store: MemoryStore, store_path: str, seed: int) void
        +rewrite(query: str, top_k: int, min_score: float) str
        +noise_floor() float
        +size() int
        -_harvest(store: MemoryStore) _Vocabulary
        -_cleanup_query(probe: Vector, top_k: int) list~tuple~str, float~~
    }

    class VocabBridgeCache {
        +MemoryStore store
        +int dim
        +str store_path
        +int seed
        -VocabBridge _bridge
        -bool _subscribed
        +__post_init__() void
        +get() VocabBridge
        +invalidate() void
    }

    class MemoryStore {
        +add_invalidation_callback(callback: callable) void
        +iter_incoming_anchor_text() iterator
        +list_belief_ids() list~str~
        +get_belief(belief_id: str) Belief | None
    }

    class Belief {
        +str content
        +int lock_level
    }

    class Vector

    class extract_entities {
        <<function>>
    }

    class bm25_tokenize {
        <<function>>
    }

    class LOCK_USER {
        <<constant>>
    }

    VocabBridgeCache o-- VocabBridge : caches
    VocabBridgeCache --> MemoryStore : uses
    VocabBridge --> MemoryStore : builds_from
    VocabBridge --> _Vocabulary : aggregates
    VocabBridge --> Vector : uses
    VocabBridge --> bm25_tokenize : tokenizes
    VocabBridge --> extract_entities : extracts
    VocabBridge --> LOCK_USER : checks
    MemoryStore --> Belief : returns
Loading

Flow diagram for use_vocab_bridge flag resolution precedence

flowchart LR
    A[Start flag resolution] --> B[Read env AELFRICE_VOCAB_BRIDGE]
    B --> C{Env is truthy or falsy?}
    C -- Yes --> D[Return env value]
    C -- No --> E[Check explicit kwarg use_vocab_bridge]
    E --> F{explicit is not None?}
    F -- Yes --> G[Return explicit]
    F -- No --> H[Read TOML retrieval.use_vocab_bridge]
    H --> I{TOML value is not None?}
    I -- Yes --> J[Return TOML value]
    I -- No --> K[Return default False]
Loading

File-Level Changes

Change Details Files
Add deterministic HRR-based query-side vocabulary bridge and cache over MemoryStore.
  • Introduce VocabBridge dataclass that harvests surface forms from incoming anchor text and belief content, builds canonical and token HRR vectors with dual RNG streams and a single composite bridge_vec, and provides a rewrite() method that appends canonical tokens above a noise-floor threshold.
  • Implement internal _Vocabulary helper to aggregate canonical-to-surface mappings with token-length and multi-word filters, plus a path-derived MD5-based seed helper for deterministic RNG seeding.
  • Add VocabBridge._cleanup_query(), noise_floor(), and size() helpers, plus inlined cleanup-memory matrix/norm caching for efficient cosine-based canonical recovery.
  • Introduce VocabBridgeCache wrapper that subscribes to the store invalidation hook, lazily builds a VocabBridge on first get(), and invalidates on store mutations with optional dim/store_path/seed configuration.
src/aelfrice/vocab_bridge.py
Integrate vocabulary bridge into retrieve_v2 behind a new feature flag with env/TOML/kwarg precedence and deprecated alias handling.
  • Define VOCAB_BRIDGE_FLAG and ENV_VOCAB_BRIDGE constants and implement _env_vocab_bridge_override() to normalise truthy/falsy env values.
  • Add resolve_use_vocab_bridge() that applies 4-stage precedence: env var, explicit kwarg, TOML [retrieval] use_vocab_bridge, then default False.
  • Extend retrieve_v2 signature to accept use_vocab_bridge and vocab_bridge_cache kwargs, change use_hrr default to None, and implement alias resolution where explicit use_vocab_bridge wins over use_hrr.
  • Insert pre-fan-out query rewrite: when resolve_use_vocab_bridge() returns True, obtain a VocabBridge via an injected or new VocabBridgeCache and rewrite the query string before passing it to the existing retrieval lanes.
src/aelfrice/retrieval.py
Add unit tests for VocabBridge build, algebraic properties, and integration tests for flag resolution and retrieve_v2 behaviour.
  • Create tests/test_vocab_bridge.py with fixtures to populate a MemoryStore and tests covering empty-store behaviour, deterministic builds from path/seed, vector-stream distinctness, self-recovery cosine vs noise_floor, unseen/short-token handling, rewrite determinism and preservation of original query, and anchor-text harvesting.
  • Create tests/test_vocab_bridge_integration.py to verify resolve_use_vocab_bridge precedence (env/kwarg/TOML), default-OFF and explicit-OFF paths are byte-identical, flag-ON paths execute without error, use_hrr alias semantics (including override by use_vocab_bridge), and correct use plus invalidation of VocabBridgeCache in retrieve_v2.
tests/test_vocab_bridge.py
tests/test_vocab_bridge_integration.py
Add bench-gate test suite to validate vocabulary bridge uplift prerequisite on a labeled corpus.
  • Introduce tests/bench_gate/test_vocab_bridge_uplift.py that loads a vocab_bridge corpus, seeds a MemoryStore with beliefs and optional synthetic anchor edges, builds a VocabBridge per row, and asserts that rewrites append at least one expected canonical for ≥50% of annotated rows.
  • Integrate with existing bench_gated infrastructure and corpus loader, including a skip path when no labeled rows are present or corpus root is absent.
tests/bench_gate/test_vocab_bridge_uplift.py
Document the new use_vocab_bridge configuration knob and its behaviour in CONFIG docs.
  • Extend CONFIG.md to include use_vocab_bridge in the [retrieval] knob list and sample .aelfrice.toml, mirroring other retrieval flags.
  • Add a dedicated CONFIG.md subsection explaining the vocabulary bridge rewrite pipeline, its non-lane nature, usage semantics and precedence chain, and the deprecated use_hrr alias on retrieve_v2.
docs/CONFIG.md

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@coderabbitai

coderabbitai Bot commented May 8, 2026

Copy link
Copy Markdown

Warning

Rate limit exceeded

@robotrocketscience has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 9 minutes and 39 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 499843b7-c547-4760-832a-8d8aa3389b87

📥 Commits

Reviewing files that changed from the base of the PR and between a6f1582 and 0ed6d83.

📒 Files selected for processing (6)
  • docs/CONFIG.md
  • src/aelfrice/retrieval.py
  • src/aelfrice/vocab_bridge.py
  • tests/bench_gate/test_vocab_bridge_uplift.py
  • tests/test_vocab_bridge.py
  • tests/test_vocab_bridge_integration.py
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/issue-433-vocab-bridge

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@robotrocketscience robotrocketscience added author-Gylf PR coordination mutex attn:review Needs review (PR open, awaiting reviewer) labels May 8, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 3 issues, and left some high level feedback:

  • In retrieve_v2, constructing a new VocabBridgeCache(store) when vocab_bridge_cache is None will register a fresh invalidation callback on every call and leak callbacks on the store; consider either reusing a singleton cache per store or deferring subscription until the cache is intended to be long-lived.
  • In VocabBridge._cleanup_query, normalized = self._cleanup_matrix / self._cleanup_norms[:, np.newaxis] is recomputed on every rewrite; you could precompute and store the row-normalized matrix at build time to avoid this per-query cost.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In `retrieve_v2`, constructing a new `VocabBridgeCache(store)` when `vocab_bridge_cache` is `None` will register a fresh invalidation callback on every call and leak callbacks on the store; consider either reusing a singleton cache per store or deferring subscription until the cache is intended to be long-lived.
- In `VocabBridge._cleanup_query`, `normalized = self._cleanup_matrix / self._cleanup_norms[:, np.newaxis]` is recomputed on every rewrite; you could precompute and store the row-normalized matrix at build time to avoid this per-query cost.

## Individual Comments

### Comment 1
<location path="src/aelfrice/vocab_bridge.py" line_range="335-336" />
<code_context>
+        probe_norm = float(np.linalg.norm(probe))
+        if probe_norm == 0:
+            return []
+        normalized = self._cleanup_matrix / self._cleanup_norms[:, np.newaxis]
+        sims = (normalized @ (probe / probe_norm)).astype(np.float64)
+        n = len(self.canonicals)
+        k = min(top_k, n)
</code_context>
<issue_to_address>
**suggestion (performance):** Avoid recomputing the normalized cleanup matrix on every rewrite for better performance.

In `_cleanup_query`, `normalized` is recomputed on every call even though `_cleanup_matrix` and `_cleanup_norms` are fixed after `build()`, adding an extra O(N * dim) pass and allocation per rewrite. Consider precomputing and storing the normalized matrix (or normalizing rows in-place once) in `build()` so `_cleanup_query` only performs the matrix–vector multiply and top‑k selection.

Suggested implementation:

```python
        if (
            self._cleanup_matrix is None
            or self._cleanup_norms is None
            or getattr(self, "_normalized_cleanup_matrix", None) is None
        ):
            return []

```

```python
        probe_norm = float(np.linalg.norm(probe))
        if probe_norm == 0:
            return []
        sims = (
            self._normalized_cleanup_matrix @ (probe / probe_norm)
        ).astype(np.float64)

```

To fully implement the optimization, you should:

1. In the class's `build()` (or equivalent) method, after computing `self._cleanup_matrix` and `self._cleanup_norms`, precompute and store the normalized matrix, e.g.:
   ```python
   # After _cleanup_matrix (shape: [N, dim]) and _cleanup_norms (shape: [N]) are finalized
   norms = self._cleanup_norms
   # Avoid division by zero; skip or mask zero-norm rows as appropriate
   nonzero = norms != 0
   normalized = np.zeros_like(self._cleanup_matrix, dtype=np.float64)
   normalized[nonzero] = (
       self._cleanup_matrix[nonzero] / norms[nonzero, np.newaxis]
   )
   self._normalized_cleanup_matrix = normalized
   ```
2. Ensure `self._normalized_cleanup_matrix` is initialized (e.g., to `None`) in `__init__` so the attribute exists before `build()` is called.
3. If there are other code paths that mutate `_cleanup_matrix` or `_cleanup_norms` after `build()`, make sure they also recompute `_normalized_cleanup_matrix` to keep them in sync.
</issue_to_address>

### Comment 2
<location path="src/aelfrice/vocab_bridge.py" line_range="293-300" />
<code_context>
+        # Canonical tokens already present in the raw query do not
+        # need rebridging — skip them so the rewriter is idempotent
+        # over its own output.
+        already_in_query: set[str] = set(bm25_tokenize(query))
+
+        for token in bm25_tokenize(query):
+            tv = self.token_vecs.get(token)
+            if tv is None:
</code_context>
<issue_to_address>
**suggestion (performance):** Reuse tokenization of the query instead of calling `bm25_tokenize` twice.

`bm25_tokenize(query)` is called twice: once for `already_in_query` and again in the loop. To avoid repeated work, compute `tokens = list(bm25_tokenize(query))` once, build `already_in_query = set(tokens)`, and then iterate over `tokens` in the loop.

```suggestion
        appended: list[str] = []
        appended_set: set[str] = set()
        # Canonical tokens already present in the raw query do not
        # need rebridging — skip them so the rewriter is idempotent
        # over its own output.
        tokens = list(bm25_tokenize(query))
        already_in_query: set[str] = set(tokens)

        for token in tokens:
```
</issue_to_address>

### Comment 3
<location path="docs/CONFIG.md" line_range="324" />
<code_context>

+### `use_vocab_bridge`
+
+Boolean, default `false`, opt-in (v2.1+, #433). Enables the HRR vocabulary-bridge query rewrite. When on, `retrieve_v2` builds (or fetches a cached) `VocabBridge` over the per-project store and prepends the rewrite stage before lane fan-out:
+
+```
</code_context>
<issue_to_address>
**suggestion (typo):** Clarify the phrase "fetches a cached" to avoid a dangling adjective.

"builds (or fetches a cached) `VocabBridge`" is slightly ungrammatical because "cached" lacks a noun. Consider rephrasing to "builds (or fetches a cached `VocabBridge`)" or "builds (or fetches a cached instance of `VocabBridge`)" for clarity.

```suggestion
Boolean, default `false`, opt-in (v2.1+, #433). Enables the HRR vocabulary-bridge query rewrite. When on, `retrieve_v2` builds (or fetches a cached instance of `VocabBridge`) over the per-project store and prepends the rewrite stage before lane fan-out:
```
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment on lines +335 to +336
normalized = self._cleanup_matrix / self._cleanup_norms[:, np.newaxis]
sims = (normalized @ (probe / probe_norm)).astype(np.float64)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (performance): Avoid recomputing the normalized cleanup matrix on every rewrite for better performance.

In _cleanup_query, normalized is recomputed on every call even though _cleanup_matrix and _cleanup_norms are fixed after build(), adding an extra O(N * dim) pass and allocation per rewrite. Consider precomputing and storing the normalized matrix (or normalizing rows in-place once) in build() so _cleanup_query only performs the matrix–vector multiply and top‑k selection.

Suggested implementation:

        if (
            self._cleanup_matrix is None
            or self._cleanup_norms is None
            or getattr(self, "_normalized_cleanup_matrix", None) is None
        ):
            return []
        probe_norm = float(np.linalg.norm(probe))
        if probe_norm == 0:
            return []
        sims = (
            self._normalized_cleanup_matrix @ (probe / probe_norm)
        ).astype(np.float64)

To fully implement the optimization, you should:

  1. In the class's build() (or equivalent) method, after computing self._cleanup_matrix and self._cleanup_norms, precompute and store the normalized matrix, e.g.:
    # After _cleanup_matrix (shape: [N, dim]) and _cleanup_norms (shape: [N]) are finalized
    norms = self._cleanup_norms
    # Avoid division by zero; skip or mask zero-norm rows as appropriate
    nonzero = norms != 0
    normalized = np.zeros_like(self._cleanup_matrix, dtype=np.float64)
    normalized[nonzero] = (
        self._cleanup_matrix[nonzero] / norms[nonzero, np.newaxis]
    )
    self._normalized_cleanup_matrix = normalized
  2. Ensure self._normalized_cleanup_matrix is initialized (e.g., to None) in __init__ so the attribute exists before build() is called.
  3. If there are other code paths that mutate _cleanup_matrix or _cleanup_norms after build(), make sure they also recompute _normalized_cleanup_matrix to keep them in sync.

Comment on lines +293 to +300
appended: list[str] = []
appended_set: set[str] = set()
# Canonical tokens already present in the raw query do not
# need rebridging — skip them so the rewriter is idempotent
# over its own output.
already_in_query: set[str] = set(bm25_tokenize(query))

for token in bm25_tokenize(query):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (performance): Reuse tokenization of the query instead of calling bm25_tokenize twice.

bm25_tokenize(query) is called twice: once for already_in_query and again in the loop. To avoid repeated work, compute tokens = list(bm25_tokenize(query)) once, build already_in_query = set(tokens), and then iterate over tokens in the loop.

Suggested change
appended: list[str] = []
appended_set: set[str] = set()
# Canonical tokens already present in the raw query do not
# need rebridging — skip them so the rewriter is idempotent
# over its own output.
already_in_query: set[str] = set(bm25_tokenize(query))
for token in bm25_tokenize(query):
appended: list[str] = []
appended_set: set[str] = set()
# Canonical tokens already present in the raw query do not
# need rebridging — skip them so the rewriter is idempotent
# over its own output.
tokens = list(bm25_tokenize(query))
already_in_query: set[str] = set(tokens)
for token in tokens:

Comment thread docs/CONFIG.md

### `use_vocab_bridge`

Boolean, default `false`, opt-in (v2.1+, #433). Enables the HRR vocabulary-bridge query rewrite. When on, `retrieve_v2` builds (or fetches a cached) `VocabBridge` over the per-project store and prepends the rewrite stage before lane fan-out:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (typo): Clarify the phrase "fetches a cached" to avoid a dangling adjective.

"builds (or fetches a cached) VocabBridge" is slightly ungrammatical because "cached" lacks a noun. Consider rephrasing to "builds (or fetches a cached VocabBridge)" or "builds (or fetches a cached instance of VocabBridge)" for clarity.

Suggested change
Boolean, default `false`, opt-in (v2.1+, #433). Enables the HRR vocabulary-bridge query rewrite. When on, `retrieve_v2` builds (or fetches a cached) `VocabBridge` over the per-project store and prepends the rewrite stage before lane fan-out:
Boolean, default `false`, opt-in (v2.1+, #433). Enables the HRR vocabulary-bridge query rewrite. When on, `retrieve_v2` builds (or fetches a cached instance of `VocabBridge`) over the per-project store and prepends the rewrite stage before lane fan-out:


import hashlib
from dataclasses import dataclass, field
from pathlib import Path
@robotrocketscience

Copy link
Copy Markdown
Owner Author

[claim:review:Setr:2026-05-08T19:30:17Z]

@robotrocketscience

Copy link
Copy Markdown
Owner Author

[claim:review:kulili:2026-05-08T19:31:33Z]

@robotrocketscience

Copy link
Copy Markdown
Owner Author

[release:review:kulili:2026-05-08T19:31:38Z]

@robotrocketscience
robotrocketscience merged commit 0ed6d83 into main May 8, 2026
21 of 29 checks passed
@robotrocketscience
robotrocketscience deleted the feat/issue-433-vocab-bridge branch May 8, 2026 19:32
@robotrocketscience

Copy link
Copy Markdown
Owner Author

[release:review:Setr:2026-05-08T19:32:32Z]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

attn:review Needs review (PR open, awaiting reviewer) author-Gylf PR coordination mutex

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants