feat(vocab_bridge): pure HRR query-side vocabulary bridge + retrieve_v2 integration (#433) - #495
Conversation
Closes the vocabulary-gap-recovery claim on the query side. A query
token whose surface form does not appear verbatim in the corpus's
canonical vocabulary gets bridged to one or more canonical-entity
tokens before any retrieval lane fires.
Pure linear algebra over the surface-form universe the corpus already
exposes — no learned components, no LLM call, no embedding model.
Builds a single (dim,) HRR composite from sum_{c, s in surface_forms(c)}
bind(token_vec[s], canonical_vec[c]); rewrite step unbinds the query
token and cleanup-memory-queries against the canonical vec set.
Surface-form harvest combines incoming-anchor-text edges (#148) and
belief content (entity-extractor lane fallback per spec open-question
1; the beliefs.entity_id column does not yet exist on the v2.0.0
schema). Lock-asserted statements are included via source #2.
Determinism mirrors HRRStructIndex: dual-Generator pattern with a
salted seed for the token stream, path-derived seed convention.
In-memory only at v2.0.0; no schema impact, no new dependency
(numpy already runtime-required at v1.5.0). Module is self-contained
— not yet wired into retrieve_v2 (commit 2 lands the flag).
Spec: docs/feature-hrr-vocab-bridge.md
Adds the standard 4-stage flag-resolution surface (env > kwarg > TOML > default-OFF) for the HRR vocabulary bridge, plus a VocabBridgeCache mirror of BM25IndexCache so long-running consumers can amortise the build cost across queries. When the flag resolves True, retrieve_v2 builds (or fetches a cached) VocabBridge against the store and rewrites the query before the lane fan-out. The bridge appends canonical-entity rewrites; original tokens are preserved verbatim, so a flag-OFF call is byte-identical to the pre-bridge behaviour. `use_hrr` becomes a deprecated alias for `use_vocab_bridge` per spec §Configuration. Lab v2.0.0 adapters that pass `use_hrr=True` route to the bridge automatically; new callers should use `use_vocab_bridge` directly. Default for both is None (falls through to the resolution chain). Phase-1 ship: mechanically-correct, default-OFF. The bench gate (A2: NDCG@k uplift on the labeled vocab_bridge fixture) is deferred to a follow-up PR after the lab-side bench cut. Spec ship-or-defer policy allows this — see docs/feature-hrr-vocab-bridge.md § Bench-gate.
Unit (test_vocab_bridge.py — 18 tests): - Build determinism: same store + same path → byte-identical bridge_vec; different paths → different vectors; explicit seed overrides path. - Algebraic invariant: unbind(token_vec[c], bridge_vec) recovers canonical_vec[c] above noise_floor for every canonical. - Token-universe filter: unseen query tokens short-circuit; tokens shorter than 3 chars drop at harvest. - Empty-store no-op; high min_score drops all appendees; top_k=0 short-circuits. - Anchor-text harvest: tokens visible only via incoming anchors land in the canonical set. - Stream split: token_vec and canonical_vec for the same canonical are distinct draws (mirrors HRRStructIndex's id/role split). Integration (test_vocab_bridge_integration.py — 14 tests): - Flag precedence: kwarg/env/TOML/default-OFF, env-garbage falls through, TOML resolves when kwarg+env are unset. - Default-OFF byte-identity: pre-bridge call shape is preserved when the flag does not resolve True. - Deprecated `use_hrr` alias: True routes to bridge; explicit use_vocab_bridge wins when both are passed; None falls through. - VocabBridgeCache: explicit cache injection works; invalidation hook fires on store mutation and rebuilds on next get(). Cross-talk magnitudes are corpus-size-dependent (HRR fundamentals); small-fixture tests verify structural correctness only. The bench gate (A2 NDCG@k uplift on a labeled corpus) is the empirical-uplift gate and is deferred to the lab-side bench cut.
Lab-mounted bench gate. Skips on public CI via the autouse `bench_gated` marker. When AELFRICE_CORPUS_ROOT points at a populated tests/corpus/v2_0/vocab_bridge/ directory, asserts the upstream invariant A2 depends on: per labeled (query, expected_canonicals) row, the bridge appends at least one expected canonical above the noise floor. Threshold is loose (≥ 50% row-level coverage) — a regression in surface-form harvest or rewrite that drops ALL bridging trips immediately, but per-row tuning is left to A2-strict NDCG@k once the budget-rewrite-style follow-up runs the full retrieve_v2 path against the corpus. Row schema documented in the gate's module docstring; same shape convention as compression_uplift (#434).
Adds use_vocab_bridge to: - The [retrieval] section header line listing v1.3+ knobs - The .aelfrice.toml example block (alongside use_type_aware_compression) - A dedicated Keys subsection with the rewrite-pipeline diagram, the use_hrr deprecation note, and the precedence chain Per spec §A4 (acceptance: documentation entry). The bench-gate follow-up will be linked once the lab-side cut lands.
Reviewer's GuideImplements a deterministic, in-memory HRR-based query-side vocabulary bridge over the store, wires it into retrieve_v2 behind a new use_vocab_bridge feature flag (with env/TOML/kwarg precedence and a deprecated use_hrr alias), adds a cache layer to amortise bridge build cost, and documents plus bench-gates the feature via new tests and CONFIG docs updates. Class diagram for VocabBridge and VocabBridgeCacheclassDiagram
class _Vocabulary {
+dict~str, set~str~~ canonical_to_surfaces
+add(canonical: str, surface: str) void
+canonicals() list~str~
+surface_forms(canonical: str) list~str~
+all_surfaces() list~str~
}
class VocabBridge {
+int dim
+int seed
+list~str~ canonicals
+list~str~ surfaces
+dict~str, Vector~ canonical_vecs
+dict~str, Vector~ token_vecs
+Vector bridge_vec
-Vector _cleanup_matrix
-Vector _cleanup_norms
+build(store: MemoryStore, store_path: str, seed: int) void
+rewrite(query: str, top_k: int, min_score: float) str
+noise_floor() float
+size() int
-_harvest(store: MemoryStore) _Vocabulary
-_cleanup_query(probe: Vector, top_k: int) list~tuple~str, float~~
}
class VocabBridgeCache {
+MemoryStore store
+int dim
+str store_path
+int seed
-VocabBridge _bridge
-bool _subscribed
+__post_init__() void
+get() VocabBridge
+invalidate() void
}
class MemoryStore {
+add_invalidation_callback(callback: callable) void
+iter_incoming_anchor_text() iterator
+list_belief_ids() list~str~
+get_belief(belief_id: str) Belief | None
}
class Belief {
+str content
+int lock_level
}
class Vector
class extract_entities {
<<function>>
}
class bm25_tokenize {
<<function>>
}
class LOCK_USER {
<<constant>>
}
VocabBridgeCache o-- VocabBridge : caches
VocabBridgeCache --> MemoryStore : uses
VocabBridge --> MemoryStore : builds_from
VocabBridge --> _Vocabulary : aggregates
VocabBridge --> Vector : uses
VocabBridge --> bm25_tokenize : tokenizes
VocabBridge --> extract_entities : extracts
VocabBridge --> LOCK_USER : checks
MemoryStore --> Belief : returns
Flow diagram for use_vocab_bridge flag resolution precedenceflowchart LR
A[Start flag resolution] --> B[Read env AELFRICE_VOCAB_BRIDGE]
B --> C{Env is truthy or falsy?}
C -- Yes --> D[Return env value]
C -- No --> E[Check explicit kwarg use_vocab_bridge]
E --> F{explicit is not None?}
F -- Yes --> G[Return explicit]
F -- No --> H[Read TOML retrieval.use_vocab_bridge]
H --> I{TOML value is not None?}
I -- Yes --> J[Return TOML value]
I -- No --> K[Return default False]
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Hey - I've found 3 issues, and left some high level feedback:
- In
retrieve_v2, constructing a newVocabBridgeCache(store)whenvocab_bridge_cacheisNonewill register a fresh invalidation callback on every call and leak callbacks on the store; consider either reusing a singleton cache per store or deferring subscription until the cache is intended to be long-lived. - In
VocabBridge._cleanup_query,normalized = self._cleanup_matrix / self._cleanup_norms[:, np.newaxis]is recomputed on every rewrite; you could precompute and store the row-normalized matrix at build time to avoid this per-query cost.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `retrieve_v2`, constructing a new `VocabBridgeCache(store)` when `vocab_bridge_cache` is `None` will register a fresh invalidation callback on every call and leak callbacks on the store; consider either reusing a singleton cache per store or deferring subscription until the cache is intended to be long-lived.
- In `VocabBridge._cleanup_query`, `normalized = self._cleanup_matrix / self._cleanup_norms[:, np.newaxis]` is recomputed on every rewrite; you could precompute and store the row-normalized matrix at build time to avoid this per-query cost.
## Individual Comments
### Comment 1
<location path="src/aelfrice/vocab_bridge.py" line_range="335-336" />
<code_context>
+ probe_norm = float(np.linalg.norm(probe))
+ if probe_norm == 0:
+ return []
+ normalized = self._cleanup_matrix / self._cleanup_norms[:, np.newaxis]
+ sims = (normalized @ (probe / probe_norm)).astype(np.float64)
+ n = len(self.canonicals)
+ k = min(top_k, n)
</code_context>
<issue_to_address>
**suggestion (performance):** Avoid recomputing the normalized cleanup matrix on every rewrite for better performance.
In `_cleanup_query`, `normalized` is recomputed on every call even though `_cleanup_matrix` and `_cleanup_norms` are fixed after `build()`, adding an extra O(N * dim) pass and allocation per rewrite. Consider precomputing and storing the normalized matrix (or normalizing rows in-place once) in `build()` so `_cleanup_query` only performs the matrix–vector multiply and top‑k selection.
Suggested implementation:
```python
if (
self._cleanup_matrix is None
or self._cleanup_norms is None
or getattr(self, "_normalized_cleanup_matrix", None) is None
):
return []
```
```python
probe_norm = float(np.linalg.norm(probe))
if probe_norm == 0:
return []
sims = (
self._normalized_cleanup_matrix @ (probe / probe_norm)
).astype(np.float64)
```
To fully implement the optimization, you should:
1. In the class's `build()` (or equivalent) method, after computing `self._cleanup_matrix` and `self._cleanup_norms`, precompute and store the normalized matrix, e.g.:
```python
# After _cleanup_matrix (shape: [N, dim]) and _cleanup_norms (shape: [N]) are finalized
norms = self._cleanup_norms
# Avoid division by zero; skip or mask zero-norm rows as appropriate
nonzero = norms != 0
normalized = np.zeros_like(self._cleanup_matrix, dtype=np.float64)
normalized[nonzero] = (
self._cleanup_matrix[nonzero] / norms[nonzero, np.newaxis]
)
self._normalized_cleanup_matrix = normalized
```
2. Ensure `self._normalized_cleanup_matrix` is initialized (e.g., to `None`) in `__init__` so the attribute exists before `build()` is called.
3. If there are other code paths that mutate `_cleanup_matrix` or `_cleanup_norms` after `build()`, make sure they also recompute `_normalized_cleanup_matrix` to keep them in sync.
</issue_to_address>
### Comment 2
<location path="src/aelfrice/vocab_bridge.py" line_range="293-300" />
<code_context>
+ # Canonical tokens already present in the raw query do not
+ # need rebridging — skip them so the rewriter is idempotent
+ # over its own output.
+ already_in_query: set[str] = set(bm25_tokenize(query))
+
+ for token in bm25_tokenize(query):
+ tv = self.token_vecs.get(token)
+ if tv is None:
</code_context>
<issue_to_address>
**suggestion (performance):** Reuse tokenization of the query instead of calling `bm25_tokenize` twice.
`bm25_tokenize(query)` is called twice: once for `already_in_query` and again in the loop. To avoid repeated work, compute `tokens = list(bm25_tokenize(query))` once, build `already_in_query = set(tokens)`, and then iterate over `tokens` in the loop.
```suggestion
appended: list[str] = []
appended_set: set[str] = set()
# Canonical tokens already present in the raw query do not
# need rebridging — skip them so the rewriter is idempotent
# over its own output.
tokens = list(bm25_tokenize(query))
already_in_query: set[str] = set(tokens)
for token in tokens:
```
</issue_to_address>
### Comment 3
<location path="docs/CONFIG.md" line_range="324" />
<code_context>
+### `use_vocab_bridge`
+
+Boolean, default `false`, opt-in (v2.1+, #433). Enables the HRR vocabulary-bridge query rewrite. When on, `retrieve_v2` builds (or fetches a cached) `VocabBridge` over the per-project store and prepends the rewrite stage before lane fan-out:
+
+```
</code_context>
<issue_to_address>
**suggestion (typo):** Clarify the phrase "fetches a cached" to avoid a dangling adjective.
"builds (or fetches a cached) `VocabBridge`" is slightly ungrammatical because "cached" lacks a noun. Consider rephrasing to "builds (or fetches a cached `VocabBridge`)" or "builds (or fetches a cached instance of `VocabBridge`)" for clarity.
```suggestion
Boolean, default `false`, opt-in (v2.1+, #433). Enables the HRR vocabulary-bridge query rewrite. When on, `retrieve_v2` builds (or fetches a cached instance of `VocabBridge`) over the per-project store and prepends the rewrite stage before lane fan-out:
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| normalized = self._cleanup_matrix / self._cleanup_norms[:, np.newaxis] | ||
| sims = (normalized @ (probe / probe_norm)).astype(np.float64) |
There was a problem hiding this comment.
suggestion (performance): Avoid recomputing the normalized cleanup matrix on every rewrite for better performance.
In _cleanup_query, normalized is recomputed on every call even though _cleanup_matrix and _cleanup_norms are fixed after build(), adding an extra O(N * dim) pass and allocation per rewrite. Consider precomputing and storing the normalized matrix (or normalizing rows in-place once) in build() so _cleanup_query only performs the matrix–vector multiply and top‑k selection.
Suggested implementation:
if (
self._cleanup_matrix is None
or self._cleanup_norms is None
or getattr(self, "_normalized_cleanup_matrix", None) is None
):
return [] probe_norm = float(np.linalg.norm(probe))
if probe_norm == 0:
return []
sims = (
self._normalized_cleanup_matrix @ (probe / probe_norm)
).astype(np.float64)To fully implement the optimization, you should:
- In the class's
build()(or equivalent) method, after computingself._cleanup_matrixandself._cleanup_norms, precompute and store the normalized matrix, e.g.:# After _cleanup_matrix (shape: [N, dim]) and _cleanup_norms (shape: [N]) are finalized norms = self._cleanup_norms # Avoid division by zero; skip or mask zero-norm rows as appropriate nonzero = norms != 0 normalized = np.zeros_like(self._cleanup_matrix, dtype=np.float64) normalized[nonzero] = ( self._cleanup_matrix[nonzero] / norms[nonzero, np.newaxis] ) self._normalized_cleanup_matrix = normalized
- Ensure
self._normalized_cleanup_matrixis initialized (e.g., toNone) in__init__so the attribute exists beforebuild()is called. - If there are other code paths that mutate
_cleanup_matrixor_cleanup_normsafterbuild(), make sure they also recompute_normalized_cleanup_matrixto keep them in sync.
| appended: list[str] = [] | ||
| appended_set: set[str] = set() | ||
| # Canonical tokens already present in the raw query do not | ||
| # need rebridging — skip them so the rewriter is idempotent | ||
| # over its own output. | ||
| already_in_query: set[str] = set(bm25_tokenize(query)) | ||
|
|
||
| for token in bm25_tokenize(query): |
There was a problem hiding this comment.
suggestion (performance): Reuse tokenization of the query instead of calling bm25_tokenize twice.
bm25_tokenize(query) is called twice: once for already_in_query and again in the loop. To avoid repeated work, compute tokens = list(bm25_tokenize(query)) once, build already_in_query = set(tokens), and then iterate over tokens in the loop.
| appended: list[str] = [] | |
| appended_set: set[str] = set() | |
| # Canonical tokens already present in the raw query do not | |
| # need rebridging — skip them so the rewriter is idempotent | |
| # over its own output. | |
| already_in_query: set[str] = set(bm25_tokenize(query)) | |
| for token in bm25_tokenize(query): | |
| appended: list[str] = [] | |
| appended_set: set[str] = set() | |
| # Canonical tokens already present in the raw query do not | |
| # need rebridging — skip them so the rewriter is idempotent | |
| # over its own output. | |
| tokens = list(bm25_tokenize(query)) | |
| already_in_query: set[str] = set(tokens) | |
| for token in tokens: |
|
|
||
| ### `use_vocab_bridge` | ||
|
|
||
| Boolean, default `false`, opt-in (v2.1+, #433). Enables the HRR vocabulary-bridge query rewrite. When on, `retrieve_v2` builds (or fetches a cached) `VocabBridge` over the per-project store and prepends the rewrite stage before lane fan-out: |
There was a problem hiding this comment.
suggestion (typo): Clarify the phrase "fetches a cached" to avoid a dangling adjective.
"builds (or fetches a cached) VocabBridge" is slightly ungrammatical because "cached" lacks a noun. Consider rephrasing to "builds (or fetches a cached VocabBridge)" or "builds (or fetches a cached instance of VocabBridge)" for clarity.
| Boolean, default `false`, opt-in (v2.1+, #433). Enables the HRR vocabulary-bridge query rewrite. When on, `retrieve_v2` builds (or fetches a cached) `VocabBridge` over the per-project store and prepends the rewrite stage before lane fan-out: | |
| Boolean, default `false`, opt-in (v2.1+, #433). Enables the HRR vocabulary-bridge query rewrite. When on, `retrieve_v2` builds (or fetches a cached instance of `VocabBridge`) over the per-project store and prepends the rewrite stage before lane fan-out: |
|
|
||
| import hashlib | ||
| from dataclasses import dataclass, field | ||
| from pathlib import Path |
|
[claim:review:Setr:2026-05-08T19:30:17Z] |
|
[claim:review:kulili:2026-05-08T19:31:33Z] |
|
[release:review:kulili:2026-05-08T19:31:38Z] |
|
[release:review:Setr:2026-05-08T19:32:32Z] |
Closes-progress on #433 (HRR vocabulary bridge). Lands the mechanically-correct, default-OFF form per the spec's bench-gate / ship-or-defer policy (
docs/feature-hrr-vocab-bridge.md§ Bench-gate).What landed
1.
src/aelfrice/vocab_bridge.py(commit 1)VocabBridgebuilds a single(dim,)HRR composite fromsum_{c, s in surface_forms(c)} bind(token_vec[s], canonical_vec[c])over the corpus's surface-form universe. Pure linear algebra — no learned components, no LLM call, no embedding model. Surface-form harvest combines:store.iter_incoming_anchor_text, [retrieval] Augmented BM25F (incoming-edge anchor text) + vectorized BM25 sparse matvec #148 substrate) — same field BM25F consumesbeliefs.entity_idcolumn does not yet exist on the v2.0.0 schema)Surface-form filter: single-token canonicals only (multi-word noun phrases are dropped — the bridge rewrites query tokens, not query spans), tokens ≥ 3 chars (one- and two-letter words bind generically and bloat the bridge with no recoverable signal).
rewrite()unbinds each query token from the composite, cleanup-queries against the canonical-vec set, and appends canonicals above the noise floor (1/sqrt(dim)). Original-query tokens are preserved verbatim — bridged candidates are appended, never substituted, so flag-OFF / unseen-token paths are byte-identical to the pre-bridge call.Determinism mirrors
HRRStructIndex: dual-Generator stream split (canonical_rng↔seed,token_rng↔seed ^ _TOKEN_SALT), path-derived MD5-based 64-bit seed.VocabBridgeCachemirrorsBM25IndexCache: subscribes to the store's invalidation hook, rebuilds on nextget()after any belief / edge mutation.2.
retrieve_v2()integration (commit 2)use_vocab_bridgewith the established 4-stage precedence (envAELFRICE_VOCAB_BRIDGE> kwarg > TOML[retrieval] use_vocab_bridge> default-OFF).vocab_bridge_cache: VocabBridgeCache | None = Nonefor amortised builds across long-running consumers.use_hrrbecomes a deprecated alias foruse_vocab_bridge(per spec §Configuration). Lab v2.0.0 adapters that passuse_hrr=Trueroute to the bridge automatically;use_vocab_bridge=Falsewins when both are passed.When the flag resolves True, the bridge runs before lane fan-out:
3. Tests (commits 3 + 4)
tests/test_vocab_bridge.py— 18 unit tests: build determinism, algebraic self-recovery cosine, token-universe filter, empty-store no-op, anchor-text harvest, distinct stream verification, etc.tests/test_vocab_bridge_integration.py— 14 integration tests: flag precedence, byte-identical default-OFF path, deprecateduse_hrralias semantics,VocabBridgeCacheinjection + invalidation.tests/bench_gate/test_vocab_bridge_uplift.py— A2 precondition. Skips on public CI; withAELFRICE_CORPUS_ROOTpopulated, asserts ≥50% row-level coverage ofexpected_canonicals(loose threshold; A2's strict NDCG@k is the lab-side follow-up).4.
docs/CONFIG.md(commit 5)use_vocab_bridgeadded to the[retrieval]knob list, the.aelfrice.tomlexample, and a dedicated Keys subsection with the rewrite-pipeline diagram, deprecated-alias note, and precedence chain.Acceptance status
tests/corpus/v2_0/vocab_bridge/row schema documented in the gate's module docstring.docs/RETRIEVAL_COMPOSITION.mddoes not exist yet (per [retrieval] Pipeline composition tracker — unified retrieve() with feature-flag gate #154; same status as [v2.0] Type-aware compression — tokens-per-belief reduction on retrieved output #434 PR-1's A5 deferral).docs/CONFIG.md.docs/COMMANDS.mdis CLI-focused and does not document Python kwargs.Test plan
uv run pytest tests/test_vocab_bridge.py tests/test_vocab_bridge_integration.py -q— 32 passeduv run pytest tests/bench_gate/test_vocab_bridge_uplift.py -q— skipped (public CI, corpus absent)uv run pytest tests/ -q --ignore=tests/bench_gate— 2870 passed, 24 skipped, no regressions vsmainAELFRICE_CORPUS_ROOT=~/projects/aelfrice-lab/tests/corpus/v2_0 uv run pytest tests/bench_gate/test_vocab_bridge_uplift.py— operator/lab-side; ratifies A2 preconditionFollow-up issues
vocab_bridgecorpus through fullretrieve_v2and compute recall@k uplift vs flag-OFF baseline. Threshold per spec §A2: strictly positive uplift.docs/RETRIEVAL_COMPOSITION.md([retrieval] Pipeline composition tracker — unified retrieve() with feature-flag gate #154 territory) — same path as [v2.0] Type-aware compression — tokens-per-belief reduction on retrieved output #434's A5 deferral.Refs #433, #154 (composition tracker), #148 (BM25F anchor substrate), #152 (HRR struct index — sibling lane).
Summary by Sourcery
Introduce an HRR-based query-side vocabulary bridge and integrate it into the retrieval pipeline behind a feature flag.
New Features:
Enhancements:
Documentation:
Tests: