feat(retrieval): adaptive expansion_gate token threshold (#760) - #799
Conversation
…coding + meta-belief env flag First slice of #760 consumer wiring. Pure additions — no caller changes yet, so expansion gate is byte-identical until the read-path commits land. Adds: - EXPANSION_GATE_TOKEN_THRESHOLD_FLOOR (20), EXPANSION_GATE_TOKEN_THRESHOLD_CEIL (320), META_EXPANSION_GATE_TOKEN_THRESHOLD_KEY, META_EXPANSION_GATE_TOKEN_THRESHOLD_STATIC_DEFAULT (0.5), META_EXPANSION_GATE_TOKEN_THRESHOLD_POSTERIOR_DECAY_SECONDS (30d). - ENV_META_BELIEF_EXPANSION_GATE_TOKEN_THRESHOLD = "AELFRICE_META_BELIEF_EXPANSION_GATE_TOKEN_THRESHOLD", ships default-OFF behind the #437 A/B bench-gate clause. - decode_expansion_gate_token_threshold(v) — log-linear interpolation [0, 1] → [20, 320] with int(round(...)). v=0.5 decodes to 80 exactly: the geometric mean of 20 and 320 is sqrt(6400) = 80, so cold-start byte-identical to BROAD_PROMPT_TOKEN_THRESHOLD. - is_meta_belief_expansion_gate_token_threshold_enabled() — truthy- token resolver mirroring is_meta_belief_bfs_depth_budget_enabled(), including the '=enabled' spelling. 23 new tests cover encoding boundaries (floor, ceil, mid=80), clamping for out-of-range inputs, int return, bounds on [0,100], monotonicity, env-flag truthy/falsy tokens, constants pin, and the cold-start byte-identical property.
…elper + meta-aware resolver Second slice of #760. Still no caller change in expansion_gate.py — the new resolve_expansion_gate_token_threshold_with_meta() function exists in parallel, ready for the third commit to wire in at the token-threshold resolution site. Adds: - install_expansion_gate_token_threshold_meta_belief(store, *, now_ts) — idempotent install of the #760 row with the v3.x ratified defaults: relevance signal only (expansion quality is measured by whether injected beliefs are referenced, not by latency), 30d posterior decay, static_default=0.5 which decodes to 80 exactly via decode_expansion_gate_token_threshold. - resolve_expansion_gate_token_threshold_with_meta(store, *, now_ts, explicit=None) — two-tier precedence: explicit kwarg (positive int) > meta-belief (gated on the env flag) > BROAD_PROMPT_TOKEN_THRESHOLD. No TOML or env-var override layer — the token threshold has never had a user-facing config knob. store=None collapses to the static default. - Updates get_active_meta_belief_consumers() to include META_EXPANSION_GATE_TOKEN_THRESHOLD_KEY when the env flag is on, so the #779 UPS sweeper delivers relevance evidence to it at the next turn. 12 new tests cover: install idempotency, signal-class pin (relevance only), cold-start value == BROAD_PROMPT_TOKEN_THRESHOLD, five resolver precedence paths, 100-strong-positive-evidence responsiveness, and two get_active_meta_belief_consumers inclusion/sorting checks.
… into expansion gate Closes the #760 loop in should_run_expansion. The hardcoded BROAD_PROMPT_TOKEN_THRESHOLD comparison at the long-prompt heuristic is now resolved through resolve_expansion_gate_token_threshold_with_meta when a store is supplied; falls back to the static 80 when no store or flag-off. Wiring: - should_run_expansion gains optional store + now_ts kwargs (both default None). When store is provided and the env flag is on, the threshold is read from the meta-belief and decoded; otherwise the hardcoded BROAD_PROMPT_TOKEN_THRESHOLD is used. TYPE_CHECKING import for MemoryStore avoids a runtime import cycle; the resolver is imported deferred inside the function body to mirror the existing deferred-import pattern throughout expansion_gate.py. - retrieve() and retrieve_with_tiers() in retrieval.py both pass store=store, now_ts=int(time.time()) into should_run_expansion, activating the adaptive layer for production callers. retrieve_v2() reaches the same path through retrieve_with_tiers. - BROAD_PROMPT_TOKEN_THRESHOLD is NOT deleted — it remains as the cold-start fallback, the static_default reference, and a named constant for contributors. The old hardcoded value is still the default when the feature flag is off. - Relevance signal update: the #779 UPS sweeper already handles this via get_active_meta_belief_consumers() (updated in Commit 2). No additional signal-update wiring is needed inside expansion_gate.py. Default-OFF byte-identical contract: with the env flag unset, resolve_expansion_gate_token_threshold_with_meta collapses to BROAD_PROMPT_TOKEN_THRESHOLD (80), identical to the pre-#760 path. 4 new integration tests cover: flag-off byte-identical, flag-on + meta-absent fallback, flag-on + cold-start value byte-identical, and flag-on + high-threshold lets medium-prompt through.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
robotrocketscience
added
the
attn:review
PR-size soft capThis PR is over the advisory size threshold:
Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the |
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
[claim:review:bagheera:2026-05-14T17:59:27Z] |
|
Review (bagheera, queue review):
Labeling |
robotrocketscience
added
ready-to-merge
|
[release:review:bagheera:2026-05-14T18:01:58Z] |
|
merge-train: blocked 2 review thread(s) are unresolved on these files: src/aelfrice/expansion_gate.py, src/aelfrice/retrieval.py. Resolve them on the PR (click 'Resolve conversation' on each) and re-add the label. The |
github-actions
Bot
removed
the
ready-to-merge
|
Resolved 2 CodeQL cyclic-import threads (retrieval ↔ expansion_gate). The cycle is pre-existing and mitigated by deferred function-body imports + |
robotrocketscience
added
the
ready-to-merge
github-actions
Bot
removed
the
ready-to-merge
|
merge-train: merged e52d93e → |
2 participants
Closes #760
Sub-task F of umbrella #480 (adaptive meta-belief layer). Wires
meta:retrieval.expansion_gate.token_thresholdinto the expansion-gate long-prompt heuristic. Reframed per the 2026-05-14T16:55Z operator decision (single-key reframe + wait for #789, both satisfied).What this PR adds
[20, 320]for the new meta-belief, mirroring the [v3.x] E: meta:retrieval.bfs_depth_budget — adaptive L2 depth cap (#480 sub-task) #759bfs_depth_budgetpattern. Geometric mean of 20 and 320 is exactly 80, sodecode(0.5) == BROAD_PROMPT_TOKEN_THRESHOLD == 80with zero floating-point error — cold-start is byte-identical to today's hardcoded path.META_EXPANSION_GATE_TOKEN_THRESHOLD_KEY = "meta:retrieval.expansion_gate.token_threshold",STATIC_DEFAULT = 0.5,POSTERIOR_DECAY_SECONDS = 30 * 24 * 3600(30d).decode_expansion_gate_token_threshold(v) -> int,is_meta_belief_expansion_gate_token_threshold_enabled() -> bool,install_expansion_gate_token_threshold_meta_belief(store, *, now_ts) -> None,resolve_expansion_gate_token_threshold_with_meta(store, *, now_ts, explicit=None) -> int.should_run_expansiongains optionalstore+now_tskwargs; bothretrieve()andretrieve_with_tiers()now pass these through, activating the adaptive layer for production callers when the env flag is on.get_active_meta_belief_consumers()updated to include the new key when the env flag is on — this closes the Live close-the-loop relevance-signal infrastructure — #756 / #480 prereq #779 relevance-signal loop for this consumer (UPS sweeper deliversreferenced ∈ {0, 1}evidence on the next turn).AELFRICE_META_BELIEF_EXPANSION_GATE_TOKEN_THRESHOLD(also accepts=enabled, mirroring the sibling pattern). Ships default-OFF per the [exploration] Adaptive half-life as a Bayesian-engine-governed meta-belief #480 bench-gate clause.What this PR does NOT change
BROAD_PROMPT_TOKEN_THRESHOLD = 80stays as the cold-start fallback and named reference constant._QUESTION_FORM_PREFIXES(literal-prefix list) and_ISSUE_REF_RE/_FILE_PATH_RE(regex patterns) stay non-adaptive — same shape as the [v3.x] C: meta:retrieval.bm25f_field_weights — adaptive per-field weights (#480 sub-task) #757 reframe explicitly excluded them from the adaptive surface.relevanceis wired (operator:relevanceonly).benchmarks/results/v2.0.0.jsonis canonical,uv sync && aelf bench all#437 A/B corpus shows ≥ +2pp recall-at-k with no >5% p95 latency regression.Atomic commits
16778a7c feat(retrieval): log-linear bounded expansion_gate token threshold encoding + meta-belief env flagbff9cd05 feat(retrieval): install_expansion_gate_token_threshold_meta_belief helper + meta-aware resolverae06caa4 feat(expansion_gate): wire meta-belief expansion_gate_token_threshold into expansion gatee52d93ee docs(changelog): note adaptive expansion_gate token threshold consumer (#760)All four commits SSH-signed (
%G? = G).Tests
tests/test_expansion_gate_token_threshold_meta.py(new, 39 tests): encoding boundaries + int return + bounds + monotonicity + clamping, env-flag tokens, constants pin, cold-start byte-identical property, install idempotency, signal-class pin torelevance, resolver precedence (five paths), 100-strong-positive-evidence responsiveness,get_active_meta_belief_consumersinclusion and sorting, and four integration tests (flag-off byte-identical, flag-on + meta-absent fallback, flag-on + cold-start byte-identical, flag-on + high-threshold lets medium-prompt through).4163 passed, 62 skipped, 75 xfailed in 81.9s.Acceptance check
BROAD_PROMPT_TOKEN_THRESHOLD) wired throughresolve_expansion_gate_token_threshold_with_metaat theshould_run_expansioncall site (verified atretrieval.pycall sites — bothretrieve()andretrieve_with_tiers()passstore, now_ts).get_active_meta_belief_consumers()(no separate write-path needed; the Live close-the-loop relevance-signal infrastructure — #756 / #480 prereq #779 UPS sweeper handles thereferencedsignal).[0.0, 1.0](decode floor/ceil are floor=20, ceil=320). Threshold cannot escape[20, 320].decode_*, monotone inv, deterministic decay.benchmarks/results/v2.0.0.jsonis canonical,uv sync && aelf bench all#437 A/B job, not in scope for this sub-task.Out of scope (per spec)
_QUESTION_FORM_PREFIXES,_ISSUE_REF_RE,_FILE_PATH_RE— not adaptive.