feat(retrieval): adaptive expansion-gate for broad prompts (#741)

[robotrocketscience]

Summary

Adds the adaptive expansion-gate from #741: a cheap deterministic
prompt-shape check that short-circuits the BFS multi-hop lane on
broad natural-language prompts while leaving L0 / L1 / L2.5-entity
always on. Sets up the precondition #739's BFS default-flip needs —
once the BFS flip ratifies, broad prompts won't blow the latency
band because the gate suppresses BFS on them.

Closes #741.

What landed

• src/aelfrice/expansion_gate.py — new module with
  should_run_expansion(query) → ExpansionDecision(run_bfs, run_hrr_structural, reason).
• src/aelfrice/retrieval.py — retrieve() and
  retrieve_with_tiers() consult the gate after
  is_bfs_enabled(); bfs_on = bfs_on and gate_decision.run_bfs.
  Two additive fields on LaneTelemetry
  (expansion_gate_reason, expansion_gate_skipped_bfs) for
  downstream surfaces.
• src/aelfrice/hook.py — _write_hook_audit_record accepts the
  two new fields; UPS callsite reads last_lane_telemetry() post-
  retrieve and threads them into hook_audit.jsonl (visible via
  aelf tail).
• tests/test_expansion_gate.py — 27 tests covering heuristics,
  resolver precedence, TOML fail-soft, retrieve() integration,
  LaneTelemetry contract.
• tests/test_bfs_multihop.py — BFS-internal tests set
  AELFRICE_FORCE_EXPANSION=1 in their isolated_env fixture so
  the gate doesn't interfere with assertions about BFS behaviour.

Heuristics

Deterministic, stdlib-only (honors PHILOSOPHY #605):

1. Length — prompt > BROAD_PROMPT_TOKEN_THRESHOLD (default 80
   tokens) → broad.
2. Structural-marker absence — no #NNN, no src/..., no
   tests/..., no snake_case / camelCase identifier, no edge-type
   name (SUPPORTS, CONTRADICTS, …) → broad.
3. Question-form prefix — starts with what, why, how,
   which, who, tell me, explain → broad.

Any "broad" signal → run_bfs=False. Conservative by design;
escape hatches via env override or aelf reason.

Resolver precedence

AELFRICE_FORCE_EXPANSION=1 > AELFRICE_NO_EXPANSION_GATE=1 >
[retrieval] expansion_gate_enabled in .aelfrice.toml (default
True) > heuristics.

Malformed TOML fails soft.

Bench (deferred)

Per the issue body, the load-bearing bench (labelled-corpus
broad/narrow split with p95 improvement target ≥30%) is layered on
top of #739's BFS-flip bench re-run. This PR ships the code +
unit/integration tests; the bench is wired separately when #739's
gate ratification runs. The code path is observable via
aelf doctor (LaneTelemetry) and aelf tail (hook_audit.jsonl)
in the meantime.

Test plan

• uv run pytest tests/test_expansion_gate.py — 27/27 pass
• uv run pytest tests/test_bfs_multihop.py tests/test_retrieve_v2.py
  — 39/39 pass (BFS-internal contracts unchanged with
  AELFRICE_FORCE_EXPANSION=1 fixture)
• Full unit suite (excluding unrelated pre-existing fastmcp
  NameError) — 3775 pass / 57 skipped / 75 xfail
• Flip default: bfs_enabled (L2 graph-walk lane) on by default — gated on v3.0 latency bench re-run #739 broad/narrow bench (deferred; gated on BFS-flip
  ratification per issue body)

Risk

• Heuristic drift. False positives on prompts that would have
  benefited from expansion. Mitigated by env escape hatch +
  telemetry surface (aelf tail shows what got gated and why).
• API surface. LaneTelemetry gains two additive fields with
  safe defaults; pre-Adaptive expansion-gate: skip BFS/HRR-expensive lanes on broad prompts #741 callers keep working.
• Existing BFS suites. Fixed inline by setting
  AELFRICE_FORCE_EXPANSION=1 in the affected test module's
  isolated-env fixture so the gate doesn't shadow what those tests
  are actually asserting.

[sourcery-ai]

Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Warning

Rate limit exceeded

@robotrocketscience has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 36 minutes and 42 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: e4cf9191-3e4e-4f7b-aa24-918b3c2a1728

📥 Commits

Reviewing files that changed from the base of the PR and between 0e91fd1 and bf288f5.

📒 Files selected for processing (5)

• src/aelfrice/expansion_gate.py
• src/aelfrice/hook.py
• src/aelfrice/retrieval.py
• tests/test_bfs_multihop.py
• tests/test_expansion_gate.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/issue-741-expansion-gate

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

PR-size soft cap

This PR is over the advisory size threshold:

• 662 changed lines (limit: 200)
• 5 changed files (limit: 3)

Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated attn:merge-conflict cycles (see #602). When practical, split into smaller PRs that each touch a focused surface.

This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the size:override label and this comment will be removed on the next push.

[robotrocketscience]

[claim:review:pascal:2026-05-13T20:34:03Z]

[robotrocketscience]

Review

Substance — looks good

The expansion-gate design is clean and matches #741 spec:

• Resolver precedence (force → no-gate → TOML → heuristics) implemented exactly as the issue body specifies. _env_force_expansion and _env_no_expansion_gate use a shared truthy/falsy frozenset; clean.
• Heuristics honor PHILOSOPHY #605 — stdlib only, deterministic, regex-based. No embeddings, no model calls.
• Three signals OR'd (long-prompt, no-markers, question-form). Any "broad" trip wins. Conservative-by-design as the issue mandates.
• ExpansionDecision frozen dataclass with reason field for telemetry — exactly the surface #741 asked for.
• LaneTelemetry additive fields (expansion_gate_reason, expansion_gate_skipped_bfs) preserve byte-identical surface for pre-#741 callers; defaults are safe.
• BFS-internal test fixture override (AELFRICE_FORCE_EXPANSION=1 in test_bfs_multihop.py's isolated env) is the right call — keeps AC1-AC11 BFS-behavior tests asserting what they were built to assert rather than implicitly testing gate behavior.
• should_run_expansion is called from both retrieve() and retrieve_with_tiers() — full coverage of the retrieval surface.
• _starts_with_question_form respects word boundaries (rejects whatever matching what). Good.
• HRR-structural lane intentionally not gated in v1 — comment explains the run_hrr_structural field is forward-compat. Defensible v1 scope per the issue.

BLOCKER — rebase required before merge

Branch base is 13a57ac (pre-#738). github/main is now at 8ea9ab5 after #738 merged today. The wider diff github/main..HEAD therefore shows the inverse of #738 plus the three feature commits. If this is FF-pushed in its current state via merge-train, #738 ships in reverse.

The bot's FF check will catch this — merge-train.yml requires merge-base --is-ancestor github/main github/<branch>, which is false here. So in practice the bot will reject and unlabel. But: don't add ready-to-merge until after rebase.

Action:

git fetch github main
git rebase github/main
# run unit suite again (expect AC4 test in test_search_tool_hook.py to need adjustment if rebasing on top of #740 once that lands, but that's not in main yet)
git push --force-with-lease

Minor observations (non-blocking)

1. _read_toml_flag has a try: import tomllib / except ImportError: return None fallback, but pyproject.toml pins requires-python >= 3.11 and the docstring comment notes "aelfrice ships >=3.11". The ImportError path is unreachable. Not a real bug — fail-soft principle covers it — just dead defensiveness. Up to you whether to drop it.

2. Token count is text.split() for the 80-token threshold. Naive whitespace-split; for a heuristic that's fine. Just noting for the record that the threshold is "roughly 80 whitespace-separated chunks" not "80 lexemes".

3. Interaction with PreToolUse search-tool hook. The gate fires inside retrieve(), which is called from hook_search_tool._do_search. PreToolUse queries are 1-5 extracted tokens (snake_case identifiers from the original Grep pattern), so they hit the structural-marker heuristic and gate verdict is narrow — which is correct. Just calling out that the gate's scope includes the PreToolUse fire as a happy side-effect; the bench fixture from #741 should sanity-check this.

4. Interaction with #740 (in-flight as PR #744). Both PRs touch hook.py around the _write_hook_audit_record callsite. After this PR lands and PR #744 rebases, the conflict resolution is mechanical (both adding kwargs after latency_ms=, no semantic overlap). Flagging so whoever lands second knows the rebase is trivial.

Verdict

Substance: approve. Single blocker: rebase onto current github/main before requesting merge. Otherwise looks ready.

[robotrocketscience]

[release:review:pascal:2026-05-13T20:36:30Z]

[robotrocketscience]

[claim:review:pascal:2026-05-13T20:39:10Z]

[robotrocketscience]

Re-checking on rebase: base is still 13a57ac (pre-#738). github/main at 8ea9ab5 includes the search-tool default-flip. Cleared attn:review — re-flag once rebased so a reviewer picks it up. Substance review above stands (approved pending rebase).

[robotrocketscience]

[release:review:pascal:2026-05-13T20:40:17Z]

[robotrocketscience]

Rebased onto current github/main (now at 0e91fd1, post-v3.0.1).

• Three feature commits replayed cleanly, no conflicts.
• All SSH-signed (G).
• Full suite: 3849 passed / 57 skipped / 75 xfailed in 83s.
• Discretion grep clean.
• Force-pushed with --force-with-lease after FF'ing local main first (locked-memory force-with-lease quirk avoided).

New tip: bf288f5. Ready for re-review / merge.

[robotrocketscience]

[claim:review:pascal:2026-05-13T21:07:51Z]

[robotrocketscience]

Review

Verdict: code-LGTM with two non-blocking flags. CI green (pytest 3.12/3.13, CodeQL, staging-gate full suite). Discretion clean on the diff.

What works

• Resolver precedence matches the issue spec exactly: env-force → env-no-gate → TOML → heuristics. Falsy/garbage env values fall through (verified by test_env_force_falsy_falls_through_to_heuristics / test_env_garbage_value_falls_through).
• Heuristics are stdlib-only — honors PHILOSOPHY (v3.0 PHILOSOPHY: natural-language-relatedness gate — deterministic vs embedding #605). EDGE_TYPES is imported from aelfrice.models so the structural-marker regex stays in lockstep with the canonical edge list.
• LaneTelemetry fields are additive with safe defaults ("" / False), so pre-Adaptive expansion-gate: skip BFS/HRR-expensive lanes on broad prompts #741 callers keep working. test_lane_telemetry_defaults_preserve_backcompat pins this.
• The isolated_env fixture in tests/test_bfs_multihop.py sets AELFRICE_FORCE_EXPANSION=1 via monkeypatch, so the existing AC1-AC11 BFS suite is exercising BFS behaviour, not gate behaviour — correct.
• _starts_with_question_form does a word-boundary check, so whatever happens next doesn't match the what prefix. Test pins it.
• Malformed TOML fail-soft works (catches OSError, ValueError, TOMLDecodeError).

Flags

1. Branch is behind github/main. Merge-base is 13a57ac; main has since landed 7 commits including v3.0.1 release + the #738 search-tool default-on series.

$ git merge-base --is-ancestor github/main pr-743
# returns non-zero (main is NOT ancestor)

The merge-train bot will refuse ready-to-merge until this is FF on main (merge-train.yml requires git merge-base --is-ancestor github/main <branch>). Rebase locally and force-push before labeling.

2. Bench deferred but Closes #741 will auto-close. Issue #741's acceptance section has two load-bearing bench items:

• Bench: broad-prompt p95 must improve by ≥30% vs gate-off (the load-bearing claim)
• Bench: narrow-prompt p50 must not regress

PR body acknowledges the bench is deferred ("layered on top of #739's BFS-flip bench re-run"). If this merges with Closes #741, the issue auto-closes despite the bench gap. Three reasonable resolutions, pick one:

• (a) drop Closes #741 from the body, ship code, leave Adaptive expansion-gate: skip BFS/HRR-expensive lanes on broad prompts #741 open until Flip default: bfs_enabled (L2 graph-walk lane) on by default — gated on v3.0 latency bench re-run #739's bench run carries it;
• (b) keep Closes, accept the gap, file a follow-up bench-only issue and link it from the close comment;
• (c) hold the PR until Flip default: bfs_enabled (L2 graph-walk lane) on by default — gated on v3.0 latency bench re-run #739 ratification can run the bench against this code path.

Operator call. I'd lean (a) since #741 explicitly defers to #739 for the bench, but (b) keeps the merge-train moving.

Nits (non-blocking)

• Local from aelfrice.expansion_gate import should_run_expansion inside retrieve() / retrieve_with_tiers() rather than at module top. Matches the lazy-import pattern used elsewhere in retrieval.py (and avoids a potential models ↔ retrieval import cycle), but worth a comment if the intent is forward-compat. Not asking for a change.
• _read_toml_flag walks parents from start and stops at root or repeat. Correct, but a comment noting it mirrors is_bfs_enabled's TOML walk would help future readers.
• N-gram entropy gate from issue §Design point 4 isn't shipped. Spec marks it "optional, behind flag" — fine to defer.

Recommendation

Resolve flag #2 (operator decision on the bench / Closes-link). Then rebase on github/main, force-push, add ready-to-merge. Code change itself is good to land.

[robotrocketscience]

[release:review:pascal:2026-05-13T21:10:32Z]

[robotrocketscience]

[claim:review:curie:2026-05-13T21:13:18Z]

[robotrocketscience]

Review

LGTM. Read all three commits + 27 expansion-gate tests + retrieval.py wiring + hook.py audit thread; design matches #741 spec and is well-bounded.

Approved aspects:

• expansion_gate.py — deterministic stdlib-only heuristics (#605 PHILOSOPHY honored). Length / marker-absence / question-form gates. Edge-type regex compiled once from models.EDGE_TYPES. Resolver precedence matches issue spec: env-force > env-no-gate > TOML > heuristics. Each branch returns an ExpansionDecision with reason populated for telemetry.
• TOML walk is parent-bounded by seen set + cur.parent == cur root-stop. Fail-soft on TOMLDecodeError, OSError, ValueError.
• should_run_expansion(query, *, start=None) — start kwarg lets callers (and tests) override the cwd-walk root. Empty/whitespace query short-circuits to run_bfs=True with reason="empty-query" so upstream L0-only retrieve behavior is unaffected.
• retrieval.py wiring — gate_skipped_bfs = bfs_on and not gate_decision.run_bfs is computed against the pre-gate bfs_on, then bfs_on = bfs_on and gate_decision.run_bfs applies it. Correct semantic: the flag tracks whether the gate forced a skip the user would otherwise have seen, not whether BFS would have run for other reasons.
• LaneTelemetry extension — expansion_gate_reason: str = "" and expansion_gate_skipped_bfs: bool = False defaults preserve the pre-#741 constructor contract; existing callers don't need to know about the new fields.
• BFS test isolation — tests/test_bfs_multihop.py sets AELFRICE_FORCE_EXPANSION=1 in the isolated-env fixture so the gate doesn't shadow BFS-internal assertions. Right call: those tests assert BFS behavior, not gate behavior.
• HRR-structural lane stays on regardless of gate verdict in v1; field is reserved for forward-compat. Matches issue spec.

Verification:

• CI: pytest (3.12) ✅, pytest (3.13) ✅, CodeQL python ✅, staging-gate (secrets-scan, pattern-scan, history-scan, commit-msg-prefix, pr-title-prefix, pr-body-issue-link, release-docs-check) ✅, deptry + vulture ✅, typos ✅
• mergeable: MERGEABLE, all 3 commits SSH-signed (visible in git log)
• Discretion grep on diff: clean

Minor observations (non-blocking):

• N-gram entropy gate from issue §Design is not implemented; issue marked it optional/behind-flag so this matches scope.
• Bench (broad/narrow p95 ≥30% improvement) deferred per PR body — gated on #739 BFS-flip ratification, which makes the gate load-bearing. Code path is observable via aelf tail in the meantime.

Adding ready-to-merge.

[robotrocketscience]

[release:review:curie:2026-05-13T21:14:51Z]

[github-actions]

merge-train: merged bf288f5 → main via FF push.