feat(retrieval): third meta-belief consumer — adaptive bfs_depth_budget (#759)

CHANGELOG.md

@@ -14,6 +14,8 @@ installable release; see the roadmap in [README.md](README.md).
 
 - **v3.1 close of #592 hot-start AC — sharpened `JUDGE_PROMPT_TEMPLATE` + widened hot_start fixture to n=10 eval turns** ([#778](https://github.com/robotrocketscience/aelfrice/issues/778)). Run 2 of the multi-judge κ sweep (2026-05-12/13, see [#687](https://github.com/robotrocketscience/aelfrice/issues/687)) saw `inter_judge_kappa.min = 0.0` (gate 0.7) and `hot_start_fidelity_mean = 0.667` (gate 0.8), diagnosed as two compounding causes: (a) the prompt template's "load-bearing claim" clause was operationally under-specified — judges disagreed on which part of each reference counted as load-bearing on 2 of 3 fixture turns; (b) statistical thinness on n=3 binary samples, where a single judge flip drops the κ catastrophically. (a) is addressed by replacing the single-clause definition in `benchmarks/context-rebuilder/judges/llm_judge.py` with a three-step rubric that names the unit of scoring as "specific facts" (concrete atoms — names, paths, values, actions, numeric quantities), requires the candidate to convey every such fact, and defines `matched=true` as "a reader of the candidate alone could state each specific fact"; plus two worked examples (multi-fact and low-content) drawn from outside the hot_start fixture to avoid biasing scoring on the corpus being scored. (b) is addressed by extending `hot_start_debugging_session_001.{jsonl,meta.json}` from 14 to 28 turns, keeping the same fork (turn 8) and the same synthetic CRLF-vs-LF `_dedup_key` debugging scenario, and adding 7 new user/assistant pairs that probe pre-fork facts the prior cut did not exercise (intermittency root cause, literal code change, why-fix-in-`_dedup_key`-not-conftest rationale, dedup tuple shape, NFC normalization form, repro command, conftest file:line). New `eval_turns = [9, 11, 13, 15, 17, 19, 21, 23, 25, 27]`. Test assertion in `tests/test_context_rebuilder_eval_harness_wiring.py` updated to the new tuple. Fixture content remains fully synthetic — no `_dedup_key` or `test_ingest_jsonl_idempotent` exists in source on `github/main`. The κ sweep re-run is operator-dispatched (subagent path per locked bench convention, not direct API) and gates closing the v3.0-deferred hot-start AC ≥80% bullet.
 
+- **Fifth meta-belief consumer: adaptive `bfs_depth_budget`** ([#759](https://github.com/robotrocketscience/aelfrice/issues/759), umbrella [#480](https://github.com/robotrocketscience/aelfrice/issues/480)). Wires `meta:retrieval.bfs_depth_budget` into `retrieve_v2`'s `bfs_max_depth` resolution. Per the 2026-05-13 operator ratification on #756 (C-F template): log-linear bounded `[1, 6]` encoding lives in the consumer with a rounding-to-int step (new wrinkle vs #756/#757 — `expand_bfs` takes `int max_depth`); MVP ships with the `latency` signal only (`bfs_depth` close-the-loop signal deferred to [#779](https://github.com/robotrocketscience/aelfrice/issues/779) per the same D4-deferral pattern #756 used for `relevance`). Rationale for latency-only MVP: `bfs_depth` requires detecting which surfaced beliefs the consuming agent referenced and at what BFS hop they originated — infrastructure not yet available. `latency` is live today and is load-bearing for safety because BFS depth dominates p95 retrieval latency. New module-level constants: `META_BFS_DEPTH_BUDGET_KEY`, `BFS_DEPTH_BUDGET_FLOOR` (1), `BFS_DEPTH_BUDGET_CEIL` (6), `META_BFS_DEPTH_BUDGET_STATIC_DEFAULT` (0.5 → decodes to 2 = round(sqrt(6))), `META_BFS_DEPTH_BUDGET_POSTERIOR_DECAY_SECONDS` (30d). New API: `decode_bfs_depth_budget(v)`, `is_meta_belief_bfs_depth_budget_enabled()`, `install_bfs_depth_budget_meta_belief(store, *, now_ts)`, `resolve_bfs_depth_budget_with_meta(store, *, now_ts, explicit=None)`. Resolver precedence: explicit positive-int kwarg → meta-belief → `BFS_DEFAULT_MAX_DEPTH`; no env-var or TOML layer because `bfs_max_depth` has never had a user-facing config knob. `retrieve_v2` wires the resolver between `effective_now_ts` calculation and the `retrieve_with_tiers` call; the explicit clause passes `None` when the caller passes the default so the meta-belief layer is not inadvertently bypassed. A second independent latency-signal update fires after `retrieve_with_tiers` when the depth-budget flag is on; both the #756 half-life and #759 depth-budget updates run on every call when their respective flags are on. `BFS_DEFAULT_MAX_DEPTH` verified as 2 (not 3 as the spec estimated): `decode_bfs_depth_budget(0.5) = 2 = BFS_DEFAULT_MAX_DEPTH`, so cold-start byte-identity holds. Ships default-OFF behind `AELFRICE_META_BELIEF_BFS_DEPTH_BUDGET` (also accepts `=enabled`); flips on after the #437 corpus A/B bench gate clears. 27 new tests cover log-linear encoding boundaries + int return + bounds + monotonicity + rounding-band transition, env-flag tokens, constants pin, install idempotency + signal-class assertion, five resolver precedence paths, 100-strong-positive-evidence responsiveness, default-OFF byte-identical `retrieve_v2`, and determinism (same evidence sequence + same now_ts → same resolved depth on two fresh stores).
+
 - **Second meta-belief consumer: adaptive `bm25f_anchor_weight`** ([#757](https://github.com/robotrocketscience/aelfrice/issues/757), umbrella [#480](https://github.com/robotrocketscience/aelfrice/issues/480)). Wires `meta:retrieval.bm25f_anchor_weight` into `_l1_hits`'s BM25F branch. Pattern reuse of [#756](https://github.com/robotrocketscience/aelfrice/issues/756): log-linear bounded `[1, 10]` encoding lives in the consumer, MVP ships with the `bm25_l0_ratio` signal only (relevance deferred to [#779](https://github.com/robotrocketscience/aelfrice/issues/779)). Scope-down note: the issue body asked for per-field BM25F weights, but aelfrice's BM25F has one tunable — `anchor_weight`, the weight on the incoming-anchor token stream from [#148](https://github.com/robotrocketscience/aelfrice/issues/148) R3 — so the scope was ratified down to that single integer before any code was written. Cold-start parity: `static_default = 0.5` decodes to anchor_weight = 3, matching `bm25.DEFAULT_ANCHOR_WEIGHT` exactly so installing the meta-belief preserves byte-identical retrieval order. New module-level constants: `META_BM25F_ANCHOR_WEIGHT_KEY`, `BM25F_ANCHOR_WEIGHT_FLOOR` (1), `BM25F_ANCHOR_WEIGHT_CEIL` (10), `META_BM25F_ANCHOR_WEIGHT_STATIC_DEFAULT` (0.5), `META_BM25F_ANCHOR_WEIGHT_POSTERIOR_DECAY_SECONDS` (30d). New API: `decode_meta_bm25f_anchor_weight(v)`, `is_meta_belief_bm25f_anchor_weight_enabled()`, `install_bm25f_anchor_weight_meta_belief(store, *, now_ts)`, `resolve_bm25f_anchor_weight_with_meta(store, *, now_ts, explicit=None)`. Resolver precedence is explicit kwarg → meta-belief → `DEFAULT_ANCHOR_WEIGHT`; no env-var or TOML layer because `anchor_weight` has never had a user-facing config knob. `_l1_hits` honours a caller-supplied `bm25f_cache` unconditionally (bench harness pin contract preserved); only the inline `BM25IndexCache(store)` construction is rerouted through the resolver. On every BM25F-branch query when the meta env flag is on, the `bm25_l0_ratio` sub-posterior is updated with `evidence = locked_in_bm25f_topk / total_locked` inside a try/except — store errors print to stderr and are swallowed, mirroring the #756 latency-signal posture. Empty L0 in the store skips the update (the L0-coverage metric is undefined). Ships default-OFF behind `AELFRICE_META_BELIEF_BM25F_ANCHOR_WEIGHT` (also accepts `=enabled`); flips on after the #437 corpus A/B bench gate clears. 30 new tests cover the log-linear encoding boundaries (including int-valued return + monotonic-non-decreasing), env-flag tokens, install idempotency, five resolver precedence paths, default-OFF byte-identical `_l1_hits`, end-to-end bm25_l0_ratio evidence persistence, flag-on-no-install no-crash, and the explicit-cache bypass contract.
 
 - **First meta-belief consumer: adaptive `temporal_half_life_seconds`** ([#756](https://github.com/robotrocketscience/aelfrice/issues/756), umbrella [#480](https://github.com/robotrocketscience/aelfrice/issues/480)). Wires `meta:retrieval.temporal_half_life_seconds` into `retrieve_v2`'s `temporal_sort` re-rank. Per the 2026-05-13 operator ratification: log-linear bounded `[3d, 14d]` encoding lives in the consumer (substrate stays pattern-uniform across B–F); MVP ships with `latency` signal only (relevance signal deferred to [#779](https://github.com/robotrocketscience/aelfrice/issues/779)). New module-level constants: `META_HALF_LIFE_KEY`, `HALF_LIFE_FLOOR_SECONDS` (3d), `HALF_LIFE_CEIL_SECONDS` (14d), `META_HALF_LIFE_STATIC_DEFAULT` (0.5 → ~6.5d cold start), `META_HALF_LIFE_POSTERIOR_DECAY_SECONDS` (30d), `LATENCY_TARGET_SECONDS` (0.080s per #437 bench floor). New API: `decode_meta_half_life(v)`, `is_meta_belief_half_life_enabled()`, `install_temporal_half_life_meta_belief(store, *, now_ts)`, `resolve_temporal_half_life_with_meta(store, *, now_ts, explicit=None, start=None)`. Resolver precedence is env → kwarg → TOML → meta-belief → 7d static, so operator overrides still bypass the adaptive layer. `retrieve_v2` gains a `now_ts: int | None = None` kwarg (defaults to `int(time.time())` so production callers don't need to pass it). On every `temporal_sort` invocation when the meta env flag is on, the latency-signal sub-posterior is updated with `evidence = clip(LATENCY_TARGET_SECONDS / observed, 0, 1)` inside a try/except — store errors print to stderr and are swallowed, mirroring the deferred-feedback enqueue's fail-soft posture. Ships default-OFF behind `AELFRICE_META_BELIEF_HALF_LIFE` (also accepts the `=enabled` spelling from the issue body); flips on after the #437 corpus A/B bench gate clears. 38 new tests cover the log-linear encoding boundaries, env-flag tokens, install idempotency, six resolver precedence paths, default-OFF byte-identical retrieve_v2, end-to-end latency-evidence persistence, stability (100 strong-positive events stay monotonic-non-decreasing and bounded by HALF_LIFE_CEIL), and determinism (same evidence sequence + same now_ts ⇒ byte-identical resolved half-life).

src/aelfrice/retrieval.py

@@ -277,6 +277,32 @@
 ENV_META_BELIEF_BM25F_ANCHOR_WEIGHT: Final[str] = (
     "AELFRICE_META_BELIEF_BM25F_ANCHOR_WEIGHT"
 )
+# ---------------------------------------------------------------------------
+# #759 BFS depth-budget meta-belief consumer (sub-task E of umbrella #480)
+# ---------------------------------------------------------------------------
+# `expand_bfs` takes `int max_depth`. Bounds `[1, 6]` honour the
+# single-hop floor (a depth-0 expansion is a no-op) and the latency
+# safety ceiling documented in `docs/bfs_multihop.md`. BFS depth
+# dominates p95 retrieval latency, so this is load-bearing for safety.
+META_BFS_DEPTH_BUDGET_KEY: Final[str] = "meta:retrieval.bfs_depth_budget"
+BFS_DEPTH_BUDGET_FLOOR: Final[int] = 1
+BFS_DEPTH_BUDGET_CEIL: Final[int] = 6
+# Static-default `value` for the meta-belief. Mid-range under the
+# log-linear bounds: `decode_bfs_depth_budget(0.5)` ≈ sqrt(1*6) ≈ 2.45
+# → rounds to 2. This is intentionally one hop below
+# `BFS_DEFAULT_MAX_DEPTH` (2): shallow-only finds → posterior pulls
+# budget down, consistent with the #759 spec rationale.
+META_BFS_DEPTH_BUDGET_STATIC_DEFAULT: Final[float] = 0.5
+# Sub-posterior decay — 30d, matching #756 and #757 so all #480 sub-
+# tasks are comparable and a single one-off spike doesn't shift the
+# surfaced depth.
+META_BFS_DEPTH_BUDGET_POSTERIOR_DECAY_SECONDS: Final[int] = 30 * 24 * 3600
+# Default-OFF feature flag. Ships behind the #437 A/B bench-gate
+# clause, same as #756 and #757. Setting this env var truthy switches
+# `resolve_bfs_depth_budget_with_meta` to read the meta-belief first.
+ENV_META_BELIEF_BFS_DEPTH_BUDGET: Final[str] = (
+    "AELFRICE_META_BELIEF_BFS_DEPTH_BUDGET"
+)
 # Number of decimal places used to round `posterior_weight` before
 # inclusion in the cache key. Two callers passing weights that
 # differ by less than this granularity collapse to the same key.
@@ -809,6 +835,50 @@ def is_meta_belief_bm25f_anchor_weight_enabled() -> bool:
     return norm in _ENV_TRUTHY or norm == "enabled"
 
 
+def decode_bfs_depth_budget(value: float) -> int:
+    """Decode a `[0, 1]` meta-belief value into an integer BFS max-depth
+    via log-linear interpolation between
+    :data:`BFS_DEPTH_BUDGET_FLOOR` (1) and
+    :data:`BFS_DEPTH_BUDGET_CEIL` (6).
+
+    `v=0.0` → 1, `v=1.0` → 6, `v=0.5` → 2 (rounded from ~2.45,
+    one hop below :data:`BFS_DEFAULT_MAX_DEPTH` so a cold-start
+    install with the meta-belief on gently trims the budget until
+    evidence accrues).
+
+    Values outside `[0, 1]` are clamped — the substrate's
+    `posterior_mean` is mathematically bounded to `[0, 1]` but
+    ``value`` may be the static_default fallback on a misconfigured
+    row. The result is rounded to the nearest int because
+    ``expand_bfs`` accepts only ``int max_depth``.
+
+    Per the 2026-05-13 #756 ratification: encoding lives in the
+    consumer, substrate stays pattern-uniform across #480 B–F.
+    """
+    v = max(0.0, min(1.0, value))
+    ln_floor = math.log(BFS_DEPTH_BUDGET_FLOOR)
+    ln_ceil = math.log(BFS_DEPTH_BUDGET_CEIL)
+    return int(round(math.exp(ln_floor + v * (ln_ceil - ln_floor))))
+
+
+def is_meta_belief_bfs_depth_budget_enabled() -> bool:
+    """Return True iff :data:`ENV_META_BELIEF_BFS_DEPTH_BUDGET` is set
+    to a recognised truthy value.
+
+    Ships default-OFF per the #759 bench-gate clause: until #437 A/B
+    corpus evidence clears, BFS still uses :data:`BFS_DEFAULT_MAX_DEPTH`
+    from ``bfs_multihop``. Operators flip this on per-shell to opt into
+    the adaptive depth budget. The ``=enabled`` spelling is honoured
+    alongside the codebase-standard truthy tokens (``1``, ``true``,
+    ``yes``, ``on``), mirroring :func:`is_meta_belief_half_life_enabled`.
+    """
+    raw = os.environ.get(ENV_META_BELIEF_BFS_DEPTH_BUDGET)
+    if raw is None:
+        return False
+    norm = raw.strip().lower()
+    return norm in _ENV_TRUTHY or norm == "enabled"
+
+
 def resolve_temporal_half_life(
     explicit: float | None = None,
     *,
@@ -977,6 +1047,78 @@ def resolve_bm25f_anchor_weight_with_meta(
     return DEFAULT_ANCHOR_WEIGHT
 
 
+def install_bfs_depth_budget_meta_belief(
+    store: MemoryStore,
+    *,
+    now_ts: int,
+) -> bool:
+    """Idempotent install of the #759 meta-belief on ``store``.
+
+    Returns True on first install, False if the row already exists.
+    Mirrors :func:`install_temporal_half_life_meta_belief`'s contract —
+    existing rows are not overwritten because the surfaced depth budget
+    would silently shift under BFS.
+
+    The install signature pins the v3.x ratified defaults: latency
+    signal only (`bfs_depth` signal deferred to #779 per the same D4
+    split that #756 and #757 followed), 30d posterior decay, cold-start
+    ``value`` = 0.5 which decodes to 2 via :func:`decode_bfs_depth_budget`.
+    """
+    from aelfrice.meta_beliefs import SIGNAL_LATENCY
+    return store.install_meta_belief(
+        META_BFS_DEPTH_BUDGET_KEY,
+        static_default=META_BFS_DEPTH_BUDGET_STATIC_DEFAULT,
+        half_life_seconds=META_BFS_DEPTH_BUDGET_POSTERIOR_DECAY_SECONDS,
+        signal_weights={SIGNAL_LATENCY: 1.0},
+        now_ts=now_ts,
+    )
+
+
+def resolve_bfs_depth_budget_with_meta(
+    store: MemoryStore | None,
+    *,
+    now_ts: int,
+    explicit: int | None = None,
+    start: None = None,  # reserved for future TOML layer; unused by #759 MVP
+) -> int:
+    """Resolve the BFS max-depth knob with meta-belief consultation.
+
+    Returns an ``int`` because ``expand_bfs`` takes ``int max_depth``.
+
+    Precedence (first decisive wins):
+      1. Explicit ``explicit`` kwarg from the caller (positive int) —
+         for the bench harness override path.
+      2. **Meta-belief** (#759) — only when
+         :data:`ENV_META_BELIEF_BFS_DEPTH_BUDGET` resolves truthy AND
+         ``store`` has the meta-belief installed. Decodes the substrate's
+         `[0, 1]` value through :func:`decode_bfs_depth_budget` into
+         the `[1, 6]` band, rounded to int.
+      3. Default: :data:`BFS_DEFAULT_MAX_DEPTH` from
+         ``aelfrice.bfs_multihop``.
+
+    No env-var or TOML override layer — bfs_max_depth has never had a
+    user-facing config knob, so we do not synthesize one. Operators
+    override via explicit kwarg (bench) or meta-belief (production).
+    ``None`` ``store`` collapses to the static default.
+
+    The ``explicit`` clause is so a caller explicitly passing the
+    :data:`BFS_DEFAULT_MAX_DEPTH` default doesn't disable the
+    meta-belief; only a non-default explicit override bypasses the
+    adaptive layer. Callers that do not override should pass
+    ``explicit=bfs_max_depth if bfs_max_depth != BFS_DEFAULT_MAX_DEPTH
+    else None`` so the default falls through to the meta-belief.
+    """
+    if explicit is not None and explicit > 0:
+        return int(explicit)
+    if store is not None and is_meta_belief_bfs_depth_budget_enabled():
+        meta_value = store.read_meta_belief_value(
+            META_BFS_DEPTH_BUDGET_KEY, now_ts=now_ts,
+        )
+        if meta_value is not None:
+            return decode_bfs_depth_budget(meta_value)
+    return BFS_DEFAULT_MAX_DEPTH
+
+
 def _belief_age_seconds(b: Belief, now: datetime) -> float:
     """Seconds between `now` and `b.created_at` (clamped at 0).
 
@@ -2267,6 +2409,19 @@ def retrieve_v2(
 
     effective_now_ts = now_ts if now_ts is not None else int(time.time())
 
+    # #759 BFS depth-budget resolver. Fires before retrieve_with_tiers so
+    # the effective max_depth is locked in before the BFS expansion runs.
+    # The explicit clause: only pass the caller's bfs_max_depth when it
+    # deviates from the default — a caller that passes the default
+    # deliberately shouldn't suppress the meta-belief layer.
+    bfs_max_depth = resolve_bfs_depth_budget_with_meta(
+        store,
+        now_ts=effective_now_ts,
+        explicit=(
+            bfs_max_depth if bfs_max_depth != BFS_DEFAULT_MAX_DEPTH else None
+        ),
+    )
+
     retrieve_start = time.perf_counter()
     (
         out,
@@ -2329,6 +2484,31 @@ def retrieve_v2(
                     file=sys.stderr,
                 )
 
+    # #759 latency-signal update for bfs_depth_budget meta-belief. This
+    # is a SECOND meta-belief update, independent of the #756 half-life
+    # update above. Both fire on every retrieve_v2 call when their
+    # respective flags are on. Same evidence formula and fail-soft
+    # try/except posture as #756: store errors print to stderr and are
+    # swallowed; retrieval must never raise on a meta-belief write failure.
+    if is_meta_belief_bfs_depth_budget_enabled() and retrieve_elapsed > 0.0:
+        try:
+            from aelfrice.meta_beliefs import SIGNAL_LATENCY as _SIGNAL_LATENCY
+            evidence = max(0.0, min(
+                1.0, LATENCY_TARGET_SECONDS / retrieve_elapsed,
+            ))
+            store.update_meta_belief(
+                META_BFS_DEPTH_BUDGET_KEY,
+                _SIGNAL_LATENCY,
+                evidence=evidence,
+                now_ts=effective_now_ts,
+            )
+        except Exception as exc:  # noqa: BLE001
+            print(
+                "aelfrice retrieval: meta-belief bfs_depth_budget latency "
+                f"update failed: {exc}",
+                file=sys.stderr,
+            )
+
     compressed: list[CompressedBelief] = []
     if resolve_use_type_aware_compression(use_type_aware_compression):
         compressed = [