diff --git a/CHANGELOG/v3.md b/CHANGELOG/v3.md index e887968a0..c2d00a12b 100644 --- a/CHANGELOG/v3.md +++ b/CHANGELOG/v3.md @@ -10,6 +10,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **Production retrieval converges on `retrieve_v2`; temporal-spine lane goes live end-to-end ([#1107](https://github.com/robotrocketscience/aelfrice/issues/1107), Phase 2; [#1064](https://github.com/robotrocketscience/aelfrice/issues/1064)).** Several staged retrieval lanes were wired into `retrieve_v2()` only, so their default-ON flips changed benchmark/eval consumers but never what a live host retrieved — the production hook path (`hook_search.retrieve()`, `context_rebuilder`, `mcp_server`) still called the legacy `retrieve()`. The #1107 cutover makes `retrieve()` a thin adapter over `retrieve_v2` (Phase 1, behaviour-preserving: all six staged lanes forced off, pinned byte-identical by `tests/test_retrieve_v2_equivalence.py`), then graduates lanes onto the live path one at a time behind a hot-path latency gate. The **temporal-spine lane** ([#1064](https://github.com/robotrocketscience/aelfrice/issues/1064)) is the first to graduate: the shim now passes `use_temporal_spine` through resolver-driven (env → TOML → default) and its default is flipped **ON**, so — together with the writer flip ([#1111](https://github.com/robotrocketscience/aelfrice/issues/1111)) — the +14.6pp-coverage spine is live on every host's `retrieve()`, not just the eval surface. G3 latency stays in-band (`benchmarks/temporal_spine_latency.py`, Δp95 ≤ 50 ms); the lane is a no-op on spineless stores. Opt out with `AELFRICE_TEMPORAL_SPINE=0` or `[retrieval] use_temporal_spine = false`. The other five staged lanes remain forced off pending their own latency-gated graduation. Deterministic per #605. + - **Origin-priority retrieval tie-break, default-off ([#1089](https://github.com/robotrocketscience/aelfrice/issues/1089), axis 2).** When two retrieval candidates tie on relevance, the higher-trust *origin* now wins — so a belief curated from a claude-memory `user`/`feedback` fact file (`origin=user_validated`) outranks one auto-captured from a chat transcript (`origin=user_transcript`). It is a pure within-tier **tie-break**, never a primary rerank term: the origin key sits *between* the relevance score and the id tie-break, so relevance always dominates and byte-identical behaviour is preserved when the flag is off. This is deliberately not a resurrected origin *rerank lane* — that was refuted on LoCoMo in #1013, where the failure was a BM25 *recall* limit (the trusted fact never became a candidate), which reranking cannot fix. The priority ladder mirrors `contradiction.precedence_class` (a sync-guard test keeps the two copies aligned) and is applied in both ranked candidate tiers: the L1 FTS rerank (`_l1_hits`) and the L2.5 entity-index overlap (`MemoryStore.lookup_entities`), so it covers both keyword-rerank and exact-entity-match hits. Flag-gated (`AELFRICE_ORIGIN_TIEBREAK` / `use_origin_tiebreak`), deterministic per #605; `benchmarks/origin_tiebreak_ablation.py` is the offline ablation. The mechanism lands default-off; the default-ON flip is gated on a LoCoMo retrieval no-regression run (note that a single-provenance corpus like LoCoMo shares one origin tier, so the tie-break is inert there and cannot regress recall) and is a separate operator call. - **`aelf introspect`: a native honest-signal view over stored beliefs ([#1081](https://github.com/robotrocketscience/aelfrice/issues/1081)).** Answers "look at my conversations and analyse the beliefs it extracted" without hand-rolled SQL. The command groups the active beliefs by ingest session (or `--by project`) and, for each, surfaces the signals that already live in the store but were never shown together: posterior mean μ and its evidence weight (α+β), **recurrence** (corroboration count, labelled explicitly as recurrence — *not* truth, since a junk line re-captured every session scores high on it alone; #1086), **grounding** (the #1096 entity-persistence axis — `durable` / `ephemeral` / `neutral` — i.e. standalone-meaningful vs context-bound), floated-vs-decided **status** read off `RESOLVES` / `POTENTIALLY_STALE` edges, and a **noise** flag (the #1081 `is_stranded_capture_noise` predicate — orphan headers and shell echoes). Stranded-capture rows float to the top of each group as the prime retire candidates. Read-only and deterministic per #605 (pure counts and edge lookups, no model, no clock); curation stays in the verbs the footer points at. Flags: `--session` / `--project` to scope, `--only-noise` for the retire shortlist, `--limit N` (0 = no cap), `--json`. @@ -24,7 +26,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Dispatched subagents now inherit memory context ([#1068](https://github.com/robotrocketscience/aelfrice/issues/1068)).** Workers spawned via agent dispatch previously ran blind to the belief store — no SessionStart baseline, no per-prompt retrieval, and no locked constraints crossed the dispatch boundary. A new `PreToolUse:^(Agent|Task)$` hook (`aelf-agent-context-hook`) rewrites the worker's prompt through the harness's `updatedInput` channel, prepending a bounded `` block: L0 locked beliefs (with #1016 provenance-aware framing and reference-lock manifest bounding) plus beliefs relevant to the worker's task at the reduced auxiliary budget the Grep|Glob lane uses. Both harness channels were probed live before implementation (see the issue): `updatedInput` applies without a `permissionDecision` — so the hook never touches the user's permission flow — while `SubagentStart` was rejected for this lane (its payload carries no prompt text, so it cannot do query-aware retrieval). Fail-open everywhere (no store / malformed payload / retrieval error → byte-identical passthrough), idempotent on already-tagged prompts (nested dispatch never double-injects), and deterministic per #605. Installed by default via `aelf setup` (skip with `--no-agent-context`); runtime kill switch `AELFRICE_AGENT_CONTEXT=0`. Claude-host only for now — the Codex desired-set is unchanged pending #1056 live validation. -- **Temporal spine: ingest-time chronological edges + a dedicated retrieval lane, default-off ([#1064](https://github.com/robotrocketscience/aelfrice/issues/1064)).** The largest retrieval-coverage gain measured on this codebase, deterministic and embeddings-free: per-session `TEMPORAL_NEXT` chains written at ingest (`[ingest] write_temporal_spine` / `AELFRICE_TEMPORAL_SPINE_WRITE`), an idempotent `aelf spine backfill` for existing stores plus an `aelf doctor` spine row, and a retrieval lane (`[retrieval] use_temporal_spine` / `AELFRICE_TEMPORAL_SPINE`, node budget via `temporal_spine_budget`) that traverses the chains from the top-5 L1 seeds and appends chronological neighbours after L1. Reaches gold that shares zero salient terms with the question through chronological adjacency: confirmed **+14.6pp** gold-set coverage on LoCoMo (0.460 → 0.606; temporal +17.2pp, multi-hop +10.4pp, 10× the shuffled control), out-of-sample gain exceeding dev (+12.7pp on LongMemEval). The ingest **writer** ships default-ON (every pre-registered gate passed — G1 +14.6pp coverage, G2 trim survival + top-rank invariance, G3 latency, G5 determinism — and the G4 auto-once backfill [#1090](https://github.com/robotrocketscience/aelfrice/issues/1090) is on `main`), so fresh ingests chain the spine by default and existing stores backfill on first `aelf setup`; opt out via `AELFRICE_TEMPORAL_SPINE_WRITE=0`. The **retrieval lane** stays default-OFF: it is wired into `retrieve_v2` only and is inert on the production `retrieve()` hook path until the cutover ([#1107](https://github.com/robotrocketscience/aelfrice/issues/1107)), so writing the spine now lets stores accumulate it ahead of that read-side flip. `LaneTelemetry` gains `temporal_spine` + `temporal_spine_candidates`; `benchmarks/temporal_spine_ablation.py` is the lane's permanent coverage ablation (baseline / +spine / seeded shuffled-control). +- **Temporal spine: ingest-time chronological edges + a dedicated retrieval lane, default-off ([#1064](https://github.com/robotrocketscience/aelfrice/issues/1064)).** The largest retrieval-coverage gain measured on this codebase, deterministic and embeddings-free: per-session `TEMPORAL_NEXT` chains written at ingest (`[ingest] write_temporal_spine` / `AELFRICE_TEMPORAL_SPINE_WRITE`), an idempotent `aelf spine backfill` for existing stores plus an `aelf doctor` spine row, and a retrieval lane (`[retrieval] use_temporal_spine` / `AELFRICE_TEMPORAL_SPINE`, node budget via `temporal_spine_budget`) that traverses the chains from the top-5 L1 seeds and appends chronological neighbours after L1. Reaches gold that shares zero salient terms with the question through chronological adjacency: confirmed **+14.6pp** gold-set coverage on LoCoMo (0.460 → 0.606; temporal +17.2pp, multi-hop +10.4pp, 10× the shuffled control), out-of-sample gain exceeding dev (+12.7pp on LongMemEval). The ingest **writer** ships default-ON (every pre-registered gate passed — G1 +14.6pp coverage, G2 trim survival + top-rank invariance, G3 latency, G5 determinism — and the G4 auto-once backfill [#1090](https://github.com/robotrocketscience/aelfrice/issues/1090) is on `main`), so fresh ingests chain the spine by default and existing stores backfill on first `aelf setup`; opt out via `AELFRICE_TEMPORAL_SPINE_WRITE=0`. The **retrieval lane** is also default-ON as of the #1107 Phase-2 cutover (see the entry above); until that landed it was wired into `retrieve_v2` only and inert on the production `retrieve()` hook path. `LaneTelemetry` gains `temporal_spine` + `temporal_spine_candidates`; `benchmarks/temporal_spine_ablation.py` is the lane's permanent coverage ablation (baseline / +spine / seeded shuffled-control). - **Temporal-spine flip migration resolved: auto-once backfill + reversible `aelf spine clear` ([#1064](https://github.com/robotrocketscience/aelfrice/issues/1064), G4).** So the default-ON flip is effective on existing stores from day one rather than only after enough new turns accrue, the backfill now runs itself exactly once: `maybe_backfill_temporal_spine` is sentinel-gated on `~/.aelfrice/spine-backfilled` and wired into `aelf setup` beside the uv-migration. It is **gated on the writer flag**, so it stays inert while the spine is default-off (pre-flip) and re-arms — the first `aelf setup` after the flip turns the writer on fires it once, then the sentinel makes it a no-op; a fresh install is untouched until then. New **`aelf spine clear`** deletes every `TEMPORAL_NEXT` edge (beliefs untouched) as the reversibility path, and because the backfill is deterministic a later re-backfill rebuilds the identical spine. The flip itself (writer + lane defaults ON) remains a separate operator release call. diff --git a/docs/design/feature-temporal-spine.md b/docs/design/feature-temporal-spine.md index bd0cf9d22..9be15af36 100644 --- a/docs/design/feature-temporal-spine.md +++ b/docs/design/feature-temporal-spine.md @@ -1,13 +1,14 @@ # Temporal spine — ingest-time chronological edges + dedicated retrieval lane (#1064) -Status: **writer default-ON, lane default-off** (v4.0, #1064). Every -evidence gate is complete — G1, G2 (both halves), G3, and G5 are DONE, -and G4 is resolved (auto-once backfill, below). On the operator's go the -ingest **writer** was flipped default-ON so stores accumulate the spine; -the **retrieval lane** stays default-off because it is wired into -`retrieve_v2` only and does not reach the production `retrieve()` hook -path until the cutover (#1107). The lane's own default-ON flip lands with -that cutover (its Phase 2), which is the remaining operator step. +Status: **writer default-ON, lane default-ON** (v4.0, #1064) — the +feature is live end-to-end. Every evidence gate is complete — G1, G2 +(both halves), G3, and G5 are DONE, and G4 is resolved (auto-once +backfill, below). On the operator's go the ingest **writer** was flipped +default-ON (#1111) so stores accumulate the spine, and the **retrieval +lane** was flipped default-ON with the `retrieve_v2` production cutover +(#1107 Phase 2): `retrieve()` is now a thin adapter over `retrieve_v2` +that passes the lane through resolver-driven, so a fresh install both +writes the spine and reads it on the live `retrieve()` hook path. ## Mechanism @@ -93,10 +94,10 @@ density. ## Flip gate — pre-registered default-ON criteria Default-off was the **landing posture, not the end state**. All criteria -passed; the ingest **writer** was flipped default-ON (with the backfill -path for existing stores). The **lane** flag flips with the `retrieve_v2` -production cutover (#1107, its Phase 2) rather than here, since it is -inert on the production path until then: +passed; the ingest **writer** was flipped default-ON (#1111, with the +backfill path for existing stores) and the **lane** flag was flipped +default-ON with the `retrieve_v2` production cutover (#1107 Phase 2), +which is what exposes it on the live `retrieve()` hook path: - **G1 — confirmatory evidence:** DONE (above; recorded in #1064). - **G2 — production operating point:** coverage delta + top-rank @@ -189,9 +190,9 @@ inert on the production path until then: install is untouched until then. Reversible via **`aelf spine clear`** (deletes every `TEMPORAL_NEXT` edge; beliefs untouched) — and because the backfill is deterministic (G5), a later re-backfill rebuilds the - identical spine. The writer default-ON flip has now landed; the lane - default-ON flip remains a separate operator release call (the #1107 - cutover, Phase 2). This mechanism only readies the migration. + identical spine. Both flips have now landed: the writer default-ON + flip (#1111) and the lane default-ON flip with the #1107 Phase-2 + cutover. This mechanism readied the migration for that flip. - **G5 — determinism/repro:** two-build byte-identity of the spine table on a fixed corpus; ablation bench green in CI. - **DONE.** `tests/test_temporal_spine_repro.py` pins both halves in the diff --git a/docs/user/CONFIG.md b/docs/user/CONFIG.md index a3f3209ec..ba637ddb9 100644 --- a/docs/user/CONFIG.md +++ b/docs/user/CONFIG.md @@ -9,7 +9,7 @@ This is the reference for power users whose project has a documentation idiom or A single optional TOML file at the root of a project (or any ancestor). It exposes the following power-user surfaces: - `[noise]` — onboard-time belief filter. Changes how `aelf onboard` ingests beliefs; nothing else. -- `[retrieval]` (v1.3+) — retrieval-time tier toggles + ranking. Knobs: `entity_index_enabled` (L2.5), `bfs_enabled` (L3), `posterior_weight` (partial Bayesian-weighted L1 ranking), `l1_limit` + `token_budget` (the #1045 wide-retrieval knobs — BM25 candidate cap + token budget, default 50/2400; raise both together for multi-hop recall), `use_bm25f_anchors` (BM25F-with-anchor-text since v1.7), `use_heat_kernel` (authority scoring lane, default-on since v2.1), `use_hrr_structural` (HRR structural-query lane, default-on since v2.1), `hrr_persist` (HRR structural-index on-disk persistence, default-on since v3.0), `use_type_aware_compression` (per-belief retention-class compression, default-on since #769), `use_intentional_clustering` (co-locating related beliefs, default-on since v3.0), `expansion_gate_enabled`, `use_gamma_posterior_temperature` (default off), and `use_zeta_posterior_rerank` (default off; mutually exclusive with the γ flag — `retrieve()` raises `ValueError` when both are on), `use_temporal_spine` + `temporal_spine_budget` (the #1064 chronological-adjacency lane, default off/32; pairs with `[ingest] write_temporal_spine`), `use_entity_persist_demote` (the #1096 entity-persistence demotion / organic-sink rerank modifier, default **on** in `retrieve_v2` since v4.0; not exposed on the legacy `retrieve()` path), `use_origin_tiebreak` (the #1089 origin-priority within-tier tie-break, default off). Two placeholder flags (`use_signed_laplacian`, `use_posterior_ranking`) are recognised but emit a deprecation warning if set — their lanes have not yet shipped. +- `[retrieval]` (v1.3+) — retrieval-time tier toggles + ranking. Knobs: `entity_index_enabled` (L2.5), `bfs_enabled` (L3), `posterior_weight` (partial Bayesian-weighted L1 ranking), `l1_limit` + `token_budget` (the #1045 wide-retrieval knobs — BM25 candidate cap + token budget, default 50/2400; raise both together for multi-hop recall), `use_bm25f_anchors` (BM25F-with-anchor-text since v1.7), `use_heat_kernel` (authority scoring lane, default-on since v2.1), `use_hrr_structural` (HRR structural-query lane, default-on since v2.1), `hrr_persist` (HRR structural-index on-disk persistence, default-on since v3.0), `use_type_aware_compression` (per-belief retention-class compression, default-on since #769), `use_intentional_clustering` (co-locating related beliefs, default-on since v3.0), `expansion_gate_enabled`, `use_gamma_posterior_temperature` (default off), and `use_zeta_posterior_rerank` (default off; mutually exclusive with the γ flag — `retrieve()` raises `ValueError` when both are on), `use_temporal_spine` + `temporal_spine_budget` (the #1064 chronological-adjacency lane, default **on**/32 since v4.0 — live on the production `retrieve()` path via the #1107 cutover; pairs with `[ingest] write_temporal_spine`), `use_entity_persist_demote` (the #1096 entity-persistence demotion / organic-sink rerank modifier, default **on** in `retrieve_v2` since v4.0; not exposed on the legacy `retrieve()` path), `use_origin_tiebreak` (the #1089 origin-priority within-tier tie-break, default off). Two placeholder flags (`use_signed_laplacian`, `use_posterior_ranking`) are recognised but emit a deprecation warning if set — their lanes have not yet shipped. - `[rebuilder]` (v1.4+) — context-rebuilder knobs: `turn_window_n` (default 50), `token_budget` (default 4000), `trigger_mode` (`manual`|`threshold`|`dynamic`, default `threshold`), `threshold_fraction` (default 0.6), and `query_strategy` (v1.7+, default `stack-r1-r3` since v3.0). `[rebuild_floor]` (v1.7+) sets the token-budget floors for the session-scoped and L1 belief lanes (`[rebuild_floor] session` and `[rebuild_floor] l1`). - `[onboard.llm]` (v1.3.0+) — direct-API onboard classifier gate; documented under [Keys § `[onboard.llm]`](#onboardllm-v130) below. - `[cadence]`, `[implicit_feedback]`, and `[hook_audit]` — feedback-cadence scoring, deferred retrieval-exposure feedback, and the per-turn hook audit log. Recognised here but documented in their module docstrings (`src/aelfrice/cadence.py`, `src/aelfrice/deferred_feedback.py`, `src/aelfrice/hook.py`). @@ -152,11 +152,12 @@ use_intentional_clustering = true # neighbours after the L1 candidates. Reaches gold that shares zero # salient terms with the question through chronological adjacency — # confirmed +14.6pp gold-coverage on LoCoMo, 10x its shuffled control. -# No-op on stores with zero TEMPORAL_NEXT edges (run `aelf spine -# backfill` to build the spine on an existing store, and enable the -# [ingest] write_temporal_spine writer to keep it growing). -# AELFRICE_TEMPORAL_SPINE env var overrides. -use_temporal_spine = false +# Default `true` since the v4.0 lane flip (#1064, #1107 Phase 2) — live +# on the production retrieve() hook path. No-op on stores with zero +# TEMPORAL_NEXT edges (run `aelf spine backfill` to build the spine on an +# existing store; the [ingest] write_temporal_spine writer keeps it +# growing). Opt out with `AELFRICE_TEMPORAL_SPINE=0` or this key = false. +use_temporal_spine = true # v4.0.0+ (#1064). Node budget for the temporal-spine lane traversal # (default 32). The confirmatory budget curve is monotone (~+2.5pp @@ -177,9 +178,9 @@ temporal_spine_budget = 32 # v4.0.0+ (#1064). Default `true` since the writer flip. Every belief # insert chains to its session predecessor with a TEMPORAL_NEXT edge # (src = successor, weight 0.8), building the per-session temporal spine. -# The retrieval lane (use_temporal_spine above) stays default-off until -# the retrieve_v2 production cutover (#1107), so the two resolve -# independently. One edge per belief, O(1) per insert; explicit opt-out +# The retrieval lane (use_temporal_spine above) is also default-on since +# the #1107 Phase-2 cutover; the two flags resolve independently. One edge +# per belief, O(1) per insert; explicit opt-out # is byte-identical. AELFRICE_TEMPORAL_SPINE_WRITE env var overrides. write_temporal_spine = true @@ -518,7 +519,7 @@ Precedence (first decisive wins): env var `AELFRICE_TYPE_AWARE_COMPRESSION=0`/`1 ### `use_temporal_spine` / `temporal_spine_budget` -v4.0.0+ (#1064). Default `false` / `32`. The temporal-spine retrieval lane: +v4.0.0+ (#1064). Default `true` / `32`. The temporal-spine retrieval lane: an additive candidate source after L1 that traverses `TEMPORAL_NEXT` chronological chains from the top-5 packed L1 seeds (both directions, depth 1) and appends the neighbours — never displacing L1 pre-packing. @@ -527,9 +528,12 @@ zero salient terms with the question become reachable through chronological adjacency to beliefs that do match. No-op guard: stores with zero `TEMPORAL_NEXT` edges get byte-identical output at ~zero cost. Precedence: `AELFRICE_TEMPORAL_SPINE` / `AELFRICE_TEMPORAL_SPINE_BUDGET` -env → explicit kwarg → TOML → default. Default-off is the landing -posture; the default-ON flip is gated on the pre-registered criteria in -[docs/design/feature-temporal-spine.md](../design/feature-temporal-spine.md). +env → explicit kwarg → TOML → default. Default-**ON** since the #1107 +Phase-2 cutover — the lane is live on the production `retrieve()` hook +path, not just `retrieve_v2` — after every pre-registered gate in +[docs/design/feature-temporal-spine.md](../design/feature-temporal-spine.md) +cleared. Opt out with `AELFRICE_TEMPORAL_SPINE=0` or `[retrieval] +use_temporal_spine = false`. ### Placeholder flags @@ -544,9 +548,9 @@ Default `true` since the v4.0 writer flip (#1064); opt out with belief insert links to the previous belief in the same session (`created_at` order, insertion-order tie-break) with a `TEMPORAL_NEXT` edge — the per-session temporal spine the `use_temporal_spine` retrieval -lane traverses. That lane stays default-off until the `retrieve_v2` -production cutover (#1107), so writing the spine now lets stores -accumulate it ahead of the read-side flip. One edge per belief, +lane traverses. That lane is default-on since the #1107 Phase-2 cutover, +so a fresh install writes the spine and reads it end-to-end. One edge per +belief, O(1) per insert, idempotent; the opt-out path is byte-identical to today. Existing stores predate the writer: `aelf spine backfill` builds their chains (idempotent, `--dry-run` supported), and `aelf doctor` reports diff --git a/src/aelfrice/retrieval.py b/src/aelfrice/retrieval.py index f9a9c4c86..670b77b00 100644 --- a/src/aelfrice/retrieval.py +++ b/src/aelfrice/retrieval.py @@ -1938,15 +1938,18 @@ def is_temporal_spine_enabled( 1. AELFRICE_TEMPORAL_SPINE env var (truthy / falsy normalised). 2. Explicit `explicit` kwarg from the caller. 3. `[retrieval] use_temporal_spine` in `.aelfrice.toml`. - 4. Default: **False** — the lane lands default-OFF. The default-ON - flip is gated on the pre-registered #1064 criteria (G2-G5), a - release deliverable rather than a config change. Distinct from + 4. Default: **True** — the lane is default-ON since the #1064 lane + flip (v4.0, #1107 Phase 2). Every pre-registered gate cleared + (G1 +14.6pp LoCoMo coverage, G2 trim survival + top-rank + invariance, G3 latency delta_p95 in-band, G5 determinism); the + production `retrieve()` hook path exposes it via the #1107 shim. + Opt out with `AELFRICE_TEMPORAL_SPINE=0` or `[retrieval] + use_temporal_spine = false`. Distinct from `AELFRICE_TEMPORAL_SPINE_WRITE` (the ingest-time writer flag in - `aelfrice.temporal_spine`) — the two flip together at release - time but resolve independently. + `aelfrice.temporal_spine`) — they resolve independently. Passing the flag (any rung) never raises — an unset / unrecognised - value falls through to the next rung and ultimately to False. + value falls through to the next rung and ultimately to the default (True). """ env = _env_temporal_spine_override() if env is not None: @@ -1956,7 +1959,7 @@ def is_temporal_spine_enabled( toml_value = _read_toml_flag_for(TEMPORAL_SPINE_FLAG, start) if toml_value is not None: return toml_value - return False + return True def resolve_temporal_spine_budget( @@ -2993,15 +2996,17 @@ def retrieve( bare `retrieve()` so `rebuild_v14`'s call site observes the toggle that A4 (#775) measures. """ - # #1107 Phase 1: `retrieve()` is now a thin adapter over `retrieve_v2`, - # the single retrieval implementation the production hook path shares - # with the benchmark/eval surface. The lane config below reproduces the - # historical bare-`retrieve()` behaviour exactly: every post-v2.1 staged - # lane (temporal spine, entity-persist demotion, origin tie-break, - # HRR-expand, intentional clustering, HRR-structural) is forced OFF, - # because `retrieve()`'s own pack loop never ran them. Byte-identical - # equivalence is pinned by tests/test_retrieve_v2_equivalence.py. Lanes - # light up in production one at a time as each clears its latency gate. + # #1107 Phase 2: `retrieve()` is a thin adapter over `retrieve_v2`, the + # single retrieval implementation the production hook path shares with + # the benchmark/eval surface. Lanes light up in production one at a time + # as each clears its latency gate. The **temporal-spine** lane is the + # first to graduate (#1064): its flag is passed through as `None` so the + # production `retrieve()` path honours the `is_temporal_spine_enabled` + # resolver (env -> TOML -> default) exactly like the eval surface. The + # remaining five staged lanes (entity-persist demotion, origin tie-break, + # HRR-expand, intentional clustering, HRR-structural) stay forced OFF, + # because `retrieve()`'s historical pack loop never ran them; equivalence + # for those is pinned by tests/test_retrieve_v2_equivalence.py. out = retrieve_v2( store, query, @@ -3023,7 +3028,10 @@ def retrieve( eigenbasis_cache=eigenbasis_cache, use_type_aware_compression=use_type_aware_compression, manifest_reference_locks=manifest_reference_locks, - use_temporal_spine=False, + # Temporal-spine pilot lane (#1064/#1107 Phase 2): resolver-driven + # (env -> TOML -> default) rather than hard-off, so a production host + # gets the lane the moment its resolver says on. + use_temporal_spine=None, use_entity_persist_demote=False, use_origin_tiebreak=False, use_hrr_expand=False, @@ -3304,8 +3312,8 @@ def _cost(b: Belief) -> int: seen_pre.add(b.id) used += cost - # #1064 temporal-spine lane (additive, default-OFF). Traverses - # TEMPORAL_NEXT chains from the top-5 packed L1 seeds, both + # #1064 temporal-spine lane (additive, default-ON since #1107 Phase 2). + # Traverses TEMPORAL_NEXT chains from the top-5 packed L1 seeds, both # directions, depth 1 by default, and appends the neighbours after # the L1 candidates — never displacing them pre-packing. No-op # guard: a store with zero TEMPORAL_NEXT edges skips the traversal @@ -3513,10 +3521,10 @@ def retrieve_v2( directions, depth 1 by default, node budget 32 by default) and appends the chronological neighbours to the candidate set. The lane is a no-op (byte-identical output) when the store has zero - TEMPORAL_NEXT edges. Default-OFF — the default-ON flip is gated - on the pre-registered #1064 criteria. Opt in via - `AELFRICE_TEMPORAL_SPINE=1`, the kwarg, or - `[retrieval] use_temporal_spine = true`. + TEMPORAL_NEXT edges. Default-ON since the #1107 Phase-2 cutover + (every pre-registered #1064 gate cleared). Opt out via + `AELFRICE_TEMPORAL_SPINE=0`, the kwarg, or + `[retrieval] use_temporal_spine = false`. `temporal_spine_depth` / `temporal_spine_node_budget` tune the traversal (budget also via `AELFRICE_TEMPORAL_SPINE_BUDGET` env or `[retrieval] temporal_spine_budget` TOML). diff --git a/tests/test_retrieve_v2_equivalence.py b/tests/test_retrieve_v2_equivalence.py index b598c2e8d..fe0c2184e 100644 --- a/tests/test_retrieve_v2_equivalence.py +++ b/tests/test_retrieve_v2_equivalence.py @@ -1,14 +1,18 @@ -"""Equivalence guard: `retrieve_v2` with all post-#1064 lanes OFF must be -byte-identical to the legacy `retrieve()` — the regression net the #1107 -production cutover depends on. - -The cutover migrates 8 production call sites from `retrieve()` (returns -`list[Belief]`) to `retrieve_v2()` (returns `RetrievalResult`). That is only -safe if `retrieve_v2`, with the lanes the hook does not want (temporal spine, -entity-persist demotion, origin tie-break, HRR-expand) forced off, produces -the same ranked ids as `retrieve()` for the same inputs. These tests pin that -equivalence across every tier and budget regime so a later change to either -implementation cannot silently diverge production retrieval. +"""Equivalence guard: `retrieve()` (the production adapter) must be +byte-identical to `retrieve_v2` run with the exact lane config the #1107 shim +pins — the regression net the production cutover depends on. + +The cutover migrated the production call sites from the legacy bare +`retrieve()` pack loop to a thin adapter over `retrieve_v2()`. Post-#1107 +Phase 2 the shim runs the **temporal-spine** lane ON (resolver-driven, the +pilot lane graduated #1064) and the other five staged lanes (entity-persist +demotion, origin tie-break, HRR-expand, intentional clustering, +HRR-structural) OFF. `SHIM_LANES` below is that exact config, so these tests +stay true identities rather than coincidences. The L0/L1/L2.5/BFS/manifest +cases run on spineless corpora (no `TEMPORAL_NEXT` edges), where the spine +lane is a no-op — so the shim output matches for the reasons those tiers +exercise; `test_shim_runs_temporal_spine_lane_others_off` covers the spine +lane being live and the other five staying off non-vacuously. `manifest_reference_locks` parity (#1016-B) was `retrieve()`-only until the #1107 Phase-0 port; the manifest cases below would have failed before it. @@ -26,20 +30,21 @@ Belief, Edge, EDGE_DERIVED_FROM, + EDGE_TEMPORAL_NEXT, ) from aelfrice.retrieval import retrieve, retrieve_v2 from aelfrice.store import MemoryStore -# The production lane config: the exact set `retrieve()` runs. `retrieve()` -# has its own pack loop and does NOT invoke intentional clustering, -# HRR-structural, HRR-expand, temporal spine, entity-persist demotion, or -# the origin tie-break — even though several of those default ON in -# `retrieve_v2`. So all six must be forced off for `retrieve_v2` to be -# byte-identical to `retrieve()` (clustering in particular reorders the L1 -# pack on graph-connected corpora — see test_equivalence_clustered_corpus). -# This is the config the #1107 shim pins. -LANES_OFF = dict( - use_temporal_spine=False, +# The production lane config the #1107 shim pins. Post-Phase-2 the +# temporal-spine lane is resolver-driven (`None` -> `is_temporal_spine_enabled`, +# default ON) exactly as the shim passes it; the other five staged lanes stay +# forced OFF because `retrieve()`'s historical pack loop never ran them +# (clustering in particular reorders the L1 pack on graph-connected corpora — +# see test_equivalence_clustered_corpus). On the spineless corpora these +# equivalence cases use, the spine lane is a no-op, so this config reproduces +# the legacy bare-`retrieve()` output byte-for-byte. +SHIM_LANES = dict( + use_temporal_spine=None, use_entity_persist_demote=False, use_origin_tiebreak=False, use_hrr_expand=False, @@ -78,7 +83,7 @@ def _v1(store: MemoryStore, query: str, **kw) -> list[str]: def _v2(store: MemoryStore, query: str, **kw) -> list[str]: - return [b.id for b in retrieve_v2(store, query, **{**LANES_OFF, **kw}).beliefs] + return [b.id for b in retrieve_v2(store, query, **{**SHIM_LANES, **kw}).beliefs] def test_equivalence_l1_vocab() -> None: @@ -176,7 +181,7 @@ def test_equivalence_bfs() -> None: def test_equivalence_clustered_corpus() -> None: """Two dense DERIVED_FROM clusters — the case where intentional clustering reorders the L1 pack. `retrieve()` does not cluster, so - `retrieve_v2` must have clustering forced off (part of LANES_OFF) to + `retrieve_v2` must have clustering forced off (part of SHIM_LANES) to match. This is the case a 4-lane-off config silently diverges on.""" s = MemoryStore(":memory:") for i in range(6): @@ -216,15 +221,52 @@ def test_equivalence_manifest_reference_locks(manifest: bool, budget: int) -> No s.close() -def test_retrieve_keeps_staged_lanes_off() -> None: - """Contract guard for the #1107 shim: `retrieve()` (the production - adapter) must run with the staged lanes OFF. Post-shim the plain - equivalence assertions are tautological (both sides are retrieve_v2), - so this checks the live contract non-tautologically: on a clustered - corpus where the clustering lane demonstrably reorders the pack, - `retrieve()` must match retrieve_v2(clustering off) and DIFFER from - retrieve_v2(clustering on). If someone flips a lane on in the shim, - this fails.""" +def test_shim_runs_temporal_spine_lane_others_off() -> None: + """Contract guard for the #1107 Phase-2 shim: `retrieve()` (the + production adapter) runs the temporal-spine lane ON (resolver default, + the graduated pilot lane) and the other five staged lanes OFF. Both + halves are non-vacuous: + + * spine ON — on a `TEMPORAL_NEXT`-connected corpus, `retrieve()` + surfaces the chronological neighbour that shares no query vocabulary, + matching retrieve_v2(spine on) and differing from + retrieve_v2(spine off). + * others OFF — on a clustered corpus where intentional clustering + demonstrably reorders the L1 pack, `retrieve()` matches + retrieve_v2(clustering off) and differs from + retrieve_v2(clustering on). + + If the spine lane regresses off, or any of the other five is flipped on + in the shim, one of these fails.""" + # --- temporal-spine lane is live in the shim --- + s = MemoryStore(":memory:") + _mk(s, "anchor", "cache latency tuning note") + _mk(s, "neighbor", "zebra quokka xylophone unrelated marmalade") + # A single spine edge; the lane traverses depth-1 in both directions. + s.insert_edge( + Edge(src="anchor", dst="neighbor", + type=EDGE_TEMPORAL_NEXT, weight=0.8) + ) + q = "cache latency" + try: + prod = [b.id for b in retrieve(s, q, token_budget=2400)] + spine_on = [ + b.id for b in retrieve_v2( + s, q, budget=2400, + **{**SHIM_LANES, "use_temporal_spine": True}).beliefs + ] + spine_off = [ + b.id for b in retrieve_v2( + s, q, budget=2400, + **{**SHIM_LANES, "use_temporal_spine": False}).beliefs + ] + assert "neighbor" in prod, "shim must run the temporal-spine lane" + assert prod == spine_on + assert prod != spine_off, "spine lane must be load-bearing here" + finally: + s.close() + + # --- the other five staged lanes stay off --- s = MemoryStore(":memory:") for i in range(6): _mk(s, f"a{i}", f"cache latency tuning cluster note {i}") @@ -237,13 +279,9 @@ def test_retrieve_keeps_staged_lanes_off() -> None: try: prod = [b.id for b in retrieve(s, q, token_budget=2400)] clustering_on = [ - b.id - for b in retrieve_v2( + b.id for b in retrieve_v2( s, q, budget=2400, - use_temporal_spine=False, use_entity_persist_demote=False, - use_origin_tiebreak=False, use_hrr_expand=False, - use_intentional_clustering=True, use_hrr_structural=False, - ).beliefs + **{**SHIM_LANES, "use_intentional_clustering": True}).beliefs ] clustering_off = _v2(s, q, budget=2400) assert prod == clustering_off, "retrieve() must have clustering OFF" diff --git a/tests/test_temporal_spine.py b/tests/test_temporal_spine.py index 52c28a127..be180d66c 100644 --- a/tests/test_temporal_spine.py +++ b/tests/test_temporal_spine.py @@ -495,13 +495,32 @@ def _seed_lane_store(store: MemoryStore) -> None: backfill_temporal_spine(store) -def test_lane_default_off_flag() -> None: - assert is_temporal_spine_enabled() is False - assert is_temporal_spine_enabled(explicit=True) is True +def test_lane_default_on_flag( + monkeypatch: pytest.MonkeyPatch, tmp_path, +) -> None: + # Default-ON since the #1064 lane flip (#1107 Phase 2). Start at an + # empty dir so no repo .aelfrice.toml is found. + monkeypatch.delenv(ENV_TEMPORAL_SPINE, raising=False) + assert is_temporal_spine_enabled(start=tmp_path) is True + assert is_temporal_spine_enabled(explicit=False, start=tmp_path) is False assert resolve_temporal_spine_budget() == 32 assert resolve_temporal_spine_budget(explicit=7) == 7 +def test_lane_explicit_opt_out( + monkeypatch: pytest.MonkeyPatch, tmp_path, +) -> None: + # The lane is opt-out now that the default is ON; the env var and the + # TOML key must still force it back off. + monkeypatch.setenv(ENV_TEMPORAL_SPINE, "off") + assert is_temporal_spine_enabled(start=tmp_path) is False + monkeypatch.delenv(ENV_TEMPORAL_SPINE, raising=False) + (tmp_path / ".aelfrice.toml").write_text( + "[retrieval]\nuse_temporal_spine = false\n" + ) + assert is_temporal_spine_enabled(start=tmp_path) is False + + def test_lane_off_omits_neighbours(store: MemoryStore) -> None: _seed_lane_store(store) result = retrieve_v2(store, _QUERY, use_temporal_spine=False)