feat(retrieval): temporal-spine lane onto production retrieve() + flip default ON (#1107 phase 2, #1064)

CHANGELOG/v3.md

@@ -10,6 +10,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Production retrieval converges on `retrieve_v2`; temporal-spine lane goes live end-to-end ([#1107](https://github.com/robotrocketscience/aelfrice/issues/1107), Phase 2; [#1064](https://github.com/robotrocketscience/aelfrice/issues/1064)).** Several staged retrieval lanes were wired into `retrieve_v2()` only, so their default-ON flips changed benchmark/eval consumers but never what a live host retrieved — the production hook path (`hook_search.retrieve()`, `context_rebuilder`, `mcp_server`) still called the legacy `retrieve()`. The #1107 cutover makes `retrieve()` a thin adapter over `retrieve_v2` (Phase 1, behaviour-preserving: all six staged lanes forced off, pinned byte-identical by `tests/test_retrieve_v2_equivalence.py`), then graduates lanes onto the live path one at a time behind a hot-path latency gate. The **temporal-spine lane** ([#1064](https://github.com/robotrocketscience/aelfrice/issues/1064)) is the first to graduate: the shim now passes `use_temporal_spine` through resolver-driven (env → TOML → default) and its default is flipped **ON**, so — together with the writer flip ([#1111](https://github.com/robotrocketscience/aelfrice/issues/1111)) — the +14.6pp-coverage spine is live on every host's `retrieve()`, not just the eval surface. G3 latency stays in-band (`benchmarks/temporal_spine_latency.py`, Δp95 ≤ 50 ms); the lane is a no-op on spineless stores. Opt out with `AELFRICE_TEMPORAL_SPINE=0` or `[retrieval] use_temporal_spine = false`. The other five staged lanes remain forced off pending their own latency-gated graduation. Deterministic per #605.
+
 - **Origin-priority retrieval tie-break, default-off ([#1089](https://github.com/robotrocketscience/aelfrice/issues/1089), axis 2).** When two retrieval candidates tie on relevance, the higher-trust *origin* now wins — so a belief curated from a claude-memory `user`/`feedback` fact file (`origin=user_validated`) outranks one auto-captured from a chat transcript (`origin=user_transcript`). It is a pure within-tier **tie-break**, never a primary rerank term: the origin key sits *between* the relevance score and the id tie-break, so relevance always dominates and byte-identical behaviour is preserved when the flag is off. This is deliberately not a resurrected origin *rerank lane* — that was refuted on LoCoMo in #1013, where the failure was a BM25 *recall* limit (the trusted fact never became a candidate), which reranking cannot fix. The priority ladder mirrors `contradiction.precedence_class` (a sync-guard test keeps the two copies aligned) and is applied in both ranked candidate tiers: the L1 FTS rerank (`_l1_hits`) and the L2.5 entity-index overlap (`MemoryStore.lookup_entities`), so it covers both keyword-rerank and exact-entity-match hits. Flag-gated (`AELFRICE_ORIGIN_TIEBREAK` / `use_origin_tiebreak`), deterministic per #605; `benchmarks/origin_tiebreak_ablation.py` is the offline ablation. The mechanism lands default-off; the default-ON flip is gated on a LoCoMo retrieval no-regression run (note that a single-provenance corpus like LoCoMo shares one origin tier, so the tie-break is inert there and cannot regress recall) and is a separate operator call.
 
 - **`aelf introspect`: a native honest-signal view over stored beliefs ([#1081](https://github.com/robotrocketscience/aelfrice/issues/1081)).** Answers "look at my conversations and analyse the beliefs it extracted" without hand-rolled SQL. The command groups the active beliefs by ingest session (or `--by project`) and, for each, surfaces the signals that already live in the store but were never shown together: posterior mean μ and its evidence weight (α+β), **recurrence** (corroboration count, labelled explicitly as recurrence — *not* truth, since a junk line re-captured every session scores high on it alone; #1086), **grounding** (the #1096 entity-persistence axis — `durable` / `ephemeral` / `neutral` — i.e. standalone-meaningful vs context-bound), floated-vs-decided **status** read off `RESOLVES` / `POTENTIALLY_STALE` edges, and a **noise** flag (the #1081 `is_stranded_capture_noise` predicate — orphan headers and shell echoes). Stranded-capture rows float to the top of each group as the prime retire candidates. Read-only and deterministic per #605 (pure counts and edge lookups, no model, no clock); curation stays in the verbs the footer points at. Flags: `--session` / `--project` to scope, `--only-noise` for the retire shortlist, `--limit N` (0 = no cap), `--json`.
@@ -24,7 +26,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - **Dispatched subagents now inherit memory context ([#1068](https://github.com/robotrocketscience/aelfrice/issues/1068)).** Workers spawned via agent dispatch previously ran blind to the belief store — no SessionStart baseline, no per-prompt retrieval, and no locked constraints crossed the dispatch boundary. A new `PreToolUse:^(Agent|Task)$` hook (`aelf-agent-context-hook`) rewrites the worker's prompt through the harness's `updatedInput` channel, prepending a bounded `<aelfrice-worker-context>` block: L0 locked beliefs (with #1016 provenance-aware framing and reference-lock manifest bounding) plus beliefs relevant to the worker's task at the reduced auxiliary budget the Grep|Glob lane uses. Both harness channels were probed live before implementation (see the issue): `updatedInput` applies without a `permissionDecision` — so the hook never touches the user's permission flow — while `SubagentStart` was rejected for this lane (its payload carries no prompt text, so it cannot do query-aware retrieval). Fail-open everywhere (no store / malformed payload / retrieval error → byte-identical passthrough), idempotent on already-tagged prompts (nested dispatch never double-injects), and deterministic per #605. Installed by default via `aelf setup` (skip with `--no-agent-context`); runtime kill switch `AELFRICE_AGENT_CONTEXT=0`. Claude-host only for now — the Codex desired-set is unchanged pending #1056 live validation.
 
-- **Temporal spine: ingest-time chronological edges + a dedicated retrieval lane, default-off ([#1064](https://github.com/robotrocketscience/aelfrice/issues/1064)).** The largest retrieval-coverage gain measured on this codebase, deterministic and embeddings-free: per-session `TEMPORAL_NEXT` chains written at ingest (`[ingest] write_temporal_spine` / `AELFRICE_TEMPORAL_SPINE_WRITE`), an idempotent `aelf spine backfill` for existing stores plus an `aelf doctor` spine row, and a retrieval lane (`[retrieval] use_temporal_spine` / `AELFRICE_TEMPORAL_SPINE`, node budget via `temporal_spine_budget`) that traverses the chains from the top-5 L1 seeds and appends chronological neighbours after L1. Reaches gold that shares zero salient terms with the question through chronological adjacency: confirmed **+14.6pp** gold-set coverage on LoCoMo (0.460 → 0.606; temporal +17.2pp, multi-hop +10.4pp, 10× the shuffled control), out-of-sample gain exceeding dev (+12.7pp on LongMemEval). The ingest **writer** ships default-ON (every pre-registered gate passed — G1 +14.6pp coverage, G2 trim survival + top-rank invariance, G3 latency, G5 determinism — and the G4 auto-once backfill [#1090](https://github.com/robotrocketscience/aelfrice/issues/1090) is on `main`), so fresh ingests chain the spine by default and existing stores backfill on first `aelf setup`; opt out via `AELFRICE_TEMPORAL_SPINE_WRITE=0`. The **retrieval lane** stays default-OFF: it is wired into `retrieve_v2` only and is inert on the production `retrieve()` hook path until the cutover ([#1107](https://github.com/robotrocketscience/aelfrice/issues/1107)), so writing the spine now lets stores accumulate it ahead of that read-side flip. `LaneTelemetry` gains `temporal_spine` + `temporal_spine_candidates`; `benchmarks/temporal_spine_ablation.py` is the lane's permanent coverage ablation (baseline / +spine / seeded shuffled-control).
+- **Temporal spine: ingest-time chronological edges + a dedicated retrieval lane, default-off ([#1064](https://github.com/robotrocketscience/aelfrice/issues/1064)).** The largest retrieval-coverage gain measured on this codebase, deterministic and embeddings-free: per-session `TEMPORAL_NEXT` chains written at ingest (`[ingest] write_temporal_spine` / `AELFRICE_TEMPORAL_SPINE_WRITE`), an idempotent `aelf spine backfill` for existing stores plus an `aelf doctor` spine row, and a retrieval lane (`[retrieval] use_temporal_spine` / `AELFRICE_TEMPORAL_SPINE`, node budget via `temporal_spine_budget`) that traverses the chains from the top-5 L1 seeds and appends chronological neighbours after L1. Reaches gold that shares zero salient terms with the question through chronological adjacency: confirmed **+14.6pp** gold-set coverage on LoCoMo (0.460 → 0.606; temporal +17.2pp, multi-hop +10.4pp, 10× the shuffled control), out-of-sample gain exceeding dev (+12.7pp on LongMemEval). The ingest **writer** ships default-ON (every pre-registered gate passed — G1 +14.6pp coverage, G2 trim survival + top-rank invariance, G3 latency, G5 determinism — and the G4 auto-once backfill [#1090](https://github.com/robotrocketscience/aelfrice/issues/1090) is on `main`), so fresh ingests chain the spine by default and existing stores backfill on first `aelf setup`; opt out via `AELFRICE_TEMPORAL_SPINE_WRITE=0`. The **retrieval lane** is also default-ON as of the #1107 Phase-2 cutover (see the entry above); until that landed it was wired into `retrieve_v2` only and inert on the production `retrieve()` hook path. `LaneTelemetry` gains `temporal_spine` + `temporal_spine_candidates`; `benchmarks/temporal_spine_ablation.py` is the lane's permanent coverage ablation (baseline / +spine / seeded shuffled-control).
 
 - **Temporal-spine flip migration resolved: auto-once backfill + reversible `aelf spine clear` ([#1064](https://github.com/robotrocketscience/aelfrice/issues/1064), G4).** So the default-ON flip is effective on existing stores from day one rather than only after enough new turns accrue, the backfill now runs itself exactly once: `maybe_backfill_temporal_spine` is sentinel-gated on `~/.aelfrice/spine-backfilled` and wired into `aelf setup` beside the uv-migration. It is **gated on the writer flag**, so it stays inert while the spine is default-off (pre-flip) and re-arms — the first `aelf setup` after the flip turns the writer on fires it once, then the sentinel makes it a no-op; a fresh install is untouched until then. New **`aelf spine clear`** deletes every `TEMPORAL_NEXT` edge (beliefs untouched) as the reversibility path, and because the backfill is deterministic a later re-backfill rebuilds the identical spine. The flip itself (writer + lane defaults ON) remains a separate operator release call.
 

docs/design/feature-temporal-spine.md

@@ -1,13 +1,14 @@
 # Temporal spine — ingest-time chronological edges + dedicated retrieval lane (#1064)
 
-Status: **writer default-ON, lane default-off** (v4.0, #1064). Every
-evidence gate is complete — G1, G2 (both halves), G3, and G5 are DONE,
-and G4 is resolved (auto-once backfill, below). On the operator's go the
-ingest **writer** was flipped default-ON so stores accumulate the spine;
-the **retrieval lane** stays default-off because it is wired into
-`retrieve_v2` only and does not reach the production `retrieve()` hook
-path until the cutover (#1107). The lane's own default-ON flip lands with
-that cutover (its Phase 2), which is the remaining operator step.
+Status: **writer default-ON, lane default-ON** (v4.0, #1064) — the
+feature is live end-to-end. Every evidence gate is complete — G1, G2
+(both halves), G3, and G5 are DONE, and G4 is resolved (auto-once
+backfill, below). On the operator's go the ingest **writer** was flipped
+default-ON (#1111) so stores accumulate the spine, and the **retrieval
+lane** was flipped default-ON with the `retrieve_v2` production cutover
+(#1107 Phase 2): `retrieve()` is now a thin adapter over `retrieve_v2`
+that passes the lane through resolver-driven, so a fresh install both
+writes the spine and reads it on the live `retrieve()` hook path.
 
 ## Mechanism
 
@@ -93,10 +94,10 @@ density.
 ## Flip gate — pre-registered default-ON criteria
 
 Default-off was the **landing posture, not the end state**. All criteria
-passed; the ingest **writer** was flipped default-ON (with the backfill
-path for existing stores). The **lane** flag flips with the `retrieve_v2`
-production cutover (#1107, its Phase 2) rather than here, since it is
-inert on the production path until then:
+passed; the ingest **writer** was flipped default-ON (#1111, with the
+backfill path for existing stores) and the **lane** flag was flipped
+default-ON with the `retrieve_v2` production cutover (#1107 Phase 2),
+which is what exposes it on the live `retrieve()` hook path:
 
 - **G1 — confirmatory evidence:** DONE (above; recorded in #1064).
 - **G2 — production operating point:** coverage delta + top-rank
@@ -189,9 +190,9 @@ inert on the production path until then:
     install is untouched until then. Reversible via **`aelf spine clear`**
     (deletes every `TEMPORAL_NEXT` edge; beliefs untouched) — and because
     the backfill is deterministic (G5), a later re-backfill rebuilds the
-    identical spine. The writer default-ON flip has now landed; the lane
-    default-ON flip remains a separate operator release call (the #1107
-    cutover, Phase 2). This mechanism only readies the migration.
+    identical spine. Both flips have now landed: the writer default-ON
+    flip (#1111) and the lane default-ON flip with the #1107 Phase-2
+    cutover. This mechanism readied the migration for that flip.
 - **G5 — determinism/repro:** two-build byte-identity of the spine
   table on a fixed corpus; ablation bench green in CI.
   - **DONE.** `tests/test_temporal_spine_repro.py` pins both halves in the