feat(eval-harness): wire context-rebuilder eval harness commit-1 (#592)

CHANGELOG.md

@@ -16,6 +16,8 @@ installable release; see the roadmap in [README.md](README.md).
 
 ### Added
 
+- **Context-rebuilder eval harness — three integration points wired** ([#592](https://github.com/robotrocketscience/aelfrice/issues/592)). `benchmarks/context-rebuilder/eval_harness.py` was a skeleton with five `NotImplementedError` stubs since v1.2.0; this lands the three pure-Python wirings that make `--mode threshold-sweep` and `--mode budget-sweep` runnable end-to-end. `replay_to_fork` populates a fresh in-memory `MemoryStore` via `ingest_jsonl` against a tempfile holding the first `fork_turn` lines of the case JSONL. `run_rebuilder` calls `rebuild_v14()` with `floor_session=floor_l1=trigger_threshold` (mapping the harness's threshold-sweep axis onto the v1.7 per-lane composite-score floor) and returns `(rebuilt_block, latency_ms)` measured via `time.monotonic()`. `measure_token_cost` returns `estimate_tokens(rebuilt) / estimate_tokens(pre_clear)` using `benchmarks.context_rebuilder.measure.estimate_tokens` (the 4-chars-per-token heuristic that mirrors `aelfrice.context_rebuilder._CHARS_PER_TOKEN`), so harness measurements stay aligned with the rebuilder's own budget bookkeeping. `replay_post_fork` is a non-crashing stub that returns one placeholder per `eval_turn` (`{turn_idx, expected, actual="", matched=False, reason="needs_replay_client"}`) so the harness produces valid latency + token-cost numbers before the model-invocation client lands; `score_fidelity` reads `matched=False` and returns 0.0 instead of raising. Adds `debugging_session_001.meta.json` next to the bundled 16-turn synthetic fixture (midpoint fork at turn 8, eval_turns at indices 8/10/12/14) so `--corpus benchmarks/context-rebuilder/fixtures/synthetic/` resolves out of the box. 15 new tests in `tests/test_context_rebuilder_eval_harness_wiring.py` cover each wired function plus an end-to-end threshold-sweep smoke against the bundled fixture. The model-invocation client + LLM judge stages are explicit follow-ups; this commit lands the harness's runnable contract before fidelity scoring engages.
+
 - **Session-end Stop hook prompts to lock session corrections** ([#582](https://github.com/robotrocketscience/aelfrice/issues/582)). New `aelf-stop-hook` (default-on) fires once per assistant-turn end, walks the store for unlocked correction-class beliefs created in the current `session_id`, and emits a `<aelfrice-session-end>` block to stderr listing each candidate with a pre-filled `aelf lock --statement '<...>'` command. Candidate filter: `session_id == current AND lock_level != LOCK_USER AND (type == BELIEF_CORRECTION OR origin in {agent_inferred, agent_remembered})`. Hook is informational by default; setting `AELF_AUTOLOCK_CORRECTIONS=1` in the environment makes it auto-lock the candidates instead (logs each lock to stderr for transparency). Wired into `aelf setup` / `aelf unsetup` (`--no-stop-hook` opts out) and into `aelf doctor` as a fourth default-on auto-capture hook the v2.1 nag flags when missing. Coexists with the existing transcript-ingest Stop entry as a separate entry under the same `hooks.Stop` event key. **Note**: the issue's spec assumed a `aelf correct` CLI command and a `feedback_history.kind=correct` marker that don't exist on `main` (no historical implementation); the v0 detection signal is correction-class beliefs in the current session instead. Future work to ship `aelf correct` + `feedback_history.kind` would let this hook also surface beliefs that were *modified* (not only newly-created) in the session.
 
 - **Dep-graph render workflow — Mermaid graph of open-issue Blocks/Blocked-by links** ([#581](https://github.com/robotrocketscience/aelfrice/issues/581)). New `.github/workflows/dep-graph-render.yml` (daily cron + `workflow_dispatch`) drives `scripts/dep_graph_render.py`, which pages the GraphQL `trackedIssues` / `trackedInIssues` connections for open issues matching `--label` (default `v2.1`), renders a deterministic `graph TD` block, and posts/updates a sticky comment (`<!-- dep-graph-auto -->` marker) on the milestone tracker (default #474). Nodes are class-coloured: red = has open blockers, green = leaf (work-ready), yellow = in-progress (any assignee or any `author-*` label). Determinism is byte-stable across runs so unchanged data produces a noop comment edit. Closes the parallel-session work-finding gap where each session had to scan all open issues to spot a leaf node.

benchmarks/context-rebuilder/eval_harness.py

@@ -18,10 +18,31 @@
 import argparse
 import json
 import statistics
+import sys
+import tempfile
 import time
 from dataclasses import asdict, dataclass, field
 from pathlib import Path
-from typing import Literal
+from typing import Final, Literal
+
+# Allow `python benchmarks/context-rebuilder/eval_harness.py` to import
+# aelfrice and the benchmarks.context_rebuilder package. The hyphenated
+# parent directory blocks `python -m`, so a sys.path shim is the
+# alternative. Tests already get the repo root from pytest's testpaths
+# config and skip this branch as a no-op.
+_REPO_ROOT = Path(__file__).resolve().parents[2]
+if str(_REPO_ROOT) not in sys.path:
+    sys.path.insert(0, str(_REPO_ROOT))
+
+from aelfrice.context_rebuilder import (  # noqa: E402
+    DEFAULT_N_RECENT_TURNS,
+    DEFAULT_REBUILDER_TOKEN_BUDGET,
+    RecentTurn,
+    rebuild_v14,
+)
+from aelfrice.ingest import ingest_jsonl  # noqa: E402
+from aelfrice.store import MemoryStore  # noqa: E402
+from benchmarks.context_rebuilder.measure import estimate_tokens  # noqa: E402
 
 
 Mode = Literal["threshold-sweep", "budget-sweep", "dynamic", "regression"]
@@ -82,29 +103,120 @@
     return cases
 
 
-def replay_to_fork(case: TranscriptCase) -> "object":
-    """Populate a fresh aelfrice store with turns 0..fork_turn-1.
+def replay_to_fork(case: TranscriptCase) -> MemoryStore:
+    """Populate a fresh in-memory aelfrice store with turns
+    0..fork_turn-1 from `case.path`.
 
-    Returns the store handle. TODO: wire to MemoryStore + ingest_jsonl
-    once docs/transcript_ingest.md ships.
+    Returns the store handle. Caller is responsible for closing it.
+
+    Lines are counted by file order — this matches the synthetic-fixture
+    convention where each line is one user/assistant turn. Compaction
+    markers and tool-result lines are passed through to ingest_jsonl,
+    which already skips them; they don't shift the fork boundary
+    relative to the underlying transcript file.
+
+    Implementation note: ingest_jsonl reads a file end-to-end, so we
+    write the first `fork_turn` lines to a tempfile and ingest that.
+    Modifying ingest_jsonl to take a line-count cap would couple the
+    production ingest path to harness internals; keeping the truncation
+    here is the lighter coupling.
+    """
+    store = MemoryStore(":memory:")
+    if case.fork_turn <= 0:
+        return store
+    with tempfile.NamedTemporaryFile(
+        mode="w", suffix=".jsonl", delete=False, encoding="utf-8"
+    ) as tmp:
+        tmp_path = Path(tmp.name)
+        with case.path.open("r", encoding="utf-8") as src:
+            for i, line in enumerate(src):
+                if i >= case.fork_turn:
+                    break
+                tmp.write(line)
+    try:
+        ingest_jsonl(store, tmp_path, source_label="eval-harness")
+    finally:
+        tmp_path.unlink(missing_ok=True)
+    return store
+
+
+def _recent_turns_pre_fork(
+    case: TranscriptCase, n: int = DEFAULT_N_RECENT_TURNS,
+) -> list[RecentTurn]:
+    """Build the last `n` RecentTurn records from turns 0..fork_turn-1.
+
+    Mirrors the production hook's adapter: each line is a JSON object
+    with `role` and `text`, plus an optional `session_id`. Lines that
+    don't carry role/text are skipped (compaction markers, malformed).
     """
-    raise NotImplementedError("Wire to ingest_jsonl from transcript_ingest spec")
+    turns: list[RecentTurn] = []
+    if case.fork_turn <= 0:
+        return turns
+    with case.path.open("r", encoding="utf-8") as f:
+        for i, line in enumerate(f):
+            if i >= case.fork_turn:
+                break
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                obj = json.loads(line)
+            except json.JSONDecodeError:
+                continue
+            if not isinstance(obj, dict):
+                continue
+            role = obj.get("role")
+            text = obj.get("text")
+            if not isinstance(role, str) or not isinstance(text, str):
+                continue
+            sid = obj.get("session_id")
+            turns.append(
+                RecentTurn(
+                    role=role, text=text,
+                    session_id=sid if isinstance(sid, str) else None,
+                )
+            )
+    return turns[-n:]
 
 
 def run_rebuilder(
-    store: object,
+    store: MemoryStore,
     case: TranscriptCase,
     *,
     trigger_threshold: float,
     token_budget: int,
 ) -> tuple[str, float]:
-    """Run the rebuilder against the store at the fork point.
+    """Run the v1.4 rebuilder against the store at the fork point.
+
+    Returns `(rebuilt_context_block, latency_ms)`. Latency is measured
+    via `time.monotonic()` so it stays monotonic by construction.
+
+    `trigger_threshold` is wired to `rebuild_v14`'s per-lane composite-
+    score floors (`floor_session` and `floor_l1`). The v1.7 floor knob
+    is the closest existing rebuild-time parameter to the harness's
+    "threshold-sweep" axis; sweeping it answers "at what relevance
+    floor does the rebuilder still pack useful content for this task?"
+    L0 locked beliefs are unaffected by the floor — they always pack
+    per the documented v1.4 contract.
 
-    Returns (rebuilt_context_block, latency_ms). TODO: wire to
-    aelfrice.context_rebuilder.rebuild() once context_rebuilder spec
-    ships an implementation.
+    `token_budget` is forwarded directly to `rebuild_v14`.
     """
-    raise NotImplementedError("Wire to context_rebuilder.rebuild()")
+    recent = _recent_turns_pre_fork(case)
+    t0 = time.monotonic()
+    rebuilt = rebuild_v14(
+        recent, store,
+        token_budget=token_budget,
+        floor_session=trigger_threshold,
+        floor_l1=trigger_threshold,
+    )
+    latency_ms = (time.monotonic() - t0) * 1000.0
+    return rebuilt, latency_ms
+
+
+REPLAY_PENDING_REASON: Final[str] = "needs_replay_client"
+"""Marker emitted by `replay_post_fork` until the model-invocation
+client lands. Surfaced in `failures[].reason` so summary readers can
+distinguish "skipped, judge required" from a real fidelity miss."""
 
 
 def replay_post_fork(
@@ -113,14 +225,59 @@
 ) -> list[dict]:
     """Replay turns fork_turn..end with rebuilt_context as session start.
 
-    Returns one dict per eval_turn with keys:
-      {"turn_idx": int, "expected": str, "actual": str, "matched": bool}.
+    **Stub until the model-invocation client lands** (#592 commit-2 / -3).
+    Returns one placeholder dict per `case.eval_turns`:
 
-    TODO: wire to a Claude API client once we decide on the eval
-    invocation surface. Initially can be agent-API-driven; later may
-    move to a Claude-Code-harness-equivalent for higher fidelity.
+        {"turn_idx": int, "expected": str, "actual": "",
+         "matched": False, "reason": REPLAY_PENDING_REASON}
+
+    `expected` is pulled from the captured transcript's `text` field at
+    the eval-turn line index (1-indexed via the file order, matching the
+    fork_turn convention). When the line is missing or malformed the
+    expected text falls back to "" so the harness still produces a
+    coherent record.
+
+    Threading this stub keeps `run_one` end-to-end runnable: `score_fidelity`
+    reads `matched=False` and returns 0.0; `failures` populates with a
+    machine-readable `reason` so threshold-sweep / budget-sweep modes
+    surface latency + token-cost numbers without crashing on the
+    fidelity step. The real model client lands in a follow-up commit.
     """
-    raise NotImplementedError("Wire to model invocation client")
+    expected_by_idx: dict[int, str] = {}
+    if case.eval_turns:
+        wanted = set(case.eval_turns)
+        try:
+            with case.path.open("r", encoding="utf-8") as f:
+                for i, line in enumerate(f):
+                    if i not in wanted:
+                        continue
+                    line = line.strip()
+                    if not line:
+                        continue
+                    try:
+                        obj = json.loads(line)
+                    except json.JSONDecodeError:
+                        continue
+                    if isinstance(obj, dict):
+                        t = obj.get("text")
+                        if isinstance(t, str):
+                            expected_by_idx[i] = t
+        except OSError:
+            # Unreadable file → return an empty list rather than partial
+            # placeholders. The harness's load_corpus already filtered
+            # this case out for the corpus-walking path; this guard is
+            # for direct-call use.
+            return []
+    return [
+        {
+            "turn_idx": idx,
+            "expected": expected_by_idx.get(idx, ""),
+            "actual": "",
+            "matched": False,
+            "reason": REPLAY_PENDING_REASON,
+        }
+        for idx in case.eval_turns
+    ]
 
 
 def score_fidelity(replay_results: list[dict]) -> float:
@@ -136,16 +293,56 @@
     return matched / len(replay_results)
 
 
+def _pre_clear_text(case: TranscriptCase) -> str:
+    """Concatenate the `text` fields of every turn in 0..fork_turn-1.
+
+    This is the harness's stand-in for "the context the agent had open
+    just before the clear" — a deterministic, model-agnostic measure
+    that doesn't require replaying the transcript through a model.
+    """
+    parts: list[str] = []
+    if case.fork_turn <= 0:
+        return ""
+    with case.path.open("r", encoding="utf-8") as f:
+        for i, line in enumerate(f):
+            if i >= case.fork_turn:
+                break
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                obj = json.loads(line)
+            except json.JSONDecodeError:
+                continue
+            if isinstance(obj, dict):
+                t = obj.get("text")
+                if isinstance(t, str):
+                    parts.append(t)
+    return "\n".join(parts)
+
+
 def measure_token_cost(
     rebuilt_context: str,
     case: TranscriptCase,
 ) -> float:
-    """Rebuild block size / pre-clear context size.
-
-    TODO: needs a tokenizer (tiktoken or model-specific) and the
-    pre-clear context size from the captured transcript.
+    """Rebuild-block tokens / pre-clear-context tokens.
+
+    Both sides use `benchmarks.context_rebuilder.measure.estimate_tokens`
+    — the 4-chars-per-token heuristic that mirrors the rebuilder's own
+    internal estimator (`aelfrice.context_rebuilder._CHARS_PER_TOKEN`).
+    Sharing the constant keeps harness measurements aligned with the
+    rebuilder's own budget bookkeeping. A real tokenizer (tiktoken /
+    sentencepiece) would land alongside the LLM-judge in a follow-up.
+
+    Returns 0.0 when the pre-clear text is empty (no transcript content
+    pre-fork) — there's no meaningful ratio to report and division would
+    fail; 0.0 documents the corner without raising.
     """
-    raise NotImplementedError("Wire to tokenizer + transcript size")
+    pre_clear_tokens = estimate_tokens(_pre_clear_text(case))
+    if pre_clear_tokens <= 0:
+        return 0.0
+    rebuilt_tokens = estimate_tokens(rebuilt_context)
+    return rebuilt_tokens / pre_clear_tokens
 
 
 def run_one(

benchmarks/context-rebuilder/fixtures/synthetic/debugging_session_001.meta.json

@@ -0,0 +1,6 @@
+{
+  "task_type": "debug",
+  "fork_turn": 8,
+  "eval_turns": [8, 10, 12, 14],
+  "notes": "Midpoint fork on the 16-turn synthetic debugging-session fixture. Post-fork eval turns are the four user prompts at file indices 8, 10, 12, 14. Pairs with debugging_session_001.jsonl; load_corpus() resolves this via with_suffix('.meta.json')."
+}