feat(eval-harness): replay_post_fork host-agent JSONL split (closes #600)

CHANGELOG.md

@@ -44,8 +44,9 @@ installable release; see the roadmap in [README.md](README.md).
 
 ### Added
 
-- **`aelf doctor --prune-dormant` interactively deletes idle per-project DBs** ([#594](https://github.com/robotrocketscience/aelfrice/issues/594)). Companion to the #589 legacy-schema nag: scans `~/.aelfrice/projects/*/memory.db` for files whose mtime is older than `--idle-days` (default 30) and lists them with row count + size + idle days. Default action is dry-run; `--apply` prompts `[y/N]` per DB and deletes only on `y`/`yes`. There is no `--yes` shortcut — every deletion is per-DB and explicit, locking the issue body's "never silent delete" constraint. Detection is schema-agnostic (legacy and modern DBs are both pruneable when idle), and empty DBs / unreadable schemas surface with `row_count=0` rather than being skipped — an idle empty DB is the cleanest prune target. The filesystem-orphan detection mode listed in the issue body (project root no longer exists) is deferred: `project_warm.py` hashes the git-common-dir to a 12-char slug and stores no reverse mapping, so rebuilding `id → original-path` is a separate piece of work. `--projects-dir PATH` overrides the scan root for tests.
+- **Eval-harness host-agent replay path: `replay_post_fork` writes/joins per-run JSONL** ([#600](https://github.com/robotrocketscience/aelfrice/issues/600)). Commit-2 of #592, redesigned away from a direct vendor-SDK dependency. `replay_post_fork` gains an optional `run_dir: Path | None` parameter; threading via the new `--run-dir` argparse flag on `eval_harness.py` propagates it through `run_one`, `sweep_thresholds`, and `sweep_budgets`. With `run_dir` set, the harness writes one JSON row per eval_turn (`{turn_idx, rebuilt_block, user_turn, expected}`) to `<run_dir>/<case_stem>__t<threshold>__b<budget>/replay_requests.jsonl` and reads `replay_responses.jsonl` (if present) on the next invocation, joining by `turn_idx`. Substring-match (`expected.lower() in actual.lower()`) short-circuits to `matched=True`; filled rows without substring match drop to `reason="needs_llm_judge"` for commit-3 of #592. Missing or empty `actual` rows hold `reason="pending_replay"`. Without `--run-dir` the legacy `needs_replay_client` placeholder mode stays default — sweeps continue to produce valid latency + token-cost numbers without dispatching child tasks. `benchmarks/context-rebuilder/README.md` documents the operator flow (sweep → host dispatches one child task per request row → write responses → re-run sweep). aelfrice still imports zero from any vendor SDK in this surface; the model invocation is the host's responsibility, mirroring the `/aelf:onboard` polymorphic split. Five new tests (`test_replay_pending_round_trip`, etc.) cover the request-emit, response-join, partial-coverage, and per-(case, config) subdir behaviour; existing 15 wiring tests pass unchanged.
 
+- **`aelf doctor --prune-dormant` interactively deletes idle per-project DBs** ([#594](https://github.com/robotrocketscience/aelfrice/issues/594)). Companion to the #589 legacy-schema nag: scans `~/.aelfrice/projects/*/memory.db` for files whose mtime is older than `--idle-days` (default 30) and lists them with row count + size + idle days. Default action is dry-run; `--apply` prompts `[y/N]` per DB and deletes only on `y`/`yes`. There is no `--yes` shortcut — every deletion is per-DB and explicit, locking the issue body's "never silent delete" constraint. Detection is schema-agnostic (legacy and modern DBs are both pruneable when idle), and empty DBs / unreadable schemas surface with `row_count=0` rather than being skipped — an idle empty DB is the cleanest prune target. The filesystem-orphan detection mode listed in the issue body (project root no longer exists) is deferred: `project_warm.py` hashes the git-common-dir to a 12-char slug and stores no reverse mapping, so rebuilding `id → original-path` is a separate piece of work. `--projects-dir PATH` overrides the scan root for tests.
 - **`aelf doctor` flags per-project DBs on pre-v1.x schema (no `origin` column)** ([#589](https://github.com/robotrocketscience/aelfrice/issues/589)). Scans all `~/.aelfrice/projects/*/memory.db` and appends a `legacy-schema per-project DBs detected` block listing each flagged DB with belief row count and idle days. DBs on the old schema cannot participate in the v2.x lifecycle (`agent_remembered`, `user_validated`, calibrated weights, `aelf:promote`). Block is quiet when no legacy DBs are found (parity with #557 quietness). `aelf migrate` is now visible in `--help` output (previously `argparse.SUPPRESS`) so the fix line in the nag block is discoverable.
 
 - **Context-rebuilder eval harness — opt-in LLM-judge stage for open-ended turns** ([#592](https://github.com/robotrocketscience/aelfrice/issues/592)). Commit-3 of #592. New `benchmarks/context-rebuilder/judges/llm_judge.py` adds a host-polymorphic judge that scores open-ended replay rows (those tagged `reason="needs_llm_judge"` by the deterministic substring path). `write_judge_requests(rows, run_dir, max_judge_calls=N)` writes `judge_requests.jsonl` carrying strictly `(turn_idx, expected, actual)` — the rebuilt block and user turn are deliberately excluded per `docs/BENCHMARKS.md` Pass 1 / Pass 2 separation; letting the judge see retrieval context would inflate fidelity. Default `max_judge_calls=0` disables the stage so CI runs are free and outbound model traffic is opt-in. The judge tier label is `"anchor"` — the operator maps this to their host CLI's calibration-baseline tier (the tier the prior Cohen's-κ disagreement methodology was measured at); a cheaper-tier knob is intentionally not exposed because cheaper judges weaken comparability on the open-ended turns this stage exists to score. Dispatch is operator-driven: the host CLI reads the request file, issues one off-band model call per row at the anchor tier with `JUDGE_PROMPT_TEMPLATE`, writes `judge_responses.jsonl`. `read_judge_responses` + `apply_judge_verdicts` then fold verdicts back into the replay rows (clearing `reason`, setting `matched`, adding `judge_rationale`). No provider SDK import; matches `/aelf:onboard`'s polymorphic-classify pattern. 15 deterministic tests in `tests/test_context_rebuilder_eval_judge.py` cover the contamination boundary, cost-cap binding, response round-trip, malformed-line skip, fold-verdicts purity, and the no-SDK guard. Harness-side wiring (so the stage rides on `eval_harness.py --run-dir` rather than the standalone helpers) joins onto PR #601's run-dir plumbing in a follow-up.

benchmarks/context-rebuilder/README.md

@@ -277,6 +277,79 @@ These remain open for the fidelity scorer (#138):
 4. Whether augment-mode loses fidelity vs. suppress-mode (matters
    for v2.x suppress-mode promotion decision).
 
+## Host-agent eval-replay (#600)
+
+`replay_post_fork` participates in a polymorphic split that mirrors
+the `/aelf:onboard` pattern: aelfrice never imports the
+model-vendor SDK, never holds API keys, and pushes the model
+invocation to the calling host (an MCP-enabled host CLI, an
+operator-driven loop, or a private skill).
+
+### The flow
+
+1. **Sweep with `--run-dir`** writes per-(case, config) request
+   files to disk and skips the model call:
+
+   ```
+   uv run python benchmarks/context-rebuilder/eval_harness.py \
+       --mode threshold-sweep \
+       --corpus benchmarks/context-rebuilder/fixtures/synthetic/ \
+       --out /tmp/sweep1.json \
+       --run-dir /tmp/sweep1/
+   ```
+
+   For each (case, threshold, budget) cell, the harness writes:
+
+       /tmp/sweep1/<case_stem>__t<threshold>__b<budget>/replay_requests.jsonl
+
+   One JSON row per `eval_turn`:
+
+   ```json
+   {"turn_idx": 8, "rebuilt_block": "...", "user_turn": "...", "expected": "..."}
+   ```
+
+   `user_turn` is the most-recent `role=user` text at-or-before the
+   eval index — the prompt the host will replay through the rebuilt
+   context. All rows in the harness JSON output carry
+   `reason=pending_replay`; `score_fidelity` returns 0 on this
+   pass.
+
+2. **Operator dispatches one child task per row.** From a host
+   session (an MCP-enabled host CLI, an operator-driven loop, or a
+   private replay-eval skill), iterate over each
+   `replay_requests.jsonl` line and dispatch a child task prompted
+   with `rebuilt_block + "\n---\n" + user_turn`. Capture the
+   reply as `actual` and append a JSON line to
+   `replay_responses.jsonl` in the same directory:
+
+   ```json
+   {"turn_idx": 8, "actual": "..."}
+   ```
+
+   The dispatch step is operator-driven, not aelfrice. A private
+   `aelf:replay-eval` skill in `~/.claude/skills/` can automate the
+   for-loop; the contract is the on-disk request/response files.
+
+3. **Re-run the sweep**, same flags. `replay_post_fork` reads each
+   `replay_responses.jsonl`, joins by `turn_idx`, and:
+
+   * If the response is missing or `actual` is empty:
+     `matched=False`, `reason=pending_replay` (unchanged from pass 1).
+   * If `expected.lower() in actual.lower()`: `matched=True`,
+     `reason=""` (substring half of the fidelity verdict).
+   * Otherwise: `matched=False`, `reason=needs_llm_judge` —
+     open-ended turns parked for commit-3 of #592 (LLM judge).
+
+   `score_fidelity` reports the substring half of the verdict on
+   this pass.
+
+### Without `--run-dir`
+
+The legacy stub path stays default: every row holds
+`reason=needs_replay_client`, `matched=False`, the `actual` field
+is empty, and `score_fidelity` returns 0. Threshold/budget sweeps
+still produce valid latency + token-cost numbers.
+
 ## LLM-judge stage (commit-3 of #592)
 
 Open-ended replay rows that the deterministic substring scorer