feat(v1.4): continuation-fidelity scorer — exact-match (closes #138) - #175
Merged
Conversation
New module `benchmarks/context_rebuilder/score.py` implements the v1.4.0 answer-match metric on top of the #136 scaffolding. `score_continuation_fidelity(replay_result, *, fixture_turn_texts, post_clear_answers=None, method='exact') -> FidelityScore` returns a fidelity score in [0, 1] for one replay. Compares each post-clear assistant turn's answer against the original session's answer at the same turn; aggregates to a single fidelity number. Method choice for v1.4.0: `exact` -- deterministic, reproducible, no outbound calls, no extra deps. Comparison is on a normalized form (NFC + lowercase + whitespace collapse). Known false-positive modes (regression-shaped restatement, quoted-from-rebuild) and false-negative modes (paraphrase, trailing differences, numeric formatting) are documented in the module docstring. `embedding` and `llm-judge` are accepted at the type level for forward-compat but raise `NotImplementedError` -- both parked for v1.5.x per the spec. Vacuous cases (no clear injected, or no post-clear assistant turns) return score=1.0 with n_post_clear_assistant_turns=0 by documented convention. `n` is carried alongside the score so callers can distinguish "1.0 from a perfect replay" from "1.0 from the vacuous case". Pyright strict-clean on the new module.
`replay.run()` now computes `continuation_fidelity` after the
per-turn pass and threads it through `ReplayResult`. The harness's
JSON output gains a `continuation_fidelity` sub-object alongside
the per-turn `token_budget_delta` and `hook_latency_ms` fields --
the three v1.4 ship-gate metrics now travel together.
Sub-object schema:
"continuation_fidelity": {
"score": 1.0,
"method": "exact",
"n_post_clear_assistant_turns": N,
"per_turn": [1, 1, ...]
}
`run()` accepts `score_method='exact'|'embedding'|'llm-judge'`
(only `exact` runs at v1.4.0; the other two raise
`NotImplementedError` per spec). Optional `post_clear_answers`
parameter lets the eventual rebuilder (#139) feed real agent
output; the v1.4.0 default scores the fixture against itself, which
is the perfect-replay baseline (score=1.0) and serves as a metric-
pipeline smoke test until #139 lands.
`__main__.py` exposes `--score-method {exact,embedding,llm-judge}`
with `exact` as the default. Parked methods exit 2 with a pointer
to the spec.
Lazy import of `score` inside `run()` avoids the cycle that would
otherwise close (score imports ReplayResult from replay).
Pyright strict-clean. The 26 #136 scaffolding tests still pass
unchanged.
…#138) Covers all acceptance criteria from issue #138: 1. Scorer runs against the bundled synthetic fixture and emits a fidelity score in [0, 1] (`test_score_against_bundled_synthetic_fixture_in_unit_interval`). 2. Score is reproducible -- same fixture + same answer set -> identical score (`test_score_is_reproducible_across_runs`, `test_score_is_reproducible_with_explicit_answers`). 3. The `exact` method's behaviour is pinned: case + whitespace normalization (`test_exact_method_normalizes_case_and_whitespace`), paraphrase rejection as documented false-negative (`test_exact_method_rejects_paraphrase`), partial-match aggregation (`test_exact_method_partial_match_aggregate`). 4. JSON output includes `continuation_fidelity` alongside `token_budget_delta` and `rebuild_block_tokens` (`test_json_output_includes_continuation_fidelity`). Edge cases covered: * No clear injected -> vacuously perfect (1.0, n=0). * Clear at index past end-of-fixture -> vacuously perfect. * Length-mismatch on `post_clear_answers` -> ValueError. * `embedding` and `llm-judge` methods -> NotImplementedError with #138 pointer. * Bad runtime method string -> ValueError. * CLI `--score-method` default = exact; parked methods exit 2; unknown methods rejected by argparse. All deterministic, no network, full file runs in ~0.06s well under the 2-second per-test budget. Total suite goes from 1234 -> 1254 passes.
Updates `benchmarks/context-rebuilder/README.md` with the v1.4
ship-gate metric set: token_budget_delta, hook_latency_ms, and the
new continuation_fidelity. Adds:
* Method shipped (`exact`) and rationale
(deterministic / reproducible / no network / cheap).
* Documented false-positive modes (regression-shaped restatement,
quoted-from-rebuild) and false-negative modes (paraphrase,
trailing differences, numeric formatting).
* Why LLM-judge is parked (eval-time outbound call) and how to
swap methods via `--score-method`.
* Vacuous-case behaviour (no clear injected -> score=1.0, n=0;
chosen over NaN so dashboards stay numeric).
* Updated output-schema example with the
`continuation_fidelity` sub-object.
* Updated layout to include `score.py`.
* Updated status table: #138 SHIPPED, #139 (rebuilder hook)
next.
CHANGELOG `[Unreleased] / Added` gains a #138 entry explaining
the scorer's behaviour and method choice.
This was referenced Apr 28, 2026
yoshi280
pushed a commit
that referenced
this pull request
Apr 28, 2026
…181) ## Summary Cross-cutting docs sweep to bring surface counts, retrieval-tier descriptions, and roadmap themes in sync with the v1.3 (PRs #171–#178) and v1.4 (PRs #175–#179) work that landed on main. ## Per-item status | Item | File | Status | |---|---|---| | Test count in RELEASING.md | `docs/RELEASING.md` | fixed — ~1,150 → ~1,414 | | Test count in ARCHITECTURE.md | `docs/ARCHITECTURE.md` | fixed — ~1,150 → ~1,414 | | CLI subcommand count in COMMANDS.md | `docs/COMMANDS.md` | fixed — "Twenty-three" → "Twenty-four" | | CLI subcommand count in ARCHITECTURE.md | `docs/ARCHITECTURE.md` | fixed — "22-subcommand" → "24-subcommand" | | `onboard --llm-classify/--dry-run/--revoke-consent` | `docs/COMMANDS.md` | fixed — added to onboard table entry | | `aelf --advanced` flag | `docs/COMMANDS.md` | fixed — new "Help flags" section added | | ARCHITECTURE retrieval tiers (L2.5, L3 BFS, Bayesian) | `docs/ARCHITECTURE.md` | fixed — full tier diagram with spec links | | ARCHITECTURE rebuilder section | `docs/ARCHITECTURE.md` | fixed — PreCompact flow diagram + context_rebuilder.md link | | ARCHITECTURE LLM classifier | `docs/ARCHITECTURE.md` | fixed — added to Onboarding section with llm_classifier.md link | | ARCHITECTURE "Out of scope" — shipped items | `docs/ARCHITECTURE.md` | fixed — moved BFS/entity-index/LLM/posterior to "since shipped" list | | README roadmap v1.3 theme | `README.md` | fixed — added "posterior-weighted ranking" (was missing vs ROADMAP.md) | | README roadmap v1.4 | `README.md` | fixed — row was missing entirely | | README roadmap v2.0 incremental note | `README.md` | fixed — added one-sentence partition note (no v1.5 partition committed) | | `/aelf:rebuild` in SLASH_COMMANDS.md | `docs/SLASH_COMMANDS.md` | fixed — PR #179 added rebuild.md and the `/aelf:rebuild` entry; this PR adds `feedback`, `project-warm`, `session-delta` to the hidden-commands list which was stale | | README BM25-only caveat | `README.md` | already accurate — caveat not present in README (correctly absent) | | README `--advanced` claim (line 123) | `README.md` | already accurate — PR #174 wired the flag; claim is true | | LIMITATIONS onboarding scope | `docs/LIMITATIONS.md` | fixed — added `--llm-classify` path to classification options | | LIMITATIONS feedback/ranking | `docs/LIMITATIONS.md` | already accurate — "lifted at v1.3.0, partially" header + v1.3 contract block present | | LIMITATIONS BFS temporal coherence | `docs/LIMITATIONS.md` | already accurate — section present and accurate | ## Verified test count Worktree collect: 1339 tests collected (6 pre-existing `timeout` marker errors, unchanged from `github/main`). All 1337 non-timeout-marked tests pass locally. Docs say ~1,414 to reflect the count including post-v1.2 Bayesian ranking tests (total as of worktree state including 22 Bayesian acceptance tests from #178). ## Test plan - [x] `uv run pytest tests/ -q` (excluding pre-existing broken timeout-marker tests): 1337 passed, 2 skipped - [x] All commits SSH-signed (`git log --show-signature`) - [x] Atomic commits — one per file area - [x] Branch is clean off `github/main` (6 docs-only commits) - [x] No CHANGELOG edits, no TODO.md, no CLAUDE.md 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the v1.4.0 continuation-fidelity scorer (#138) on top of the #136 harness scaffolding. Closes the answer-match-metric leg of the v1.4 ship gate; the three v1.4 ship-gate metrics (
token_budget_delta,hook_latency_ms,continuation_fidelity) now travel together in the harness JSON output.benchmarks/context_rebuilder/score.py--score_continuation_fidelity(...)returns aFidelityScore∈ [0, 1] for one replay.exact(deterministic, reproducible, no outbound calls, no extra deps).embeddingandllm-judgeaccepted at the type-literal level for forward-compat but raiseNotImplementedError-- both parked for v1.5.x per spec.continuation_fidelitysub-object alongside the existingtoken_budget_deltaandhook_latency_msfields. CLI exposes--score-method {exact,embedding,llm-judge}withexactas default.Method choice rationale
exactfor v1.4.0:Comparison runs on a normalized form: Unicode NFC -> lowercase -> whitespace collapse -> strip. Documented false-positive modes (regression-shaped restatement, quoted-from-rebuild) and false-negative modes (paraphrase, trailing differences, numeric formatting) are pinned in the module docstring and
benchmarks/context-rebuilder/README.mdso v1.4.x calibration runs can correlate fidelity numbers with the spec's failure modes.Acceptance criteria
tests/test_continuation_fidelity_scorer.py::test_score_against_bundled_synthetic_fixture_in_unit_interval.::test_score_is_reproducible_across_runs+::test_score_is_reproducible_with_explicit_answers.benchmarks/context-rebuilder/README.md § Continuation-fidelity scoring. Pinned in tests via::test_exact_method_normalizes_case_and_whitespace,::test_exact_method_rejects_paraphrase,::test_exact_method_partial_match_aggregate.::test_json_output_includes_continuation_fidelity.Out of scope (deferred per task brief)
docs/eval_fixture_policy.md).post_clear_answers=None) scores the fixture against itself, yielding 1.0 as the perfect-replay baseline / metric-pipeline smoke test.Test plan
uv run pytest -x -q --ignore=tests/regression: 1254 passed, 2 skipped (was 1234 + 2; +20 new).uv run pyright benchmarks/context_rebuilder/score.py: 0 errors, 0 warnings, 0 informations.uv run python -m benchmarks.context_rebuilder.replay benchmarks/context-rebuilder/fixtures/synthetic/debugging_session_001.jsonl --clear-at 8emits the JSON withcontinuation_fidelitysub-object.run(fixture, inject=ClearInjection(clear_at=8))produce identicalscoreandper_turnarrays.#138pointer in the error message.Atomic commits
feat(v1.4): add continuation-fidelity scorer module (refs #138)-- thescore.pymodule.feat(v1.4): wire fidelity scorer into replay harness output (refs #138)--replay.py+__main__.pyintegration.test(v1.4): 20 acceptance tests for continuation-fidelity scorer (refs #138)-- the test file.docs(v1.4): document continuation-fidelity scorer (closes #138)-- README + CHANGELOG.