feat(B-0170.4): seed convention-drift fixture + regression test#4085
Merged
Conversation
Adds the 5th eval-set fixture for the substrate-claim-checker, covering the convention sub-class of the 7-class verify-then-claim taxonomy. The fixture pair (current ADR + sibling predecessor ADR support file) makes the broken half of the bidirectional ADR supersession convention reproducible without depending on any real ADR pair in the repo. Anchor: PR #2512 (the PR that shipped check-convention.ts) — synthetic exemplar, same shape as the path-form-drift fixture's synthetic case. Focused check outcomes: - bun test tools/substrate-claim-checker/fixtures.test.ts → 5 pass / 0 fail - bun test tools/substrate-claim-checker/ → 117 pass / 0 fail - Direct CLI run reports 1 convention-drift finding on line 36 with the expected reciprocal-marker reason string Composes with B-0170 (parent), B-0170.4 eval-set thread (PRs #3611, #3624, #3696, #3749). operative-authorization: aaron 2026-05-14: "- **Devil-pole** (edge-runner drive): keep pushing, discover, go hard, never-be-idle" Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds the fifth empirical eval-set fixture to tools/substrate-claim-checker, covering the convention drift checker (ADR supersession reciprocity) and wiring a regression test to ensure the checker continues to flag the missing reciprocal “Superseded by” marker pattern.
Changes:
- Added a self-contained convention-drift fixture pair (current ADR + predecessor support ADR) under
tools/substrate-claim-checker/fixtures/. - Extended
fixtures.test.tswith a regression test that pins exact finding shape (count, claim line, target path, reason substrings). - Updated the fixtures index README to include the new convention-drift fixture row.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| tools/substrate-claim-checker/fixtures/README.md | Adds the convention-drift fixture row to the eval-set index. |
| tools/substrate-claim-checker/fixtures/convention-drift-no-reciprocal-marker.md | New “current ADR” fixture containing a Supersedes ADR \...`` body claim. |
| tools/substrate-claim-checker/fixtures/_convention-drift-target-adr.md | New predecessor ADR support file intentionally missing the reciprocal “Superseded by” marker. |
| tools/substrate-claim-checker/fixtures.test.ts | Adds a convention-drift regression test that asserts exact finding count/line/target/reason substrings. |
This was referenced May 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
5th eval-set fixture for the substrate-claim-checker. Covers the convention sub-class of the 7-class verify-then-claim taxonomy via a self-contained fixture pair (current ADR + sibling predecessor ADR support file). Synthetic exemplar; anchor PR #2512 (the PR that shipped
check-convention.ts).Continues the B-0170.4 eval-set thread:
Remaining check-types for B-0170.4 thread completion: semantic-equivalence-drift, empirical-output-drift, self-recursive-drift (none yet shipped as checkers).
Files
tools/substrate-claim-checker/fixtures/convention-drift-no-reciprocal-marker.md(new) — current ADR claiming supersessiontools/substrate-claim-checker/fixtures/_convention-drift-target-adr.md(new) — predecessor ADR support file (leading-underscore marks "fixture support, not a top-level fixture")tools/substrate-claim-checker/fixtures/README.md— fixture-index row addedtools/substrate-claim-checker/fixtures.test.ts— regression test added; pins exact finding count (1), claim line (36), target path, and reason substring ("not reciprocated","Superseded by")The fixture pair is self-contained:
check-convention.ts's 3-root resolution (fileDir / parentDir / repoRoot) finds the support file viafileDirwithout dragging in any real repo ADR pair.Per PR #3611 review-thread discipline, the fixture's HTML-comment provenance intentionally avoids restating the literal
Supersedes ADR ...claim so a regression in body-claim detection cannot be masked by a comment-side match.Focused checks
bun test tools/substrate-claim-checker/fixtures.test.ts→ 5 pass / 0 fail / 30 expectsbun test tools/substrate-claim-checker/(full suite) → 117 pass / 0 fail / 274 expects (6 files)bun tools/substrate-claim-checker/check-convention.ts tools/substrate-claim-checker/fixtures/convention-drift-no-reciprocal-marker.md→ exits 1 with1 convention-drift finding(s)on line 36 with reciprocal-marker reason stringgit ls-tree HEAD= 53 entries (matchesorigin/main)Test plan
Composes with
🤖 Generated with Claude Code