feat(B-0170.4): seed existence-drift fixture + regression test#3624
Merged
Conversation
Smallest safe slice of B-0170.4 (fixture-tests + eval-set coverage). Extends PR #3611's count-drift seed to the existence-drift sub-class — the second of the 5 shipped check-types now has empirical-axis regression coverage. - New `tools/substrate-claim-checker/fixtures/existence-drift-missing-doc.md` fixture modeling the verify-then-claim memo's body table instance #8 (PR #1252 — future-domain memo referenced a docs/ markdown file that didn't actually exist). Uses a clearly synthetic path so the fixture stays stable across substrate evolution. - New `fixtures.test.ts` describe block asserting `check-existence.ts` emits exactly one drift finding at line 24 with severity "drift". - `fixtures/README.md` index gains the new fixture row. Discipline carried forward from PR #3611 review threads (chatgpt-codex-connector + copilot-pull-request-reviewer): the HTML provenance comment intentionally does NOT backtick-quote the exact fixture path. Restating the claim inside the comment would let regressions in body-claim detection slip past the test via an HTML-comment match. The test asserts exact finding count + pins the body line as the catch. Focused checks: - `bun tools/substrate-claim-checker/check-existence.ts <fixture>` — 1 drift finding at line 24, severity "drift", exit 1 - `bun test tools/substrate-claim-checker/fixtures.test.ts` — 2 pass, 12 expect() calls, exit 0 - `bun test tools/substrate-claim-checker/` (full suite) — 114 pass, 0 fail, 256 expect() calls (negative-path stderr lines are intentional error-handling cases per PR #3611 convention) Composes with: - B-0170.4 done-criteria ("fixture-tests + eval-set coverage for all shipped + new check-types") — incremental progress, one sub-class per slice per the fixtures/README.md procedure - B-0170 (parent row, decomposed) - PR #3611 (count-drift seed; same scaffolding extended here) Claim: 6c253d24-3ed0-4e89-8f3a-563b13f933cc (otto-cli, B-0170). operative-authorization: aaron 2026-05-14: "- **Devil-pole** (edge-runner drive): keep pushing, discover, go hard, never-be-idle" Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Extends the substrate-claim-checker eval-set (started in PR #3611) with the second drift sub-class — existence drift. Adds a synthetic-path fixture, a regression test asserting check-existence.ts emits exactly one drift finding at the expected line, and an index row in the fixtures README.
Changes:
- New
existence-drift-missing-doc.mdfixture using a clearly syntheticdocs/_fixture_existence_drift_target_b0170_2026_05_15.mdpath, with HTML provenance comment that deliberately avoids backtick-quoting the path - New
describeblock infixtures.test.tsasserting exactly one finding at line 24 with severity"drift" fixtures/README.mdindex gains a row for the new fixture
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| tools/substrate-claim-checker/fixtures/README.md | Adds index row for the new existence-drift fixture |
| tools/substrate-claim-checker/fixtures/existence-drift-missing-doc.md | New fixture modeling PR #1252 existence-drift instance #8 with synthetic path |
| tools/substrate-claim-checker/fixtures.test.ts | Adds regression test for check-existence.ts against the new fixture |
This was referenced May 16, 2026
AceHack
added a commit
that referenced
this pull request
May 16, 2026
* feat(B-0170.4): seed path-form-drift fixture + regression test
Adds the third eval-set fixture for B-0170.4, extending regression
coverage from {count, existence} to {count, existence, path-form}.
Same proven shape as PR #3611 (count) and PR #3624 (existence): a
small on-disk markdown file under fixtures/ plus a pinned-expectation
test in fixtures.test.ts.
The fixture references tools/substrate-claim-checker/check-counts.ts
as both a bare basename (`check-counts.ts`) and a fully-qualified
path. Both resolve to the same absolute file via check-path-forms.ts's
3-root strategy (fileDir / parentDir / repoRoot), so the drift is
deterministically detected without depending on synthetic files.
Per PR #3611 review-thread discipline (chatgpt-codex-connector +
copilot): pin exact finding count (1) AND exact body-claim line (28)
so a regression in body-claim detection cannot be silently masked by
an HTML-comment-side match. The provenance comment intentionally
avoids restating either path form.
Re-decomposition note: original B-0170 lists B-0170.1-.4 as children.
B-0170.1 (semantic-equivalence) has an in-flight branch already;
B-0170.2 / .3 introduce brand-new sub-class checkers (bigger slices).
Adding one more fixture under B-0170.4 is genuinely the smallest safe
slice — it extends the proven pattern, has no merge risk, and closes
one more line of the parent row's done-criteria (eval-set coverage).
Focused check: bun test tools/substrate-claim-checker/fixtures.test.ts
→ 3 pass, 0 fail, 17 expect() calls. Full suite: 115 pass, 0 fail.
operative-authorization: aaron 2026-05-14: "- **Devil-pole**
(edge-runner drive): keep pushing, discover, go hard, never-be-idle"
Co-Authored-By: Claude <noreply@anthropic.com>
* shard(tick): 2026-05-16T02:56Z — B-0170.4 path-form fixture slice (PR #3696)
Per-tick shard documenting the path-form-drift fixture slice landed
in PR #3696. Captures the re-decomp reasoning (B-0170.1 has in-flight
branch; B-0170.2/.3 are bigger slices; B-0170.4 fixture continuation
is smallest safe), the subdecimal-vs-top-level scheme discipline
observed (per ac9d9a4 rule), the focused-check outcome, and the
catch-43 cron sentinel re-arm at session start.
operative-authorization: aaron 2026-05-14: "- **Devil-pole**
(edge-runner drive): keep pushing, discover, go hard, never-be-idle"
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(B-0170.4): correct anchor citation to memo instance #15 / PR #1256
Per Copilot review threads on PR #3696: the path-form fixture's anchor
was cited as "taxonomy row 4" but path-form is actually instance #15
of the verify-then-claim memo's body table (PR #1256), and sub-class
#6 of the 7-class list. Corrects the README index + adds the historical
anchor comment in the test.
The current fixture remains a synthetic exemplar covering the sub-class;
instance #15's literal substance (adjacent ADR citations from PR #1256)
is queued as follow-on fixture B-0170.4.1 per the per-thread plan.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
This was referenced May 16, 2026
AceHack
added a commit
that referenced
this pull request
May 17, 2026
Adds the 5th eval-set fixture for the substrate-claim-checker, covering the convention sub-class of the 7-class verify-then-claim taxonomy. The fixture pair (current ADR + sibling predecessor ADR support file) makes the broken half of the bidirectional ADR supersession convention reproducible without depending on any real ADR pair in the repo. Anchor: PR #2512 (the PR that shipped check-convention.ts) — synthetic exemplar, same shape as the path-form-drift fixture's synthetic case. Focused check outcomes: - bun test tools/substrate-claim-checker/fixtures.test.ts → 5 pass / 0 fail - bun test tools/substrate-claim-checker/ → 117 pass / 0 fail - Direct CLI run reports 1 convention-drift finding on line 36 with the expected reciprocal-marker reason string Composes with B-0170 (parent), B-0170.4 eval-set thread (PRs #3611, #3624, #3696, #3749). operative-authorization: aaron 2026-05-14: "- **Devil-pole** (edge-runner drive): keep pushing, discover, go hard, never-be-idle" Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Smallest safe slice of B-0170.4 (fixture-tests + eval-set coverage). Extends PR #3611's count-drift seed to the existence-drift sub-class — the second of the 5 shipped check-types now has empirical-axis regression coverage.
tools/substrate-claim-checker/fixtures/existence-drift-missing-doc.mdfixture modeling the verify-then-claim memo's body-table instance Round 32 — CI parity-swap + persona memory normalization + v1 security scope #8 (PR free-memory: multi-harness alignment/convergence on design as future skill domain (Aaron 2026-05-03) #1252 — future-domain memo referenced adocs/markdown file that didn't actually exist).fixtures.test.tsassertingcheck-existence.tsemits exactly one drift finding at line 24 with severity"drift".fixtures/README.mdindex gains the new fixture row.Why the fixture path is synthetic
The fixture cites
docs/_fixture_existence_drift_target_b0170_2026_05_15.md— a clearly synthetic path no human or agent would create accidentally. This tests the empirical detection pattern, not a frozen substrate snapshot — substrate evolution can't break the regression by creating an unrelated file at the historical path.Discipline carried forward from PR #3611 review threads
chatgpt-codex-connector+copilot-pull-request-reviewerflagged a hazard on the count-drift fixture: restating the claim inside the HTML provenance comment let the test pass via the comment match alone, masking regressions in body-claim detection. Same discipline applied here:findings.length === 1(exact, not>= 1)finding.line === 24so a regression that stops detecting the body claim can't be maskedTest plan
bun tools/substrate-claim-checker/check-existence.ts <fixture>— 1 drift finding at line 24, severity"drift", exit 1bun test tools/substrate-claim-checker/fixtures.test.ts— 2 pass, 12 expect() calls, exit 0bun test tools/substrate-claim-checker/(full suite) — 114 pass, 0 fail, 256 expect() calls (negative-path stderr lines are intentional error-handling cases per PR feat(B-0170.4): seed eval-set fixture for count-drift regression coverage #3611 convention).claude/rules/codeql-no-source-on-docs-only-pr-is-broken-commit-canary.md—tools/substrate-claim-checker/tree size 13 at bothHEAD~1andHEADComposes with
fixtures/README.mdproceduredecomposed)memory/feedback_verify_then_claim_discipline_dominant_failure_mode_substrate_authoring_otto_2026_05_03.md— body-table instance Round 32 — CI parity-swap + persona memory normalization + v1 security scope #8 is the empirical anchor for this fixtureClaim
6c253d24-3ed0-4e89-8f3a-563b13f933cc(otto-cli, B-0170)operative-authorization: aaron 2026-05-14: "- Devil-pole (edge-runner drive): keep pushing, discover, go hard, never-be-idle"
🤖 Generated with Claude Code