Repository navigation
Record equivalent mutants by what they sit inside, and catch drift on every commit #2037
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 7 commits
768ec33
7e24f3a
ba9e5d3
36509a2
3397c3c
91492fa
2b01588
0b4616c
754f365
583ed06
37f70db
7bbd57f
a2ac3fd
f132c51
ba55141
74f7940
a82c6a1
368b203
cb189da
17c85b5
478e085
1d6ac5c
57a0b86
3b6977a
c6276ce
a1d33ae
b4b0b63
9184de7
9ea6e2a
c37fae9
50ab94b
de83423
28749a4
c0d11af
227c086
233d93d
a9f9fbd
f1182e6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| /** | ||
| * Check that every known-equivalent mutant entry still points at a real mutant. | ||
| * | ||
| * This is the cheap half of `mutation:audit-equivalents`: it only resolves | ||
| * entries against freshly generated mutants, running no lint, no type-check and | ||
| * no tests, so it is fast enough for `precommit`. Re-proving that each entry is | ||
| * still *equivalent* is the expensive half and stays an on-demand tool. | ||
| * | ||
| * It exists because drift used to be found in review. An entry that no longer | ||
| * resolves stops suppressing its mutant, and the next mutation run fails | ||
| * somewhere else entirely — on a branch that may not be the one that moved it. | ||
| */ | ||
|
|
||
| import { relative, resolve } from "@std/path"; | ||
| import { generateMutants } from "#scripts/mutation/generate.ts"; | ||
| import { | ||
| listRegistryFiles, | ||
| mutantKeyForPath, | ||
| parseRegistryText, | ||
| } from "#scripts/mutation/ignore.ts"; | ||
| import { projectRoot } from "#scripts/project-root.ts"; | ||
|
|
||
| interface Entry { | ||
| key: string; | ||
| registry: string; | ||
| sourcePath: string; | ||
| } | ||
|
|
||
| /** Every entry across the registry, with the file it was read from. */ | ||
| const readEntries = async (): Promise<Entry[]> => { | ||
| const entries: Entry[] = []; | ||
| for (const file of await listRegistryFiles()) { | ||
| const path = typeof file === "string" ? file : file.pathname; | ||
| const registry = relative(projectRoot, path); | ||
| for (const parsed of parseRegistryText( | ||
| await Deno.readTextFile(file), | ||
| file, | ||
| )) { | ||
| entries.push({ | ||
| key: parsed.key, | ||
| registry, | ||
| sourcePath: parsed.sourcePath, | ||
| }); | ||
| } | ||
| } | ||
| return entries; | ||
| }; | ||
|
|
||
| /** Every mutant key a source file can produce, under exhaustive generation so | ||
| * an entry for an exhaustive-only replacement still resolves. */ | ||
| const keysFor = async (sourcePath: string): Promise<Set<string>> => { | ||
| const file = resolve(projectRoot, sourcePath); | ||
| const content = await Deno.readTextFile(file); | ||
| return new Set( | ||
| generateMutants(content, file, true).map((mutant) => | ||
| mutantKeyForPath(sourcePath, mutant), | ||
| ), | ||
| ); | ||
| }; | ||
|
|
||
| export const checkEquivalentMutants = async (): Promise<string[]> => { | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
No test in this commit imports AGENTS.md reference: AGENTS.md:L74-L74 Useful? React with 👍 / 👎.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fair. Fixed in 37f70db and 7bbd57f. Adding the tests immediately proved the second half of your point: importing the module in-process pulled its Nine cases in Making it testable also removed two branches worth losing — the registry directory and project root are named by the caller now instead of defaulted, and the "which spelling is this path" branch went away because Generated by Claude Code |
||
| const entries = await readEntries(); | ||
| const problems: string[] = []; | ||
| const seen = new Set<string>(); | ||
| const byPath = new Map<string, Set<string>>(); | ||
| for (const sourcePath of new Set(entries.map((e) => e.sourcePath))) { | ||
| byPath.set(sourcePath, await keysFor(sourcePath)); | ||
| } | ||
| for (const entry of entries) { | ||
| if (seen.has(entry.key)) { | ||
| problems.push(`duplicate (${entry.registry}): ${entry.key}`); | ||
| } | ||
| seen.add(entry.key); | ||
| if (!byPath.get(entry.sourcePath)!.has(entry.key)) { | ||
| problems.push( | ||
| `stale (nothing to suppress — did it move or get renamed?) (${entry.registry}): ${entry.key}`, | ||
| ); | ||
| } | ||
| } | ||
| return problems; | ||
| }; | ||
|
|
||
| if (import.meta.main) { | ||
| const problems = await checkEquivalentMutants(); | ||
| if (problems.length === 0) { | ||
| console.log("Every equivalent-mutant entry still points at a real mutant."); | ||
| Deno.exit(0); | ||
| } | ||
| console.error(problems.join("\n")); | ||
| console.error( | ||
| `\n${problems.length} equivalent-mutant entries need attention. Each names the` + | ||
| "\nthing it sits inside; re-record it against where that code lives now, or" + | ||
| "\nremove it if the expression is gone.", | ||
| ); | ||
| Deno.exit(1); | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,146 @@ | ||
| /** | ||
| * Where a mutant sits, said in a way that survives an edit somewhere else. | ||
| * | ||
| * The equivalent-mutant registry used to record `path:line:column`, so any edit | ||
| * *above* a recorded expression silently invalidated its entry — the mutant | ||
| * stopped being suppressed and the audit failed, without the recorded | ||
| * expression having changed at all. That happened repeatedly, and was always | ||
| * found by a reviewer rather than by a check. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This module header describes the former line-and-column implementation, past review failures, and what the new code replaced rather than documenting only the current anchor contract; similar historical narratives were also added to the checker and ignore-list comments. Move that history to the commit message and keep only the present constraint, as the repository explicitly forbids old-implementation comparisons in source comments and requires comments to stay short. AGENTS.md reference: AGENTS.md:L71-L72 Useful? React with 👍 / 👎.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You're right, and I wrote those deliberately, which makes it worse. Fixed in 37f70db.
I kept one sentence that reads like history but is not: the note on Generated by Claude Code |
||
| * | ||
| * An anchor is the name of the thing the mutant sits inside — the function, | ||
| * method, or top-level value it belongs to — plus `@n` when several mutants of | ||
| * the same kind sit inside that same thing. The ordinal is written with `@` | ||
| * rather than `#` because a registry line ends with a `#` comment, and a `#` | ||
| * in the anchor would be read as the start of one. Adding a comment, an import, or a | ||
| * whole unrelated function moves no anchor. Renaming the function, or adding | ||
| * another mutant of that kind inside it, does — and both are real changes to | ||
| * the thing being recorded. | ||
| */ | ||
|
|
||
| /** Names of the declarations a mutant can sit inside, outermost first. */ | ||
| type NamePath = readonly string[]; | ||
|
|
||
| interface Span { | ||
| end: number; | ||
| start: number; | ||
| } | ||
|
|
||
| /** A node that might lend its name to whatever sits inside it. */ | ||
| interface NamedNode { | ||
| id?: { name?: string } | null; | ||
| key?: { name?: string; value?: unknown } | null; | ||
| type?: string; | ||
| } | ||
|
|
||
| /** The name a declaration contributes, or nothing when it is anonymous. Kept | ||
| * to the shapes that really carry a reader-recognisable name: an anonymous | ||
| * arrow inside a call gets its name from whatever it is assigned to instead. */ | ||
| const nameOf = (node: NamedNode): string | null => { | ||
| const named = node.id?.name ?? node.key?.name; | ||
| if (typeof named === "string" && named !== "") return named; | ||
| const literalKey = node.key?.value; | ||
| return typeof literalKey === "string" && literalKey !== "" | ||
| ? literalKey | ||
| : null; | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When object members use legal numeric keys, Oxc exposes Useful? React with 👍 / 👎.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Correct — Test: No registry entry moved, so nothing here rested on a numeric key today. That is the point of catching it now rather than the first time one appears. Generated by Claude Code |
||
| }; | ||
|
|
||
| /** Declarations whose name is worth carrying into the path. A block or an | ||
| * `if` contributes nothing: naming by them would make the anchor move when | ||
| * the surrounding code is merely re-nested. */ | ||
| const NAMING_TYPES = new Set([ | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Fresh evidence beyond the resolved property/enum/name threads is that Useful? React with 👍 / 👎.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in 9184de7. The regression test is your example: one function, two arms whose bodies are character-identical Every case label in Generated by Claude Code There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When anonymous callbacks are handed directly to different calls, such as Useful? React with 👍 / 👎.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed the collapse ( A callee genuinely is a name the code writes, unlike a ternary arm or a guessed discriminant. What sinks it is that it does not do the job. I ran the case that matters: Naming by callee turns those into Every other naming rule accepted here removes the ordinal outright, because a key, a prop, a case label and a parameter are each unique among their siblings by construction. A callee is not, and that difference is the whole argument. The measurement is in the ternary thread: 3 of 540 entries carry an ordinal, none in a call argument. The boundary and its reasoning are now at Generated by Claude Code |
||
| "ClassDeclaration", | ||
| "FunctionDeclaration", | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When named function or class expressions are used somewhere that has no property or variable name of its own, such as array entries or call arguments, this list does not collect their Useful? React with 👍 / 👎.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Correct. Both are in the test: Worth naming the pattern the last three of these share: a node carries a name in a position where nothing around it does. Generated by Claude Code |
||
| "MethodDefinition", | ||
| "PropertyDefinition", | ||
| "TSEnumDeclaration", | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When a TypeScript enum has repeated mutable initializers, this collects only the enum name, not the member name, so Useful? React with 👍 / 👎.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Right, same shape as the function-expression one. Test: Generated by Claude Code |
||
| "TSInterfaceDeclaration", | ||
| "TSModuleDeclaration", | ||
| "TSTypeAliasDeclaration", | ||
| "VariableDeclarator", | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When a function has two default parameter initializers with identical mutable text, such as a string-typed Useful? React with 👍 / 👎.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in 28749a4. A defaulted parameter now lends its name — a default is code of its own, and two parameters of one function can default to the same words — so your case gives This was live in the tree, and more so than any earlier naming finding: thirteen entries were sitting on the enclosing function alone, told apart only by order. Each moved onto the parameter it always sat in — Every one kept its fingerprint, and every new name extends the old one rather than replacing it — that pair of facts is what makes the re-anchoring checkable rather than a guess. The binding-pattern line still holds: a parameter that destructures ( Generated by Claude Code |
||
| ]); | ||
|
|
||
| const isSpan = (value: unknown): value is Span => | ||
| typeof value === "object" && | ||
| value !== null && | ||
| typeof (value as Span).start === "number" && | ||
| typeof (value as Span).end === "number"; | ||
|
|
||
| /** | ||
| * The names enclosing byte offset `offset`, outermost first. Walks down the | ||
| * tree following whichever child actually contains the offset, so the cost is | ||
| * the depth of the tree rather than its size. | ||
| */ | ||
| const namePathAt = (program: object, offset: number): NamePath => { | ||
| const names: string[] = []; | ||
| let node: object = program; | ||
| for (;;) { | ||
| const record = node as Record<string, unknown>; | ||
| if ( | ||
| typeof record.type === "string" && | ||
| NAMING_TYPES.has(record.type as string) | ||
| ) { | ||
| const named = nameOf(record as NamedNode); | ||
| if (named !== null) names.push(named); | ||
| } | ||
| const next = Object.values(record) | ||
| .flatMap((value) => (Array.isArray(value) ? value : [value])) | ||
| .find( | ||
| (value) => isSpan(value) && value.start <= offset && offset < value.end, | ||
| ); | ||
| if (!next) return names; | ||
| node = next as object; | ||
| } | ||
| }; | ||
|
|
||
| /** The anchor for a mutant at `offset`, without its ordinal. Top-level code | ||
| * belonging to no declaration anchors on the file itself, written `<file>`. */ | ||
| export const anchorNameAt = (program: object, offset: number): string => { | ||
| const names = namePathAt(program, offset); | ||
| return names.length === 0 ? "<file>" : names.join("."); | ||
| }; | ||
|
|
||
| export interface AnchoredMutant { | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
A repo-wide AGENTS.md reference: AGENTS.md:L81-L81 Useful? React with 👍 / 👎. |
||
| /** `name` plus `@n` when more than one of this kind shares the name. */ | ||
| anchor: string; | ||
| } | ||
|
|
||
| interface HasLocation { | ||
| newOperator: string; | ||
| operator: string; | ||
| start: number; | ||
| } | ||
|
|
||
| /** | ||
| * Give every mutant its anchor, numbering the ones that would otherwise | ||
| * collide. The ordinal counts only mutants sharing both the same enclosing | ||
| * name and the same `from → to`, so adding a different kind of mutant beside a | ||
| * recorded one leaves its number alone. | ||
| * | ||
| * Mutants are numbered in source order, which is the order `generateMutants` | ||
| * produces them in for a single file. | ||
| */ | ||
| export const anchorMutants = <M extends HasLocation>( | ||
| program: object, | ||
| mutants: readonly M[], | ||
| ): (M & AnchoredMutant)[] => { | ||
| const seen = new Map<string, number>(); | ||
| const named = mutants.map((mutant) => ({ | ||
| ...mutant, | ||
| name: anchorNameAt(program, mutant.start), | ||
| })); | ||
| const totals = new Map<string, number>(); | ||
| for (const mutant of named) { | ||
| const key = `${mutant.name} ${mutant.operator}→${mutant.newOperator}`; | ||
| totals.set(key, (totals.get(key) ?? 0) + 1); | ||
| } | ||
| return named.map((mutant) => { | ||
| const key = `${mutant.name} ${mutant.operator}→${mutant.newOperator}`; | ||
| const nth = (seen.get(key) ?? 0) + 1; | ||
| seen.set(key, nth); | ||
| const { name, ...rest } = mutant; | ||
| return { | ||
| ...(rest as unknown as M), | ||
| anchor: totals.get(key)! > 1 ? `${name}@${nth}` : name, | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When an enclosing name has several identical Useful? React with 👍 / 👎.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Correct, and this was the important one. Fixed in 37f70db. An anchor now carries a fingerprint of the expression it mutates, taken from the smallest node that strictly contains the mutated span. For a swapped operator that is the expression itself ( Ordinals survive only for mutants sharing a name, a Your exact scenario is now a test — Worth saying plainly: this was a shortcut, not an oversight. The TODO this PR implements asked for "the expression text plus its enclosing function name", and I substituted an ordinal to keep the registry line easy to parse. My own repointing commit on this branch then hit the failure in the wild — three entries needed their ordinals corrected because main had added mutants above them inside the same function — and I wrote that up as the system working correctly rather than as the bug you found. Generated by Claude Code |
||
| }; | ||
| }); | ||
| }; | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When an entry uses an absolute path or an in-project alias such as
../tickets/src/foo.ts,resolvereads the intended file and generated keys retain that exact spelling, so this precommit check reports success. The mutation runner, however, keys the same file through its canonical project-relative path (src/foo.ts), meaning the accepted entry never suppresses its mutant and is skipped by per-file registry validation. Reject absolute paths and requiresourcePathto equal the canonical relative path, matching the audit's path checks.Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed. Fixed in 37f70db.
whyPathUnusablenow rejects three spellings: an absolute path, one escaping the project, and one that resolves to a file inside the project but is not written as the canonical relative path. The last is the case you named — the error sayspath must be written as "src/foo.ts"rather than just refusing.Three of the nine checker tests cover them, including
../<root>/src/read.tsreaching out and back.Generated by Claude Code