Skip to content
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ cov_profile/
.deno/
.deno-cache/
.local-data/
.mutation-runs/
docs-output*/
misc/
ARCHITECTURE.md
Expand Down
34 changes: 25 additions & 9 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -475,9 +475,9 @@ logging and table-scoped cache invalidation stay automatic.
- `deno task lint:ci` - Strict, read-only lint (`check --error-on-warnings`, no `--write`). Fails on lint warnings (e.g. cognitive complexity) and on any code that *would* be reformatted, without touching the checkout. This is the lint `deno task precommit` runs in **every** environment, so a clean `precommit` locally means the lint step will pass in CI too. Run `deno task lint` to auto-fix before re-running.
- `deno task build:edge` - Build for Bunny Edge deployment
- `deno task backup` - Dump the database out-of-band to a `.zip`. Uploads to the configured storage zone by default (so it appears on the Backups page and lets the next migration skip its own inline backup); pass `--out <path>` to write a local file. Runs in a full Deno process, so unlike the in-edge backup it has no per-request subrequest budget and can dump arbitrarily large databases.
- `deno task precommit` - Run all checks (typecheck, lint, tests, changed-file mutation)
- `deno task precommit` - Run all checks (typecheck, lint, tests)
- `deno task precommit:mutation` - The precommit mutation gate, runnable on its own: mutation-test every `src/` file this branch changed against every changed `test/` file and demand a 100% kill rate. The changed set is the branch's committed diff against the integration branch (`origin/main`, else a local `main`) via `base...HEAD` — three-dot/merge-base, so it's the branch's full diff vs main and stays bounded to the branch's own commits (precommit runs post-commit on a clean tree, so the index is empty). Because the project requires 100% coverage, a src change lands with its covering test change in the same commit range, so the changed set is its own source→test mapping. Skips cheaply when there is no base ref or no changed `src/` files (and likewise when src changed without any changed test). If a badly stale local `origin/main` balloons the changed set past `STALE_BASE_SOURCE_LIMIT`, it skips with a "run `git fetch origin main`" hint instead of mutating most of the tree. See [Mutation Testing](#mutation-testing).
- `deno task mutation <source-glob> <test-glob>` - Mutation-test your tests on demand: mutate operators in the source and check your tests catch it (see [Mutation Testing](#mutation-testing))
- `deno task mutation <source-glob> <test-glob>` - Mutation-test your tests on demand in an isolated `.mutation-runs/<id>/work` copy: mutate operators in the source and check your tests catch it (see [Mutation Testing](#mutation-testing))

### Running Individual Test Files

Expand Down Expand Up @@ -651,18 +651,34 @@ bundle for each mutant, so the mutation reaches the built asset the tests load.

How it works (and why it is bespoke): it mutates the source file **in place**,
runs the mapped tests in a fresh `deno test` subprocess, then restores the
file. In-place mutation is what makes mutations bind through `#…` import-map
aliases. The operator tables and AST walk are vendored from
file. The normal `deno task mutation` command first copies the current checkout
(including dirty source/test edits, excluding `.git`, cache/report folders,
local databases, secrets, and generated assets) to `.mutation-runs/<id>/work`;
all in-place writes and per-mutant bundle rebuilds happen inside that copy, not
the live files. Each run leaves `.mutation-runs/<id>/run.json` with the child
PID/status, so a stray run is easy to find and stop:

```bash
deno task mutation --list
deno task mutation --kill <run-id> # or: all
deno task mutation --clean finished # or: <run-id> / all
```

In-place mutation inside the copied checkout is what makes mutations bind
through `#…` import-map aliases. The operator tables and AST walk are vendored from
[Mutasaurus](https://github.com/christoshrousis/mutasaurus) (MIT); its own
execution model writes a temp copy but runs the original tests, so every mutant
falsely "survives" on an alias-based project — see
`scripts/mutation/LICENSE.mutasaurus.md`. As a manual tool it is **targeted**
(run `deno task mutation` on the module you are hardening) — running it across
the whole tree would be far too slow. `deno task precommit` does run it
automatically, but **only over the files this branch changed** (its committed
diff against `origin/main`/`main`): the `precommit:mutation` step
mutates each changed `src/` file against the changed `test/` files and demands a
100% kill rate, so the cost stays bounded to what you actually changed.
the whole tree would be far too slow. The standalone
`deno task precommit:mutation` runs it automatically, but **only over the files
this branch changed** (its committed diff against `origin/main`/`main`): the
`precommit:mutation` step mutates each changed `src/` file against the changed
`test/` files and demands a 100% kill rate, so the cost stays bounded to what
you actually changed. Run `deno task precommit:mutation` before merging a
branch that changes `src/` files; the standard `deno task precommit` no longer
runs it (it was too slow for every commit).
Known-equivalent survivors recorded in
`scripts/mutation/equivalent-mutants.txt` are suppressed, as with a manual run.
That file's header warns against recording `=== → ==`/`!== → !=` mutants
Expand Down
8 changes: 8 additions & 0 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,14 @@ descriptors — PRs #1478 and others). A weak-assertion audit script also exists

**Remaining:**

- **Mutation tests removed from `deno task precommit`.** The
`precommit:mutation` step was too slow for the standard precommit run and was
removed from `scripts/precommit/steps.ts`. The mutation gate still exists as
`deno task precommit:mutation` and `deno task mutation` — run it manually on
changed src/test pairs before merging. Re-wire it into precommit (perhaps
behind a flag or with a tighter changed-set bound) only if the per-commit
mutation cost comes down.

- **Property-based tests (item 5).** `fast-check` is currently used in only one
test (`test/lib/fold-tree.test.ts`). Add properties for: slug generation, CSV
round-trips (commas / quotes / CRLF), date formatting across timezones, token
Expand Down
39 changes: 35 additions & 4 deletions scripts/mutation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,30 @@
/**
* In-house mutation tester — "tests for your tests".
*
* Mutates binary/logical/assignment operators in the given source file(s), runs the
* Copies the checkout into `.mutation-runs/<id>/work`, mutates
* binary/logical/assignment operators in the copied source file(s), runs the
* mapped test file(s), and reports which mutants SURVIVED (were not caught by
* any assertion). It is the real version of the heuristic in
* `test-quality-audit.ts`: instead of guessing which assertions look weak, it
* proves which code changes your tests fail to notice.
*
* The operator tables and AST walk are derived from Mutasaurus (MIT); the
* execution model is our own — see scripts/mutation/LICENSE.mutasaurus.md.
* execution model is our own. The child process still mutates in place inside
* the copied checkout so import-map aliases bind to the mutant — see
* scripts/mutation/LICENSE.mutasaurus.md.
*
* Usage: deno task mutation <source-glob> <test-glob> [options]
*/

import { globToRegExp, join, normalize, SEPARATOR } from "@std/path";
import { runMutationTesting } from "./mutation/runner.ts";
import { runIsolatedMutationCommand } from "./mutation/isolation.ts";
import {
MUTATION_RUN_ID_ENV,
MUTATION_RUN_ROOT_ENV,
MUTATION_SNAPSHOT_CHILD_ENV,
MUTATION_WORK_ROOT_ENV,
withMutationRunLock,
} from "./mutation/isolation-state.ts";

const DEFAULT_TIMEOUT = 10_000;

Expand Down Expand Up @@ -197,6 +207,7 @@ const main = async (): Promise<void> => {
Deno.exit(1);
}

const { runMutationTesting } = await import("./mutation/runner.ts");
const code = await runMutationTesting({
...(args.batchJobs === undefined ? {} : { batchJobs: args.batchJobs }),
exhaustive: args.exhaustive,
Expand All @@ -208,4 +219,24 @@ const main = async (): Promise<void> => {
Deno.exit(code);
};

main();
const mutationRunRootFromEnv = (): string | null => {
const id = Deno.env.get(MUTATION_RUN_ID_ENV);
const runRoot = Deno.env.get(MUTATION_RUN_ROOT_ENV);
const workRoot = Deno.env.get(MUTATION_WORK_ROOT_ENV);
return id && runRoot && workRoot ? runRoot : null;
};

const runSnapshotChild = async (): Promise<void> => {
const runRoot = mutationRunRootFromEnv();
return runRoot === null
? await main()
: await withMutationRunLock(runRoot, main);
};

if (import.meta.main) {
if (Deno.env.get(MUTATION_SNAPSHOT_CHILD_ENV) === "1") {
await runSnapshotChild();
} else {
Deno.exit(await runIsolatedMutationCommand(Deno.args));
}
}
Loading