Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ cov_profile/
.deno/
.deno-cache/
.local-data/
.mutation-runs/
docs-output*/
misc/
ARCHITECTURE.md
Expand Down
34 changes: 25 additions & 9 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -475,9 +475,9 @@ logging and table-scoped cache invalidation stay automatic.
- `deno task lint:ci` - Strict, read-only lint (`check --error-on-warnings`, no `--write`). Fails on lint warnings (e.g. cognitive complexity) and on any code that *would* be reformatted, without touching the checkout. This is the lint `deno task precommit` runs in **every** environment, so a clean `precommit` locally means the lint step will pass in CI too. Run `deno task lint` to auto-fix before re-running.
- `deno task build:edge` - Build for Bunny Edge deployment
- `deno task backup` - Dump the database out-of-band to a `.zip`. Uploads to the configured storage zone by default (so it appears on the Backups page and lets the next migration skip its own inline backup); pass `--out <path>` to write a local file. Runs in a full Deno process, so unlike the in-edge backup it has no per-request subrequest budget and can dump arbitrarily large databases.
- `deno task precommit` - Run all checks (typecheck, lint, tests, changed-file mutation)
- `deno task precommit` - Run all checks (typecheck, lint, tests)
- `deno task precommit:mutation` - The precommit mutation gate, runnable on its own: mutation-test every `src/` file this branch changed against every changed `test/` file and demand a 100% kill rate. The changed set is the branch's committed diff against the integration branch (`origin/main`, else a local `main`) via `base...HEAD` — three-dot/merge-base, so it's the branch's full diff vs main and stays bounded to the branch's own commits (precommit runs post-commit on a clean tree, so the index is empty). Because the project requires 100% coverage, a src change lands with its covering test change in the same commit range, so the changed set is its own source→test mapping. Skips cheaply when there is no base ref or no changed `src/` files (and likewise when src changed without any changed test). If a badly stale local `origin/main` balloons the changed set past `STALE_BASE_SOURCE_LIMIT`, it skips with a "run `git fetch origin main`" hint instead of mutating most of the tree. See [Mutation Testing](#mutation-testing).
- `deno task mutation <source-glob> <test-glob>` - Mutation-test your tests on demand: mutate operators in the source and check your tests catch it (see [Mutation Testing](#mutation-testing))
- `deno task mutation <source-glob> <test-glob>` - Mutation-test your tests on demand in an isolated `.mutation-runs/<id>/work` copy: mutate operators in the source and check your tests catch it (see [Mutation Testing](#mutation-testing))

### Running Individual Test Files

Expand Down Expand Up @@ -651,18 +651,34 @@ bundle for each mutant, so the mutation reaches the built asset the tests load.

How it works (and why it is bespoke): it mutates the source file **in place**,
runs the mapped tests in a fresh `deno test` subprocess, then restores the
file. In-place mutation is what makes mutations bind through `#…` import-map
aliases. The operator tables and AST walk are vendored from
file. The normal `deno task mutation` command first copies the current checkout
(including dirty source/test edits, excluding `.git`, cache/report folders,
local databases, secrets, and generated assets) to `.mutation-runs/<id>/work`;
all in-place writes and per-mutant bundle rebuilds happen inside that copy, not
the live files. Each run leaves `.mutation-runs/<id>/run.json` with the child
PID/status, so a stray run is easy to find and stop:

```bash
deno task mutation --list
deno task mutation --kill <run-id> # or: all
deno task mutation --clean finished # or: <run-id> / all
```

In-place mutation inside the copied checkout is what makes mutations bind
through `#…` import-map aliases. The operator tables and AST walk are vendored from
[Mutasaurus](https://github.com/christoshrousis/mutasaurus) (MIT); its own
execution model writes a temp copy but runs the original tests, so every mutant
falsely "survives" on an alias-based project — see
`scripts/mutation/LICENSE.mutasaurus.md`. As a manual tool it is **targeted**
(run `deno task mutation` on the module you are hardening) — running it across
the whole tree would be far too slow. `deno task precommit` does run it
automatically, but **only over the files this branch changed** (its committed
diff against `origin/main`/`main`): the `precommit:mutation` step
mutates each changed `src/` file against the changed `test/` files and demands a
100% kill rate, so the cost stays bounded to what you actually changed.
the whole tree would be far too slow. The standalone
`deno task precommit:mutation` runs it automatically, but **only over the files
this branch changed** (its committed diff against `origin/main`/`main`): the
`precommit:mutation` step mutates each changed `src/` file against the changed
`test/` files and demands a 100% kill rate, so the cost stays bounded to what
you actually changed. Run `deno task precommit:mutation` before merging a
branch that changes `src/` files; the standard `deno task precommit` no longer
runs it (it was too slow for every commit).
Known-equivalent survivors recorded in
`scripts/mutation/equivalent-mutants.txt` are suppressed, as with a manual run.
That file's header warns against recording `=== → ==`/`!== → !=` mutants
Expand Down
8 changes: 8 additions & 0 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,14 @@ descriptors — PRs #1478 and others). A weak-assertion audit script also exists

**Remaining:**

- **Mutation tests removed from `deno task precommit`.** The
`precommit:mutation` step was too slow for the standard precommit run and was
removed from `scripts/precommit/steps.ts`. The mutation gate still exists as
`deno task precommit:mutation` and `deno task mutation` — run it manually on
changed src/test pairs before merging. Re-wire it into precommit (perhaps
behind a flag or with a tighter changed-set bound) only if the per-commit
mutation cost comes down.

- **Property-based tests (item 5).** `fast-check` is currently used in only one
test (`test/lib/fold-tree.test.ts`). Add properties for: slug generation, CSV
round-trips (commas / quotes / CRLF), date formatting across timezones, token
Expand Down
39 changes: 35 additions & 4 deletions scripts/mutation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,30 @@
/**
* In-house mutation tester — "tests for your tests".
*
* Mutates binary/logical/assignment operators in the given source file(s), runs the
* Copies the checkout into `.mutation-runs/<id>/work`, mutates
* binary/logical/assignment operators in the copied source file(s), runs the
* mapped test file(s), and reports which mutants SURVIVED (were not caught by
* any assertion). It is the real version of the heuristic in
* `test-quality-audit.ts`: instead of guessing which assertions look weak, it
* proves which code changes your tests fail to notice.
*
* The operator tables and AST walk are derived from Mutasaurus (MIT); the
* execution model is our own — see scripts/mutation/LICENSE.mutasaurus.md.
* execution model is our own. The child process still mutates in place inside
* the copied checkout so import-map aliases bind to the mutant — see
* scripts/mutation/LICENSE.mutasaurus.md.
*
* Usage: deno task mutation <source-glob> <test-glob> [options]
*/

import { globToRegExp, join, normalize, SEPARATOR } from "@std/path";
import { runMutationTesting } from "./mutation/runner.ts";
import { runIsolatedMutationCommand } from "./mutation/isolation.ts";
import {
MUTATION_RUN_ID_ENV,
MUTATION_RUN_ROOT_ENV,
MUTATION_SNAPSHOT_CHILD_ENV,
MUTATION_WORK_ROOT_ENV,
withMutationRunLock,
} from "./mutation/isolation-state.ts";

const DEFAULT_TIMEOUT = 10_000;

Expand Down Expand Up @@ -197,6 +207,7 @@ const main = async (): Promise<void> => {
Deno.exit(1);
}

const { runMutationTesting } = await import("./mutation/runner.ts");
const code = await runMutationTesting({
...(args.batchJobs === undefined ? {} : { batchJobs: args.batchJobs }),
exhaustive: args.exhaustive,
Expand All @@ -208,4 +219,24 @@ const main = async (): Promise<void> => {
Deno.exit(code);
};

main();
const mutationRunRootFromEnv = (): string | null => {
const id = Deno.env.get(MUTATION_RUN_ID_ENV);
const runRoot = Deno.env.get(MUTATION_RUN_ROOT_ENV);
const workRoot = Deno.env.get(MUTATION_WORK_ROOT_ENV);
return id && runRoot && workRoot ? runRoot : null;
};

const runSnapshotChild = async (): Promise<void> => {
const runRoot = mutationRunRootFromEnv();
return runRoot === null
? await main()
: await withMutationRunLock(runRoot, main);
};

if (import.meta.main) {
if (Deno.env.get(MUTATION_SNAPSHOT_CHILD_ENV) === "1") {
await runSnapshotChild();
} else {
Deno.exit(await runIsolatedMutationCommand(Deno.args));
}
}
Loading