Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 24 additions & 7 deletions .github/workflows/stryker.yaml
Original file line number Diff line number Diff line change
@@ -1,15 +1,30 @@
# Stryker mutation testing
# Stryker mutation testing (#205 — release-gate quality bar)
#
# Runs the Stryker.NET mutation tester against the repo's test projects to
# measure mutation score. Mutation runs are slow — triggered manually
# (workflow_dispatch) and on a weekly schedule, not on every PR.
# Runs the Stryker.NET mutation tester to measure mutation score. The
# stryker-config.json `break` threshold is the documented floor: `dotnet stryker`
# exits non-zero (failing the job) if the score drops below it, so this is an
# enforced gate, not just a report. Ratchet the floor UP over time; never down.
#
# Triggers:
# - pull_request touching src/** — gates a PR that would drop the score below
# the floor, BEFORE merge. Mutation runs are slow (tens of minutes), so this
# is scoped to source changes only; docs/test-only/workflow PRs skip it.
# - schedule (weekly) + workflow_dispatch — full-project trend / on-demand runs.
#
# The workflow looks for a stryker-config.json at the repo root or under
# tests/**/. If none is present the run is a no-op (Stryker setup is a
# per-repo follow-up; this file is the canonical infrastructure).
# tests/**/. If none is present the run is a no-op.
name: Stryker (mutation testing)

on:
pull_request:
branches:
- main
- vNext
paths:
- 'src/**/*.cs'
- 'tests/**/*.cs'
- 'stryker-config.json'
- '.github/workflows/stryker.yaml'
workflow_dispatch:
schedule:
- cron: '0 6 * * 0' # weekly Sunday 06:00 UTC
Expand Down Expand Up @@ -65,7 +80,9 @@ jobs:

- name: Install dotnet-stryker
if: steps.check.outputs.found == 'true'
run: dotnet tool update -g dotnet-stryker || dotnet tool install -g dotnet-stryker
# Pinned: mutation score / the gate must be reproducible, and a floating
# Stryker can change instrumentation or break on an analyzer/Roslyn bump.
run: dotnet tool update -g dotnet-stryker --version 4.16.0 || dotnet tool install -g dotnet-stryker --version 4.16.0

- name: Run Stryker
if: steps.check.outputs.found == 'true'
Expand Down
52 changes: 52 additions & 0 deletions docs/mutation-testing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Mutation testing (#205)

Mutation testing seeds deliberate faults ("mutants") into the source and checks
that the test suite catches them. A *survived* mutant is a change to behaviour
that **no test noticed** — a hole in the suite that line coverage can't see.

This repo runs [Stryker.NET](https://stryker-mutator.io/docs/stryker-net/introduction/)
as an **enforced release-gate quality bar**, not just a report.

## The floor

`stryker-config.json` sets `thresholds.break`, which makes `dotnet stryker` exit
non-zero (failing the job) when the mutation score drops below it.

| | Value |
|---|---|
| Baseline score (full project, 2026-07-22, Stryker 4.16.0) | **74.4 %** — 248 killed / 83 survived / 2 timeout of 333 tested |
| **Enforced floor (`break`)** | **70 %** |

The floor sits a few points below the measured baseline so normal CI-runner
variance (a slow runner can turn a killed mutant into a timeout, nudging the
score) doesn't cause a spurious failure, while a real regression — deleting a
test, or adding untested behaviour — trips it.

**Policy: ratchet the floor UP, never down.** As survivors are killed and the
score climbs, raise `break` to lock in the gain. Lowering it to make a red build
pass defeats the point — fix the test gap instead.

## How it runs

- **Pull requests that touch `src/**`** run the gate before merge
(`.github/workflows/stryker.yaml`). It is scoped to source changes because a
full run takes tens of minutes; docs/test-only/workflow PRs skip it.
- **Weekly schedule + `workflow_dispatch`** run it on demand / for the trend.

## Running locally

```bash
dotnet tool install --global dotnet-stryker --version 4.16.0
dotnet stryker # full project (~4 min)
dotnet stryker --mutate "**/Report.cs" # a single file, faster
```

The HTML report under `StrykerOutput/**/reports/` lists every survived mutant with
its file, line, and the mutation applied — the worklist for raising the score.

## Not yet automated

Publishing the score trend to a chart and auto-filing `kind:mutation-survives`
issues for survivors (parts of #205) are deferred; the enforced floor above is the
load-bearing gate. The HTML/JSON reports are uploaded as a workflow artifact in the
meantime.
2 changes: 1 addition & 1 deletion stryker-config.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"thresholds": {
"high": 90,
"low": 75,
"break": 0
"break": 70
}
}
}
Loading