ci: enforce mutation-score floor as a release gate (#205) - #310
Merged
Conversation
Turns the existing (report-only) Stryker setup into an enforced release-gate quality bar. stryker-config.json `break` goes 0 -> 70: dotnet stryker now exits non-zero (failing the job) if the mutation score drops below the floor. The floor sits a few points under the measured full-project baseline (74.4% on Stryker 4.16.0, 2026-07-22) so CI-runner timeout variance doesn't flake it, while a real test-suite regression trips it. stryker.yaml gains a path-scoped pull_request trigger (src/** changes) so a PR that would drop the score below the floor is caught BEFORE merge — scoped to source because a full run takes tens of minutes; docs/test-only PRs skip it. The Stryker install is pinned to 4.16.0 so the gate is reproducible. (Note: this repo is on BannedApi 4.14 / PublicApi 3.3.4, not the 5.6 analyzers, so it is unaffected by the Stryker/CodeAnalysis-5.6 crash that blocks some fleet repos.) docs/mutation-testing.md documents the baseline, the floor, the ratchet-up-only policy, and how to run locally. Partial for #205: the enforced floor + PR gate (the load-bearing "release-gate quality bar") are done. The score-history chart and auto-filing of kind:mutation-survives issues are deferred; reports upload as a workflow artifact meanwhile. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes #205's remaining acceptance criteria on top of the enforced floor: - Score history: non-PR (schedule/dispatch) runs parse the final mutation score from the Stryker log and publish it to gh-pages /dev/stryker via github-action-benchmark (customBiggerIsBetter — same action/pin as the BDN charts), one data point per run. Trend visibility only; fail-on-alert:false so it never double-fails the run (the `break` floor is the gate). - Survivor tracking: non-PR runs write the survivor worklist (file:line + mutator, from mutation-report.json) to the job summary and keep ONE rolling kind:mutation-survives issue up to date — deliberately not one issue per survivor (dozens today = noise). Best-effort; never fails the run. The Run step now tees Stryker output with `set -o pipefail` so the gate's non-zero exit is preserved through the pipe. Publish/track steps run only on non-PR events, so the PR gate path is unchanged. Closes #205 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 24, 2026
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes Stryker mutation testing an enforced release-gate quality bar and completes the observability around it (issue #205).
The gate
stryker-config.jsonbreak: 0 → 70.dotnet strykerexits non-zero (failing the job) below the floor. The floor sits below the measured full-project baseline of 74.4% (Stryker 4.16.0; 248 killed / 83 survived / 2 timeout of 333 tested, 2026-07-22) so CI timeout variance doesn't flake it.stryker.yamlgains a path-scopedpull_requestgate (src/**changes) — a regression below the floor fails the PR before merge. Scoped to source (a full run is tens of minutes); docs/test-only PRs skip. Weekly schedule +workflow_dispatchkept.Score history + survivor tracking (non-PR runs)
gh-pages/dev/stryker/viagithub-action-benchmark(customBiggerIsBetter, same action/pin as the BDN charts). Trend visibility only —fail-on-alert: false.kind:mutation-survivesissue current — deliberately not one issue per survivor (dozens today would be noise). Best-effort; never fails the run.Runstep tees output underset -o pipefailso the gate's exit code survives the pipe.Docs
docs/mutation-testing.mddocuments the baseline, the floor, the ratchet-up-only policy, running locally, and the history/tracking behaviour.Acceptance criteria (#205)
break: 70,docs/mutation-testing.md)pull_request+break)gh-pages/dev/stryker)kind:mutation-survivesissue (single rolling issue, not per-mutant)Validation
stryker.yaml;stryker-config.jsonvalid JSON.Closes #205