Fully test the spec runner - #1963
Conversation
Takes scripts/specs/run.ts from 28.3% to 97.7% on the mutation gate, and the paths beside it to 100%. Most of the gap was where the tests were kept. The gate only runs a source file's own mirrored folder, and run-execution.test.ts sat beside run.test.ts rather than inside one, so it refused to run at all — it was not checking this file. Moving both into test/scripts/specs/run/ took the score to 77.4% before anything new was written. New tests cover what nothing had pinned down: that asking for a case nobody wrote selects nothing at all, rather than running everything; that a failing case is run once instead of being quietly retried; that the reports folder is made however deep it is; and that a run with no paths reads the specs folder rather than wherever it happens to be started from. The report folder and the support file patterns moved to specs/paths.ts, where the rest of the spec paths live. Both entry points rely on them, and a test can now check the patterns actually match files — a typo there would leave every run with no steps to call. Option defaults moved from `??` chains to destructured defaults, which mean the same thing and leave nothing to get backwards. One mutant is recorded as impossible to catch, with the reason: telling Cucumber to be strict changes only how it treats undefined and pending steps, and we already reject exactly those two statuses ourselves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EJqdnQoBuuQkQCFVcQHcVn
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 6 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (8)
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Comment |
2 participants
The next chunk after #1960, continuing #1944, #1951 and #1954: taking one
scripts/file at a time up to a full mutation score, so its tests really would catch a change to it.scripts/specs/run.tsgoes from 28.3% to 97.7%, and the paths module beside it to 100%. This is the file that runs every Cucumber story, so when it goes wrong the stories can pass while proving nothing.The tests were in the wrong place. Again.
run-execution.test.tssat besiderun.test.tsrather than inside a folder mirroring the source. The mutation gate refused it outright — it would not even start a run that included it, because it could not tell which source that file belonged to. So the file's main suite was not checking the file it was named after.Moving both into
test/scripts/specs/run/took the score from 28.3% to 77.4% before a line of new test code was written. That is four chunks in a row where the biggest gap was where the tests lived.What was untested
Where the settings live now
The reports folder and the two support-file patterns moved into
specs/paths.ts, alongside the rest of the spec paths. Both entry points depend on them, and there a test can check the patterns actually match files on disk — a typo in one would leave every run with no steps to call, and every story would fail for want of anything to do.Option defaults also moved from
??chains to destructured defaults: same meaning, nothing left to get backwards.The one that got away
Telling Cucumber to be strict is recorded as impossible to catch, with the reason written down: strictness only changes how Cucumber treats undefined and pending steps, and we already reject exactly those two statuses ourselves, so the run fails either way.
One mutant does still survive: the default
["specs"]search path. Blanking it happens to find the same stories today, because scanning from the project root reaches the same folder, so I could not write an assertion that tells the two apart without inventing one. It is a real gap, and I would rather say so than paper over it.Next
The remaining
scripts/files with gaps arestripe-mock/install.tsandmutation.ts.Generated by Claude Code