Close the remaining dotnet-test eval follow-ups (#899) - #971
Conversation
Every non-agent dotnet-test eval now clears the 5-trial floor, the two cost P1s are addressed, and the reference-skill coverage gap is recorded as a decision instead of a standing warning. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 024b3241-d7af-418a-b9cd-3bb9ee9bf0ee
Skill Coverage Report
|
There was a problem hiding this comment.
Pull request overview
This PR finishes the remaining follow-ups from #899 for the dotnet-test plugin by (1) raising underpowered evals above the minimum-trials floor via new discriminating scenarios/fixtures, (2) reducing coverage-analysis and code-testing-agent cost/verbosity (progressive disclosure + scoped pipeline), and (3) formalizing how “reference skills” (disable-model-invocation: true) are treated by the eval-quality gate and plugin docs.
Changes:
- Added new eval scenarios and fixtures across multiple
dotnet-testskills to increase statistical power withoutruns:padding. - Refactored
coverage-analysisdocumentation into gated reference files and updated eval-quality reporting to bucket reference skills separately. - Deleted duplicated/orphaned
mtp-hot-reloadfixture trees and removed stale underpowered allowlist entries.
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet-test/mtp-hot-reload/fixtures/vstest-mstest/ShippingCalculatorTests.cs | Removes an orphaned/duplicated fixture file. |
| tests/dotnet-test/mtp-hot-reload/fixtures/mtp-mstest-sdk9/TestProject.csproj | Removes an orphaned/duplicated fixture project. |
| tests/dotnet-test/mtp-hot-reload/fixtures/mtp-mstest-sdk9/PaymentProcessorTests.cs | Removes an orphaned/duplicated fixture test file. |
| tests/dotnet-test/mtp-hot-reload/fixtures/mtp-mstest-sdk9/global.json | Removes an orphaned/duplicated fixture SDK pin. |
| tests/dotnet-test/mtp-hot-reload/fixtures/mtp-mstest-sdk10/InventoryServiceTests.cs | Removes an orphaned/duplicated fixture test + production sample. |
| tests/dotnet-test/mtp-hot-reload/fixtures/mtp-mstest-sdk10/global.json | Removes an orphaned/duplicated fixture SDK pin. |
| tests/dotnet-test/mtp-hot-reload/fixtures/mtp-mstest-hotreload-installed/TestProject.csproj | Removes an orphaned/duplicated HotReload fixture project. |
| tests/dotnet-test/mtp-hot-reload/fixtures/mtp-mstest-hotreload-installed/global.json | Removes an orphaned/duplicated fixture SDK pin. |
| tests/dotnet-test/mtp-hot-reload/fixtures/mtp-mstest-hotreload-installed/EmailValidatorTests.cs | Removes an orphaned/duplicated fixture test file. |
| tests/dotnet-test/grade-tests/fixtures/dotnet-production-available/Orders.Tests/Orders.Tests.csproj | Adds nullable + references production project for “production available” grading scenario. |
| tests/dotnet-test/grade-tests/fixtures/dotnet-production-available/Orders.Tests/OrderCalculatorTests.cs | Adds C# test fixture designed to exercise grading bands with production present. |
| tests/dotnet-test/grade-tests/fixtures/dotnet-production-available/Orders.Core/Orders.Core.csproj | Adds production project for the new grading fixture. |
| tests/dotnet-test/grade-tests/fixtures/dotnet-production-available/Orders.Core/OrderCalculator.cs | Adds production code-under-test for grading eval. |
| tests/dotnet-test/grade-tests/eval.yaml | Adds new “production available” grading scenario (but currently has a YAML-structure defect). |
| tests/dotnet-test/generate-testability-wrappers/fixtures/no-di-library/AuditTrail.cs | Adds no-DI fixture to force an ambient-context seam recommendation. |
| tests/dotnet-test/generate-testability-wrappers/fixtures/no-di-library/AuditLibrary.csproj | Adds project wrapper for the no-DI fixture. |
| tests/dotnet-test/generate-testability-wrappers/eval.yaml | Adds scenario covering ambient seam expectations when DI is explicitly disallowed. |
| tests/dotnet-test/find-untested-sources/fixtures/generated-and-orphan/tests/Billing.Tests/SmokeEnvironmentTests.cs | Adds “orphan test” discriminator fixture. |
| tests/dotnet-test/find-untested-sources/fixtures/generated-and-orphan/tests/Billing.Tests/DiscountRulesTests.cs | Adds a paired test to ensure it’s not reported untested. |
| tests/dotnet-test/find-untested-sources/fixtures/generated-and-orphan/tests/Billing.Tests/Billing.Tests.csproj | Adjusts fixture wiring for src/tests split analysis. |
| tests/dotnet-test/find-untested-sources/fixtures/generated-and-orphan/src/Billing/TaxTable.Designer.cs | Adds generated-source discriminator fixture (.Designer.cs). |
| tests/dotnet-test/find-untested-sources/fixtures/generated-and-orphan/src/Billing/Resources.g.cs | Adds generated-source discriminator fixture (.g.cs). |
| tests/dotnet-test/find-untested-sources/fixtures/generated-and-orphan/src/Billing/InvoiceCalculator.cs | Adds unpaired production file to be detected as needing tests. |
| tests/dotnet-test/find-untested-sources/fixtures/generated-and-orphan/src/Billing/DiscountRules.cs | Adds paired production file (should not be reported untested). |
| tests/dotnet-test/find-untested-sources/fixtures/generated-and-orphan/src/Billing/Billing.csproj | Adds production project for the generated/orphan fixture tree. |
| tests/dotnet-test/find-untested-sources/eval.yaml | Adds scenario validating generated-file exclusion + orphan-test detection. |
| tests/dotnet-test/coverage-analysis/fixtures/spread-gap/coverage.cobertura.xml | Adds Cobertura fixture where the gap is distributed across members. |
| tests/dotnet-test/coverage-analysis/fixtures/spread-gap/Contoso.Billing/TaxService.cs | Adds fixture code matching the distributed-gap Cobertura report. |
| tests/dotnet-test/coverage-analysis/fixtures/spread-gap/Contoso.Billing/InvoiceService.cs | Adds fixture code matching the distributed-gap Cobertura report. |
| tests/dotnet-test/coverage-analysis/fixtures/spread-gap/Contoso.Billing/Contoso.Billing.csproj | Adds fixture project for the distributed-gap scenario. |
| tests/dotnet-test/coverage-analysis/eval.yaml | Adds scenarios for distributed-gap arithmetic + honoring a user-supplied report path. |
| tests/dotnet-test/code-testing-agent/fixtures/python-single-function/textkit/slugify.py | Adds a single-function Python target for proportional focused-scope testing. |
| tests/dotnet-test/code-testing-agent/fixtures/python-single-function/textkit/init.py | Adds package marker for the Python single-function fixture. |
| tests/dotnet-test/code-testing-agent/fixtures/python-single-function/tests/.gitkeep | Adds an empty tests dir placeholder for the fixture layout. |
| tests/dotnet-test/code-testing-agent/fixtures/python-single-function/pyproject.toml | Adds minimal Python project config for pytest execution. |
| tests/dotnet-test/code-testing-agent/fixtures/failing-suite/tests/test_balance.py | Adds intentionally-failing pytest suite for dormancy/decline behavior. |
| tests/dotnet-test/code-testing-agent/fixtures/failing-suite/pyproject.toml | Adds minimal Python project config for the failing-suite fixture. |
| tests/dotnet-test/code-testing-agent/fixtures/failing-suite/ledger/balance.py | Adds production code for failing-suite diagnosis. |
| tests/dotnet-test/code-testing-agent/fixtures/failing-suite/ledger/init.py | Adds package marker for the failing-suite fixture. |
| tests/dotnet-test/code-testing-agent/eval.yaml | Adds a focused-scope cost scenario + a dormancy guard scenario. |
| plugins/dotnet-test/skills/coverage-analysis/SKILL.md | Trims and restructures coverage-analysis guidance with gated reference reads. |
| plugins/dotnet-test/skills/coverage-analysis/references/test-execution.md | New Phase 2 reference content for provider detection + dotnet test commands. |
| plugins/dotnet-test/skills/coverage-analysis/references/setup-discovery.md | New Phase 1 reference content (probes + output directory + report discovery). |
| plugins/dotnet-test/skills/coverage-analysis/references/report-generation.md | New Phase 5 reference content (ReportGenerator install/invocation). |
| plugins/dotnet-test/skills/code-testing-agent/SKILL.md | Introduces focused vs broad scope sizing to reduce unnecessary pipeline overhead. |
| plugins/dotnet-test/README.md | Documents why reference skills intentionally have no direct-activation evals. |
| eng/eval-quality/underpowered-allowlist.txt | Removes allowlist entries that are no longer needed after trial-count increases. |
| eng/eval-quality/README.md | Documents reference-skill rationale in the eval-quality guidance. |
| eng/eval-quality/check_eval_quality.py | Separates reference skills in uncovered-eval reporting via SKILL.md frontmatter probe. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Files not reviewed (1)
- tests/dotnet-test/find-untested-sources/fixtures/generated-and-orphan/src/Billing/TaxTable.Designer.cs: Generated file
- Files reviewed: 47/49 changed files
- Comments generated: 1
|
👋 @Evangelink — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the |
…te it The 'production code available' scenario carried a leftover tail from the edit that moved the 'production code unavailable' one. YAML keeps the last duplicate key, so every field of the new scenario was silently overwritten: it shipped as a byte-identical rerun of its predecessor and never loaded the dotnet-production-available fixture it was built around. Parsing the spec and counting stimuli - which is what verified this PR - returns the intended 5 names either way, so only the parser can see it. The eval-quality gate now loads specs with a duplicate-key-strict loader (failing check 9), with a self-test case and the incident recorded. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 024b3241-d7af-418a-b9cd-3bb9ee9bf0ee
GitHub Actions
Inactive
|
❌ Evaluation ran but produced no results. The evaluate job completed but no |
GitHub Actions
Inactive
github-actions
Bot
added
pr-state/ready-for-eval
📊 Skill Evaluation Results5 skill(s) evaluated — ✅ 0 improved, ❌ 5 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ code-testing-agent — detailsReason: Net win +20.0% (1W/4T/0L over 5 trial(s), sign test p=0.500), mean preference +8.0% — not credible (sign test p=0.500 > 0.05)
❌ coverage-analysis — detailsReason: Net win +80.0% (4W/1T/0L over 5 trial(s), sign test p=0.063), mean preference +44.0% — not credible (sign test p=0.063 > 0.05)
❌ find-untested-sources — detailsReason: Net win +60.0% (3W/2T/0L over 5 trial(s), sign test p=0.125), mean preference +24.0% — not credible (sign test p=0.125 > 0.05)
❌ generate-testability-wrappers — detailsReason: Net win +60.0% (4W/0T/1L over 5 trial(s), sign test p=0.188), mean preference +36.0% — not credible (sign test p=0.188 > 0.05)
❌ grade-tests — detailsReason: Net win +80.0% (4W/1T/0L over 5 trial(s), sign test p=0.063), mean preference +32.0% — not credible (sign test p=0.063 > 0.05)
🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
…aint Run 30611635547 reported 0 of 5 skills improved. The skills were not the problem: across 25 trials they scored 16W/8T/1L, every skill winning and none regressing. Four of the five could not have passed at any record. The sign test conditions on discordant (non-tie) trials, so at exactly 5 trials the only passing record is a clean 5W/0T/0L sweep. One tie drops the discordant count to 4, which is back below the floor. My previous commit raised all five evals to exactly 5 - the eligibility floor, which the README I wrote says is 'not a guarantee of adequate power'. At the 32% tie rate this run measured, a genuinely-helping skill parked there is certified about one run in ten; at 15 trials it is about nine in ten. - runs: 3 on the four evals whose scenarios discriminate (5 -> 15 trials). - generate-testability-wrappers had a real self-contradiction: When Not to Use disclaimed the no-DI case while Step 5 IS the answer for it, so the ambient-seam scenario never activated and lost unassisted. Same defect #945 fixed in migrate-static-to-wrapper; fixed the same way. - The already-abstracted scenario is a dormancy guard and now says so. - code-testing-agent is left at 5 trials on purpose. The only grader its baseline ever fails is the Requirement|Evidence table regex - every functional grader passes identically in both arms - so it is inert on outcomes and more runs would only re-measure a tie at the highest cost in the plugin. Removed the skill trigger phrase the overfitting judge flagged from the one scenario this PR added. - adapt.mjs said 'not credible (p=0.500)' for a 1W/4T/0L record, which describes a measured null rather than an unwinnable test. It now names the tie count and says no record could have passed. - The gate warns on evals sitting at 5-7 trials; 21 across the repo do. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 024b3241-d7af-418a-b9cd-3bb9ee9bf0ee
Why every eval failed — and it was not the skillsPulled the artifacts from run 30611635547. Across the five evals the trials were 16W / 8T / 1L — every skill winning, not one regressing — and the gate reported Four of the five could not have passed at any record. The gate is an exact one-sided sign test, which conditions on the discordant (non-tie) trials. At exactly 5 trials the only record reaching
This is my own regression from the previous commit: I raised all five evals to exactly 5 trials — the eligibility floor, which the README I edited in the same PR describes as "not a guarantee of adequate power for a realistic effect, which needs considerably more." At the 32% tie rate this run measured, a genuinely-helping skill parked at 5 trials is certified about one run in ten; at 15 it is about nine in ten. Fixes1. Power where scenarios discriminate — 2. A real skill bug the eval caught. 3. A dormancy guard that wasn't declared. "Decline wrapper generation for already-abstracted code" correctly stays dormant — "the static is already behind an interface" is in the skill's own When Not to Use — but had no 4. 5. The report said the wrong thing. 6. Prevention. The gate now warns on evals sitting at 5–7 trials, where only a flawless sweep passes. 21 evals across the repo are in that band — this is not just my five. Checks
|
|
/evaluate |
|
👋 Two ways to run it:
|
|
/evaluate fe6df0b |
github-actions
Bot
added
pr-state/ready-for-eval
📊 Skill Evaluation Results5 skill(s) evaluated — ✅ 4 improved, ❌ 1 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ code-testing-agent — detailsReason: Net win +20.0% (1W/4T/0L over 5 trial(s), sign test p=0.500), mean preference +8.0% — not credible — 4 of 5 trial(s) tied, leaving only 1 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
Per-scenario details for 4 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown. 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
…cord Three defects, none of which was 'the skill is weak'. 1. Discovery. The skill's description said to invoke it 'even when the request says to use the repository's standard test-generation workflow'. That phrase existed in exactly two places in this repo: the description, and three of this eval's own prompts. Nowhere else - a closed loop between the eval and the skill it measures. Removing it from one prompt last commit dropped that scenario's activation to zero (run 30631489464, skillActivationCount 0) even though 'please add unit tests for that one function' is plainly a test-generation request. The description advertised only project-scale work, so the model reasonably skipped it for a one-function ask. It now states that scope changes how much of the workflow runs, never whether the skill applies - which is true since the focused/broad split landed. 2. Eval integrity. Removed the phrase from the remaining three prompts. The overfitting judge had flagged it twice as handing the skilled arm an advantage the request does not contain. Activation is now measured against how users actually write. 3. Passability. At runs: 1 this eval could not pass at any effect size. One of its five scenarios is a dormancy guard whose correct outcome is a tie, so discordant trials cap at 4 - one short of the 5 the sign test needs. It was unwinnable before a trial ran. runs: 2 gives ~8 discordant-capable trials. Held at 2, not 3: this is the plugin's most expensive eval and its cost is the open P1 on #899. Merged config into defaults per check 10 rather than adding a second block. Graders left untouched - the Requirement|Evidence contract is a real deliverable, and removing it would cut signal, not overfitting. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 024b3241-d7af-418a-b9cd-3bb9ee9bf0ee
|
/evaluate d662d04 |
There was a problem hiding this comment.
Copilot's findings
Files not reviewed (1)
- tests/dotnet-test/find-untested-sources/fixtures/generated-and-orphan/src/Billing/TaxTable.Designer.cs: Generated file
Suppressed comments (1)
eng/vally-adapter/adapt.test.mjs:342
- This test declaration is formatted as a single long line with the opening brace and
const v = …on the same line, unlike the surrounding tests in this file. Reformatting it to match the existing style will improve readability and reduce accidental diffs.
test("the smallest record that does pass is five wins and no losses", () => { const v = gate([0.4, 0.4, 0.4, 0.4, 0.4]);
- Files reviewed: 50/52 changed files
- Comments generated: 0 new
github-actions
Bot
added
waiting-on-author
…ps-899
Both branches edited the same five dotnet-test evals. Resolutions:
* code-testing-agent - both sides independently concluded runs: 2, for
different reasons: main because more scenarios of that shape cost more
than they inform, this branch because a tie-by-design dormancy guard
capped discordant trials at 4 and made the eval unwinnable. Kept both
rationales in one comment.
Main's version is currently BROKEN: it added defaults: runs: 2 while
leaving the config: block in place, which vally's loader rejects
('cannot specify both config and defaults') - the exact defect check 10
on this branch catches. Verified against vally 0.9.0's loadEvalSpec:
main's spec is rejected, so code-testing-agent produces no verdicts on
main today. The merged file has a single defaults block and loads.
* coverage-analysis - interleaved additions. Kept all four new scenarios
(this branch: spread-across-members, CI-report-without-re-running;
main: refactoring-safety, branch-vs-line coverage). The shared
environment block is reused by both new scenarios, with each keeping
its own cobertura destination path.
* grade-tests - both sides added a scenario for the same property
(grading C# with production code present) on different fixtures. Took
main's Banking fixture and dropped this branch's Orders duplicate:
main's covers a superset of grade cases (adds a conditional-logic D),
and the C-grade case this branch contributed is already covered by
three other scenarios. Removed the now-orphaned fixture.
* generate-testability-wrappers - kept this branch's anti-hijack rubric
item over main's whitespace-only change.
* Fixture rename/rename - git matched this branch's new csproj files
against the orphaned mtp-hot-reload fixtures it deleted, which main had
independently repurposed into platform-detection. Both sides agree the
orphans go; kept each side's new files.
Validation: 97/97 specs load under vally 0.9.0, eval-quality gate clean
(plain and --base-ref), 22/22 gate self-tests, 21/21 adapter tests,
markdownlint 0 errors, no dotnet-test orphaned fixtures.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 024b3241-d7af-418a-b9cd-3bb9ee9bf0ee
|
/evaluate f88376f |
There was a problem hiding this comment.
Copilot's findings
Files not reviewed (1)
- tests/dotnet-test/find-untested-sources/fixtures/generated-and-orphan/src/Billing/TaxTable.Designer.cs: Generated file
Suppressed comments (1)
eng/vally-adapter/adapt.test.mjs:342
- This test definition looks like it was accidentally collapsed onto one line (opening brace +
const v = ...), which makes the file inconsistent with the surrounding style and harder to read/maintain.
test("the smallest record that does pass is five wins and no losses", () => { const v = gate([0.4, 0.4, 0.4, 0.4, 0.4]);
- Files reviewed: 37/39 changed files
- Comments generated: 1
📊 Skill Evaluation Results5 skill(s) evaluated — ✅ 4 improved, ❌ 1 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ code-testing-agent — detailsReason: Net win +20.0% (2W/8T/0L over 10 trial(s), sign test p=0.250), mean preference +8.0% — not credible — 8 of 10 trial(s) tied, leaving only 2 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
Per-scenario details for 4 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown. 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
git checkout --ours on a rename/rename conflict selects the ours-side *path* but leaves the conflicted *content*, so this csproj shipped with <<<<<<< / ======= / >>>>>>> markers and an xunit.v3 PackageReference that belongs to main's platform-detection fixture. MSBuild cannot load it. Restored the pre-merge content: a single ProjectReference to Billing.csproj, matching the other find-untested-sources fixtures. Verified the fixture builds (0 warnings, 0 errors) and scanned every tracked file in the repo for conflict markers — this was the only one. Caught in review by copilot-pull-request-reviewer. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 024b3241-d7af-418a-b9cd-3bb9ee9bf0ee
|
/evaluate ee2dd38 |
There was a problem hiding this comment.
Copilot's findings
Files not reviewed (1)
- tests/dotnet-test/find-untested-sources/fixtures/generated-and-orphan/src/Billing/TaxTable.Designer.cs: Generated file
Suppressed comments (1)
eng/vally-adapter/adapt.test.mjs:346
- This test declaration has the opening brace and the first statement (
const v = …) on the same line, which is inconsistent with the surrounding tests and may fail formatting/lint expectations for this file. Consider splitting it to the usual multi-line form.
test("the smallest record that does pass is five wins and no losses", () => { const v = gate([0.4, 0.4, 0.4, 0.4, 0.4]);
assert.equal(v.passed, true);
assert.equal(v.underpowered, false);
assert.ok(v.signTest.pValue <= SIGN_TEST_ALPHA);
assert.match(v.reason, /credibly better/);
- Files reviewed: 37/39 changed files
- Comments generated: 0 new
|
❌ Evaluation ran but produced no results. The evaluate job completed but no |
GitHub Actions
Inactive
|
❌ Evaluation ran but produced no results. The evaluate job completed but no |
📊 Skill Evaluation Results5 skill(s) evaluated — ✅ 4 improved, ❌ 1 no credible change, 🔻 0 regressed. A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at
ℹ️ Column legend
❌ code-testing-agent — detailsReason: Net win +10.0% (1W/9T/0L over 10 trial(s), sign test p=0.500), mean preference +4.0% — not credible — 9 of 10 trial(s) tied, leaving only 1 discordant trial(s). The sign test conditions on non-tie trials and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more trials to clear the ties (more scenarios or defaults.runs)
Per-scenario details for 4 skill(s) were omitted to keep this comment under GitHub's 65,536-character limit — open the job's step summary or Full Results for the complete breakdown. 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
What & why
#899's scorecard has been worked down by #906, #907, #917, #945 and #949. The
2026-07-25 refresh
left four open items and no PR against any of them. This closes them.
agent.*dotnet-test eval now clears the floor and is off the ledgercode-testing-agentcost (+4.6 turns / +11.0 tools)coverage-analysis(+263k tok / +6.7 turns / +8.9 tools)1. Statistical power: every dotnet-test eval now clears the floor
Below five trials the pass gate cannot reach
p ≤ 0.05at any effect size(
eng/eval-quality/README.mdcheck 8), so five dotnet-test skills could not havepassed however good they were. Each gained scenarios that measure something the
eval did not already cover — no
runs:padding, which satisfies the arithmeticwithout adding cross-scenario evidence.
find-untested-sources.Designer.cs/.g.csmust not enter the untested worklist; a test referencing no production type is an orphan, not a pairinggenerate-testability-wrappersAsyncLocal<T>(not[ThreadStatic]), anIDisposablescope, and UTC semantics preserved — theDateTimeKindregression #906 fixedgrade-testsUnverified, and the first to require a D grade (self-referential assertion)coverage-analysiscode-testing-agentThe two
code-testing-agentadditions are deliberately cheap — a tiny pytestpackage and a four-test suite — because its existing three scenarios are the most
expensive in the plugin and bumping
runs:would have tripled them.2.
coverage-analysis— the cost outlier, trimmed by progressive disclosureSKILL.mdwas 30 KB, and roughly two thirds of it was PowerShell for paths thecommon request never takes. Moved verbatim behind explicit read gates:
references/setup-discovery.mdreferences/test-execution.mdreferences/report-generation.mdSKILL.md keeps the value-add — the phase map, the branch table, the mandatory
Phase 4 summary contract, validation and pitfalls — and is now 15 KB. The
"existing report, diagnose my plateau" path, which is what the evals and most
users exercise, no longer pays for the from-scratch execution and
ReportGenerator prose at all.
3.
code-testing-agent— machinery sized to the requestThe completion contract mandated
.testagent/research.md,plan.md,status.mdand atest-gap-analysis+assertion-qualitypass on everyrequest. That is the +4.6 turns / +11.0 tools. New Step 2 splits focused
(one function, class or file; "tests for X only") from broad (project,
package, coverage threshold across files): focused scopes keep the checklist
inline, read the target plus one neighbouring test, and review their own
assertions without spawning extra passes. Points 3–5 of the contract — tests
pass, every requirement backed by a concrete test, assertions reviewed — still
apply to both. The existing "extend an existing suite to QuoteAsync only"
scenario is itself a focused request, so it gets cheaper without any rubric or
grader change.
4. The four reference skills: a decision, not a to-do
filter-syntax,platform-detection,code-testing-extensionsandtest-analysis-extensionssetdisable-model-invocation: true, which drops themfrom the CLI's
<available_skills>menu — deliberately, to free menu budget(
InvestigatingResults.md). The experiment'sskilledvariant loads exactly oneskill, so a direct-activation eval would load an arm the model can never invoke:
treatment equals control by construction, which is precisely the defect
failing check 7 exists to prevent. Adding four such evals would have made the
plugin's numbers worse while looking like coverage.
So the gate now reports them in their own bucket with that reasoning, and
eng/eval-quality/README.mdplus the plugin README record how they areexercised (through the consumers that load them, and in the plugin arm). Closing
this properly needs harness support for declaring a dependency in the skilled
variant — a change to the experiment and
gen-experiment.mjs, not a per-skilleval file.
5. Drive-bys
mtp-hot-reloadfixtures deleted.mtp-mstest-sdk9,mtp-mstest-sdk10,mtp-mstest-hotreload-installedandvstest-mstestarebyte-equivalent (modulo blank lines) to the scenario-named fixtures beside
them — rename leftovers. Wiring them up would have added duplicate scenarios,
so they are removed instead.
mainsince Fix dotnet-maui skill regressions and sharpen decisiveness #947 merged: four
dotnet-mauievals now meet the floor and the ledgererrors on exemptions that are no longer needed. Removing them is what the gate
asks for, and without it nothing else here can be verified green.
Checks
python eng/eval-quality/check_eval_quality.py— No errors, both plain andwith
--base-ref origin/main(the mode CI uses on a PR). Zero dotnet-testentries remain in
underpowered-allowlist.txt; the dotnet-test orphaned-fixturewarnings are gone.
python eng/eval-quality/selftest_eval_quality.py— 20/20; the gate still fireson every bug class after the
report_uncoveredchange.markdownlint-cli2overplugins/dotnet-test/**andeng/eval-quality/**— 0 errors.spread-gap,no-di-library,generated-and-orphananddotnet-production-availableall build with 0 warnings / 0 errors. The last onecaught a real missing
using System;before it shipped — the same defect classClose the dotnet-test skill-coverage gaps from the PR 945 report #949 had to repair after the fact.
package, file summary):
spread-gapdeclares 25/42 = 0.60 and every declaredrate matches its
<lines>payload within the gate's tolerance.failing-suitegives exactly the twointended failures (both wrong expectations in the tests, not defects in
ledger/balance.py), andslugify's five documented behaviours were checkedagainst the implementation.
skill-validator checkcould not run — building it downloads@github/copilot-win32-x64from npmjs and the sandbox blocks that TLShandshake (same as Close the dotnet-test skill-coverage gaps from the PR 945 report #949). No skill frontmatter or description changed, so its
skill-menu budget and metadata checks are unaffected.
Refs #899.
Checklist
eng/known-domains.txtfor any new external domains referenced by skill content. (N/A — no new external domains)