Improve dotnet-test-migration skill signal

[Evangelink]

Summary

• trim migrate-xunit-to-mstest overhead by batching discovery, avoiding redundant builds, and reading only relevant mapping-reference sections
• strengthen migrate-mstest-v1v2-to-v3 with evidence-based activation, explicit runner preservation, and consistent MSTest.Sdk/VSTest guidance
• add a focused xUnit assertion-mapping eval, strengthen the executable v2-to-v3 migration eval, and add an already-on-v3 boundary fixture
• leave the two exemplar skills unchanged and avoid content changes for the v3-to-v4 no-verdict finding, which [skills-eval] dotnet-test-migration: 5 skills, 60% pass — 1 P0, 1 P1, 1 to strengthen, 2 keep #900 identifies as judge/scoring infrastructure

Validation

• new already-v3 fixture passes with dotnet test
• changed eval YAML files parse successfully
• diff review completed with all findings addressed
• local skill-validator execution was blocked while its Copilot SDK package attempted to download an npm payload and hit an external TLS handshake failure; /evaluate is requested below for the full harness run

Fixes #900

[Evangelink]

/evaluate

[Copilot]

Pull request overview

This PR improves the evaluation signal and guidance for the dotnet-test-migration plugin, primarily by tightening tests/evals around runner preservation and adding boundary coverage for “already migrated” scenarios.

Changes:

• Adds a focused xUnit-to-MSTest exception-assertion mapping stimulus to better measure concise, behavior-preserving answers.
• Strengthens the MSTest v2→v3 eval to require VSTest runner preservation (avoid MSTest.Sdk/MTP flips) and adds an “already on v3” boundary scenario + fixture.
• Updates the migrate-xunit-to-mstest and migrate-mstest-v1v2-to-v3 skill guidance to reduce redundant work (batch discovery, avoid unnecessary intermediate builds) and clarify runner expectations (VSTest vs MTP).

Show a summary per file

File	Description
tests/dotnet-test-migration/migrate-xunit-to-mstest/eval.yaml	Adds a new focused stimulus to validate precise exception assertion mappings without full-migration narration.
tests/dotnet-test-migration/migrate-mstest-v1v2-to-v3/fixtures/already-v3/TestProject.csproj	Introduces an “already MSTest v3” fixture project to support boundary testing.
tests/dotnet-test-migration/migrate-mstest-v1v2-to-v3/fixtures/already-v3/global.json	Adds fixture runner metadata indicating VSTest.
tests/dotnet-test-migration/migrate-mstest-v1v2-to-v3/fixtures/already-v3/CurrentTests.cs	Adds a minimal MSTest test file for the new “already-v3” fixture.
tests/dotnet-test-migration/migrate-mstest-v1v2-to-v3/eval.yaml	Strengthens v2→v3 migration requirements and adds a new “stop when already v3” scenario using the new fixture.
plugins/dotnet-test-migration/skills/migrate-xunit-to-mstest/SKILL.md	Refines workflow guidance to batch discovery, avoid redundant builds, and limit reference reading to relevant sections.
plugins/dotnet-test-migration/skills/migrate-mstest-v1v2-to-v3/SKILL.md	Clarifies preserving the test platform (VSTest vs MTP) and updates verification steps to avoid redundant build/test passes.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 7/7 changed files
• Comments generated: 0

[github-actions]

📊 Skill Evaluation Results

2 skill(s) evaluated — 0 improved, 2 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).

Skill	Result	Δ Preference [95% CI]	W/T/L	Quality	Baseline
migrate-mstest-v1v2-to-v3	❌	+16.4% [-11.1%, +43.8%]	5/4/2	4.8/5	4.3/5
migrate-xunit-to-mstest	❌	+16.9% [-1.4%, +35.3%]	4/9/0	4.9/5	4.4/5

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 929 in dotnet/skills, download eval artifacts with gh run download 29922407089 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/83df27d7a5d1bf5aad04d11ec22d894978dd158b/eng/vally-adapter/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

✅ Evaluation passed for 83df27d. cc @dotnet/dotnet-testing — please review.

[Evangelink]

/evaluate

[Copilot]

Copilot's findings

• Files reviewed: 12/12 changed files
• Comments generated: 3

[github-actions]

📊 Skill Evaluation Results

2 skill(s) evaluated — 2 improved, 0 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).

Skill	Result	Δ Preference [95% CI]	W/T/L	Quality	Baseline
migrate-mstest-v1v2-to-v3	✅	+27.3% [+0.2%, +54.3%]	8/1/2	4.9/5	4.2/5
migrate-xunit-to-mstest	✅	+30.8% [+8.9%, +52.6%]	7/6/0	4.8/5	4.2/5

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

👋 @Evangelink — this PR has 3 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
⚠️	dotnet-test-migration	migrate-mstest-v1v2-to-v3	8/16	50%
❌	dotnet-test-migration	migrate-xunit-to-mstest	0/1	0%

Uncovered: dotnet-test-migration/migrate-mstest-v1v2-to-v3

• [Validation] MSTest v3 packages (or MSTest.Sdk) correctly referenced; v1/v2 references removed (line 191)
• [Validation] Project builds with zero errors (line 192)
• [Validation] All tests pass (dotnet test) -- compare pass/fail counts to pre-migration baseline (line 193)
• [Validation] .testsettings replaced with .runsettings (if applicable) (line 194)
• [Pitfall] MSTest.Sdk tests not found by vstest.console (line 205)
• [WorkflowStep] Step 4: Update target frameworks if needed (line 131)
• [CodePattern] Assert.AreNotEqual (line 151)
• [CodePattern] Assert.AreSame (line 151)

Uncovered: dotnet-test-migration/migrate-xunit-to-mstest

• [CodePattern] Parallelize (line 120)

[Copilot]

Copilot's findings

• Files reviewed: 12/12 changed files
• Comments generated: 1

[Evangelink]

/evaluate

[Copilot]

Copilot's findings

• Files reviewed: 12/12 changed files
• Comments generated: 2

[Copilot]

Copilot's findings

• Files reviewed: 12/12 changed files
• Comments generated: 0 new

[github-actions]

📊 Skill Evaluation Results

2 skill(s) evaluated — 1 improved, 1 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).

Skill	Result	Δ Preference [95% CI]	W/T/L	Quality	Baseline
migrate-mstest-v1v2-to-v3	❌	+36.0% [-7.6%, +79.6%]	7/2/1	4.8/5	4.0/5
migrate-xunit-to-mstest	✅	+21.5% [+9.0%, +34.1%]	7/6/0	4.8/5	4.3/5

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 929 in dotnet/skills, download eval artifacts with gh run download 30015187180 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/7ec31feedc481eec97b7eb97b996d8d94215878b/eng/vally-adapter/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[Evangelink]

/evaluate

[Copilot]

Copilot's findings

• Files reviewed: 12/12 changed files
• Comments generated: 1

[Evangelink]

/evaluate

[Copilot]

Copilot's findings

• Files reviewed: 12/12 changed files
• Comments generated: 0 new

[github-actions]

📊 Skill Evaluation Results

2 skill(s) evaluated — 2 improved, 0 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).

Skill	Result	Δ Preference [95% CI]	W/T/L	Quality	Baseline
migrate-mstest-v1v2-to-v3	✅	+56.4% [+37.5%, +75.2%]	11/0/0	5.0/5	3.2/5
migrate-xunit-to-mstest	✅	+20.0% [+1.5%, +38.5%]	5/8/0	4.6/5	4.4/5

🔍 Full Results - additional metrics and failure investigation steps

[github-actions]

📊 Skill Evaluation Results

2 skill(s) evaluated — 2 improved, 0 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).

Skill	Result	Δ Preference [95% CI]	W/T/L	Quality	Baseline
migrate-mstest-v1v2-to-v3	✅	+32.7% [+4.4%, +61.1%]	7/3/1	4.9/5	3.8/5
migrate-xunit-to-mstest	✅	+33.8% [+12.7%, +55.0%]	8/5/0	4.9/5	3.8/5

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[AbhitejJohn]

@Evangelink Thanks for this — it's a genuinely strong improvement to the migration skills, and the direction is exactly right. I ran it through several independent AI code reviews plus an automated skill-quality check, and a cross-family evaluation (five models doing the migrations, scored by a separate model). Since it's already merged, treat the below as follow-up polish rather than blockers.

What stood out as great (all three reviewers agreed):

• The evals moved from loose keyword matching to actually gating on the migration result — file-contains/file-not-contains on the concrete migrated symbols plus dotnet test with Passed!. That Passed! check in particular guards against the classic false-pass where the run exits 0 simply because no tests were discovered after a botched attribute conversion.
• The new "already on v3" boundary test is the single best addition — it verifies the agent makes no changes and doesn't over-eagerly consolidate working references into the metapackage.
• The xUnit→MSTest assertion-semantics guidance is technically sound: exact-vs-assignable type checks, sequence equality, and the NU1605 downgrade warning (MSTest 4.1.0 pulls Test.Sdk 18.0.1) all check out.

A few things worth a quick follow-up, grouped by severity (already merged, so none are merge blockers — the grouping reflects how I'd prioritize the follow-up).

🟡 Non-blocking

• migrate-mstest-v1v2-to-v3 — one line contradicts the rest of the skill on version detection. The frontmatter and body correctly treat MSTest.TestFramework/TestAdapter NuGet 1.x–2.x as v2 (v1 = the old QualityTools assembly reference). But the Step-1 detection line reads "→ v1 if 1.x, v2 if 2.x," which mislabels NuGet 1.x as v1. Functionally low-impact (a NuGet reference migrates the same way regardless of the label), but it undercuts the skill's own "state the current version first" instruction. Suggest: "NuGet MSTest.TestFramework/TestAdapter 1.x or 2.x → v2."
• Untested exclusion boundary. migrate-mstest-v1v2-to-v3 says "DO NOT USE FOR migrating to MSTest v4 (use migrate-mstest-v3-to-v4)," but no eval scenario asserts non-activation for that case, so nothing stops this skill from firing on a v4 request that belongs to the sibling skill. A scenario with expect_activation: false prompting a v3→v4 upgrade would lock the boundary down. (The migrate-xunit-to-mstest skill has the same gap for its own exclusions.)

Cross-family evaluation results. Every one of the ten cells (both skills × five models) came back with a positive preference for the new version, and no model preferred the old version anywhere. Full run: https://github.com/dotnet/skills/actions/runs/30040414142

migrate-xunit-to-mstest — credibly better on all five models:

Executor model	Judge	Result	Win rate (W/T/L)	Mean preference [95% CI]
Opus	GPT	✅ credibly better	53.8% (21/18/0)	+29.2% [+18.4, +40.0]
GPT	Opus	✅ credibly better	61.5% (24/13/2)	+30.3% [+18.7, +41.8]
Sonnet 4.6	Opus	✅ credibly better	59.0% (23/16/0)	+28.2% [+19.0, +37.5]
Haiku	Opus	✅ credibly better	76.9% (30/7/2)	+42.6% [+30.1, +55.0]
MAI Flash	Opus	✅ credibly better	71.8% (28/11/0)	+44.1% [+32.0, +56.2]

migrate-mstest-v1v2-to-v3 — positive on all five; two cleared the strict bar, the other three are positive-but-inconclusive from a few unmatched/errored comparison runs (not regressions):

Executor model	Judge	Result	Win rate (W/T/L)	Mean preference [95% CI]
Opus	GPT	positive, inconclusive (unmatched)	70.0% (21/7/2)	+37.3% [+22.6, +52.1]
GPT	Opus	✅ credibly better	75.8% (25/8/0)	+46.7% [+33.6, +59.8]
Sonnet 4.6	Opus	positive, inconclusive (errored/unmatched)	64.3% (18/5/5)	+31.4% [+13.2, +49.6]
Haiku	Opus	✅ credibly better	72.7% (24/4/5)	+41.2% [+20.6, +61.8]
MAI Flash	Opus	positive, inconclusive (unmatched)	71.9% (23/4/5)	+31.9% [+16.8, +47.0]

Overall: reviewers ranged from clean approve to approve-with-one-fix, the API/version claims that matter all verified correct, and the evaluation is uniformly positive. Nothing here blocks anything — the two version-detection/boundary items are the ones I'd tidy up first. Thanks again for the careful work here!