Fix dotnet-msbuild skill eval regressions and coverage gaps (#896)

[JanKrivanek]

Addresses the cross-family skill-evaluation findings in #896 for the dotnet-msbuild plugin.

P0

• eval-performance (FIX-REGRESSION) — added diagnostic-first framing and explicit stop-conditions (do not act on compile-time slowness, incremental issues, or unmeasured builds), and reframed prescriptive Fix: bullets as measurement-gated remedies. Stops the skill over-applying and regressing baseline.
• msbuild-antipatterns (FIX-DISCOVERY) — rewrote the description with an action-verb lead (Detect and fix…) and concrete trigger phrases ('is this project file correct?', 'what's wrong with my build file?') to raise invocation.

P1

• Coverage gap — added cross-family eval.vally.yaml for extension-points, item-management, property-patterns, and target-authoring (previously excluded from the matrix), translated from each skill's base eval.yaml.
• check-bin-obj-clash (TRIM-COST) — trimmed 465 -> 315 lines by collapsing the redundant text-log fallback (Steps 2-9), removing the duplicate example workflow, and condensing tips. MCP primary workflow, Common Causes/Fixes, and the Global Properties table are kept.

P2 (strengthen)

• build-parallelism — added an opinionated Diagnose a slow parallel build (start here) critical-path checklist and sharpened the trigger.
• binlog-generation — added a one build = one binlog rule and a verify the binlog exists step.

Validation

• All 4 new vally configs and edited frontmatter parse cleanly; descriptions within the 10-1024 char bound.
• Plugin stays under the 15,000-char rendered skill-menu budget: 14,981 -> 14,873 (also widens the previously razor-thin 19-char margin).
• eval-performance rubric keywords and check-bin-obj-clash detection keywords preserved; no editor errors.
• The authoritative skill-validator check and the full cross-family vally evals were not runnable locally (SDK build blocked by network TLS; evals need live model runs) — they should run in CI to confirm the P0/P2 skills now win and the 4 newly-covered skills land on the scorecard.

Refs #896

(Coauthored by Copilot)

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
⚠️	dotnet-msbuild	check-bin-obj-clash	3/5	60%
❌	dotnet-msbuild	extension-points	0/1	0%
❌	dotnet-msbuild	property-patterns	0/1	0%
✅	dotnet-test	detect-static-dependencies	15/15	100%
✅	dotnet-test	writing-mstest-tests	45/45	100%

Uncovered: dotnet-msbuild/check-bin-obj-clash

• [WorkflowStep] Step 2: Get an overview and list projects (line 48)
• [WorkflowStep] Step 5: Check for double writes (line 62)

Uncovered: dotnet-msbuild/extension-points

• [CodePattern] [MSBuild] (line 195)

Uncovered: dotnet-msbuild/property-patterns

• [CodePattern] [MSBuild] (line 33)

[JanKrivanek]

/evaluate

[Copilot]

Pull request overview

This PR updates the dotnet-msbuild plugin’s skill docs and cross-family Vally evaluation coverage to address the evaluation findings/regressions tracked in #896, with a focus on improving skill discovery/behavior and expanding the scored test matrix.

Changes:

• Add cross-family eval.vally.yaml coverage for additional dotnet-msbuild skills (extension points, item management, property patterns, target authoring).
• Refine multiple dotnet-msbuild SKILL.md documents to improve invocation triggers, reduce over-prescription, and tighten diagnostic guidance (notably eval-performance, msbuild-antipatterns, check-bin-obj-clash, build-parallelism, binlog-generation).

Show a summary per file

File	Description
tests/dotnet-msbuild/target-authoring/eval.vally.yaml	Adds cross-family Vally stimuli/graders for target-authoring.
tests/dotnet-msbuild/property-patterns/eval.vally.yaml	Adds cross-family Vally stimuli/graders for property-patterns.
tests/dotnet-msbuild/item-management/eval.vally.yaml	Adds cross-family Vally stimuli/graders for item-management.
tests/dotnet-msbuild/extension-points/eval.vally.yaml	Adds cross-family Vally stimuli/graders for extension-points.
plugins/dotnet-msbuild/skills/msbuild-antipatterns/SKILL.md	Reworks description to improve discovery/trigger phrasing and scope clarity.
plugins/dotnet-msbuild/skills/eval-performance/SKILL.md	Adds measurement-first framing and stop-conditions to avoid misapplication.
plugins/dotnet-msbuild/skills/check-bin-obj-clash/SKILL.md	Trims/condenses the fallback workflow guidance and tips.
plugins/dotnet-msbuild/skills/build-parallelism/SKILL.md	Adds a “start here” critical-path checklist and sharpens the trigger.
plugins/dotnet-msbuild/skills/binlog-generation/SKILL.md	Adds “one build = one binlog” guidance and a “verify binlog exists” step.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 9/9 changed files
• Comments generated: 2

[github-actions]

👋 @JanKrivanek — this PR has 2 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[github-actions]

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
msbuild-antipatterns	Review MSBuild files for anti-patterns and style issues	5.0/5 → 4.3/5 🔴	✅ msbuild-antipatterns; tools: skill, glob / ✅ msbuild-antipatterns; tools: glob, task, read_agent, skill	✅ 0.09	❌
msbuild-antipatterns	Add a module to an F# project	5.0/5 → 5.0/5	⚠️ NOT ACTIVATED	✅ 0.09	❌ [1]
msbuild-antipatterns	Fix broken file order causing FS0039	4.0/5 → 4.0/5	⚠️ NOT ACTIVATED	✅ 0.09	❌ [2]
msbuild-antipatterns	Add a signature file to define public API	5.0/5 → 5.0/5	⚠️ NOT ACTIVATED	✅ 0.09	❌ [3]
property-patterns	Diagnose shared build property issues	5.0/5 → 5.0/5	✅ property-patterns; tools: skill / ✅ property-patterns; tools: skill, bash	✅ 0.08	❌ [4]
property-patterns	Diagnose multi-level property hierarchy bugs	4.0/5 → 5.0/5 🟢	✅ property-patterns; tools: skill	✅ 0.08	✅ [5]
property-patterns	Fix shared property configuration	5.0/5 → 5.0/5	✅ property-patterns; tools: skill / ✅ property-patterns; tools: skill, bash	✅ 0.08	❌ [6]
target-authoring	Diagnose custom target build regression	3.7/5 → 5.0/5 🟢	✅ target-authoring; tools: skill, edit, bash / ✅ target-authoring; tools: skill, bash, edit	🟡 0.23	✅ [7]
target-authoring	Diagnose broken SDK target chain across files	3.0/5 → 2.7/5 🔴	✅ target-authoring; tools: skill	🟡 0.23	❌ [8]
target-authoring	Fix custom target anti-patterns	4.7/5 → 4.3/5 🔴	✅ target-authoring; tools: skill / ✅ target-authoring; tools: skill, bash	🟡 0.23	❌ [9]
build-parallelism	Analyze build parallelism bottlenecks	3.3/5 ⏰ → 4.0/5 🟢	✅ build-parallelism; tools: binlog-binlog_build_graph, binlog-binlog_overview, glob, skill, binlog-binlog_expensive_projects, edit / ✅ build-parallelism; tools: binlog-binlog_build_graph, binlog-binlog_overview, binlog-binlog_projects, skill, glob, edit, binlog-binlog_expensive_projects	🟡 0.41	✅ [10]
check-bin-obj-clash	Diagnose bin/obj output path clashes	4.3/5 → 4.7/5 🟢	✅ check-bin-obj-clash; tools: skill, binlog-binlog_overview, binlog-binlog_double_writes, binlog-binlog_evaluations, binlog-binlog_properties, binlog-binlog_evaluation_global_properties, glob, binlog-binlog_evaluation_properties / ✅ check-bin-obj-clash; tools: binlog-binlog_double_writes, skill, binlog-binlog_evaluations, binlog-binlog_properties, binlog-binlog_evaluation_global_properties, edit, binlog-binlog_evaluation_properties, binlog-binlog_projects, binlog-binlog_overview	🟡 0.27	✅ [11]
extension-points	Diagnose build extension point failures	3.0/5 → 5.0/5 🟢	✅ extension-points; tools: skill, edit	✅ 0.08	✅
extension-points	Diagnose NuGet package and repo extension conflicts	3.0/5 → 3.0/5	✅ extension-points; tools: skill / ✅ extension-points; tools: skill, edit	✅ 0.08	❌ [12]
extension-points	Fix extension point anti-patterns	5.0/5 → 5.0/5	✅ extension-points; tools: skill	✅ 0.08	❌ [13]
item-management	Diagnose item group and batching issues	4.3/5 → 5.0/5 🟢	✅ item-management; tools: skill	✅ 0.17	✅ [14]
item-management	Diagnose cascading item and batching bugs in code generation pipeline	4.3/5 → 4.0/5 🔴	✅ item-management; tools: skill, edit / ✅ item-management; tools: glob, skill, edit, bash	✅ 0.17	❌ [15]
item-management	Fix item management anti-patterns	3.7/5 → 4.0/5 🟢	✅ item-management; tools: skill / ✅ item-management; tools: bash, skill	✅ 0.17	❌ [16]
binlog-generation	Build project with /bl flag	1.3/5 → 5.0/5 🟢	✅ binlog-generation; tools: skill / ⚠️ NOT ACTIVATED	🟡 0.37	✅ [17]
binlog-generation	Build with /bl in PowerShell	3.0/5 → 5.0/5 🟢	✅ binlog-generation; tools: skill	🟡 0.37	✅ [18]
binlog-generation	Build multiple configurations with unique binlogs	2.3/5 → 3.0/5 🟢	✅ binlog-generation; tools: skill / ⚠️ NOT ACTIVATED	🟡 0.37	❌ [19]
eval-performance	Analyze MSBuild evaluation performance issues	5.0/5 → 5.0/5	✅ eval-performance; tools: skill	✅ 0.20	❌

[1] (Plugin) Quality unchanged but weighted score is -3.4% due to: tokens (103651 → 171992)
[2] (Plugin) Quality unchanged but weighted score is -3.3% due to: tokens (72893 → 120587)
[3] (Plugin) Quality unchanged but weighted score is -5.0% due to: tokens (71069 → 135373)
[4] (Plugin) Quality unchanged but weighted score is -5.7% due to: tokens (85748 → 209234), time (46.8s → 59.2s)
[5] ⚠️ High run-to-run variance (CV=116%) — consider re-running with --runs 5
[6] ⚠️ High run-to-run variance (CV=168%) — consider re-running with --runs 5
[7] ⚠️ High run-to-run variance (CV=139%) — consider re-running with --runs 5
[8] ⚠️ High run-to-run variance (CV=2147%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -0.0% due to: tokens (97445 → 210017), time (63.0s → 98.4s)
[9] (Plugin) Quality improved but weighted score is -3.9% due to: tokens (62262 → 272924), time (39.6s → 65.6s), tool calls (9 → 12)
[10] ⚠️ High run-to-run variance (CV=133%) — consider re-running with --runs 5
[11] ⚠️ High run-to-run variance (CV=100%) — consider re-running with --runs 5
[12] ⚠️ High run-to-run variance (CV=72%) — consider re-running with --runs 5. (Isolated) Quality unchanged but weighted score is -16.4% due to: judgment, tokens (63319 → 106183), quality
[13] ⚠️ High run-to-run variance (CV=51%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -0.8% due to: tokens (77318 → 121403)
[14] ⚠️ High run-to-run variance (CV=102%) — consider re-running with --runs 5
[15] ⚠️ High run-to-run variance (CV=510%) — consider re-running with --runs 5. (Plugin) Quality improved but weighted score is -0.6% due to: tokens (42029 → 149210), tool calls (5 → 9), time (35.2s → 52.1s)
[16] ⚠️ High run-to-run variance (CV=555%) — consider re-running with --runs 5. (Isolated) Quality improved but weighted score is -10.5% due to: judgment
[17] ⚠️ High run-to-run variance (CV=161%) — consider re-running with --runs 5
[18] ⚠️ High run-to-run variance (CV=104%) — consider re-running with --runs 5
[19] ⚠️ High run-to-run variance (CV=166%) — consider re-running with --runs 5. (Plugin) Quality improved but weighted score is -0.7% due to: tokens (38532 → 46732)

⏰ timeout — run(s) hit the (160s) scenario timeout limit; scoring may be impacted by aborting model execution before it could produce its full output (increase via timeout in eval.yaml)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 910 in dotnet/skills, download eval artifacts with gh run download 29514597213 --repo dotnet/skills --pattern "skill-validator-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/4a9160d6cb20493050003df81a66b99d99a38ce6/eng/skill-validator/src/docs/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[Copilot]

Copilot's findings

• Files reviewed: 9/9 changed files
• Comments generated: 2

[JanKrivanek]

/evaluate

[Copilot]

Copilot's findings

• Files reviewed: 9/9 changed files
• Comments generated: 0 new

[github-actions]

Skill Validation Results

Skill	Scenario	Quality	Skills Loaded	Overfit	Verdict
msbuild-antipatterns	Review MSBuild files for anti-patterns and style issues	5.0/5 → 5.0/5	✅ msbuild-antipatterns; tools: skill, glob / ✅ msbuild-antipatterns; tools: task, read_agent, skill, glob	✅ 0.07	❌ [1]
msbuild-antipatterns	Add a module to an F# project	5.0/5 → 5.0/5	⚠️ NOT ACTIVATED	✅ 0.07	❌ [2]
msbuild-antipatterns	Fix broken file order causing FS0039	4.0/5 → 4.0/5	⚠️ NOT ACTIVATED	✅ 0.07	❌ [3]
msbuild-antipatterns	Add a signature file to define public API	5.0/5 → 5.0/5	⚠️ NOT ACTIVATED	✅ 0.07	❌ [4]
property-patterns	Diagnose shared build property issues	5.0/5 → 5.0/5	✅ property-patterns; tools: skill, bash	✅ 0.16	❌ [5]
property-patterns	Diagnose multi-level property hierarchy bugs	4.3/5 → 4.3/5	✅ property-patterns; tools: skill	✅ 0.16	✅ [6]
property-patterns	Fix shared property configuration	5.0/5 → 4.3/5 🔴	✅ property-patterns; tools: skill	✅ 0.16	❌ [7]
target-authoring	Diagnose custom target build regression	4.3/5 → 5.0/5 🟢	✅ target-authoring; tools: skill, edit, bash	🟡 0.24	✅ [8]
target-authoring	Diagnose broken SDK target chain across files	3.0/5 → 3.0/5	✅ target-authoring; tools: skill / ✅ target-authoring; tools: skill, glob	🟡 0.24	❌
target-authoring	Fix custom target anti-patterns	4.7/5 → 4.3/5 🔴	✅ target-authoring; tools: skill, bash / ✅ target-authoring; tools: bash, skill	🟡 0.24	❌ [9]
build-parallelism	Analyze build parallelism bottlenecks	3.7/5 → 4.3/5 🟢	✅ build-parallelism; tools: binlog-binlog_build_graph, binlog-binlog_overview, skill, binlog-binlog_expensive_projects, binlog-binlog_projects / ✅ build-parallelism; tools: skill, glob, binlog-binlog_build_graph, binlog-binlog_overview, edit, binlog-binlog_errors, binlog-binlog_projects, binlog-binlog_expensive_projects	🟡 0.41	✅ [10]
check-bin-obj-clash	Diagnose bin/obj output path clashes	3.7/5 → 5.0/5 🟢	✅ check-bin-obj-clash; tools: skill, binlog-binlog_overview, binlog-binlog_double_writes, binlog-binlog_evaluations, binlog-binlog_evaluation_global_properties, binlog-binlog_properties, binlog-binlog_evaluation_properties, edit, glob / ✅ check-bin-obj-clash; tools: skill, binlog-binlog_double_writes, binlog-binlog_evaluations, binlog-binlog_properties, binlog-binlog_evaluation_global_properties, binlog-binlog_evaluation_properties, edit, binlog-binlog_overview, glob	✅ 0.19	✅ [11]
extension-points	Diagnose build extension point failures	3.0/5 → 5.0/5 🟢	✅ extension-points; tools: skill / ✅ extension-points; tools: skill, edit	✅ 0.09	✅ [12]
extension-points	Diagnose NuGet package and repo extension conflicts	3.0/5 → 3.0/5	✅ extension-points; tools: skill, edit	✅ 0.09	❌ [13]
extension-points	Fix extension point anti-patterns	4.3/5 → 5.0/5 🟢	✅ extension-points; tools: skill / ✅ extension-points; tools: skill, bash	✅ 0.09	✅ [14]
item-management	Diagnose item group and batching issues	5.0/5 → 5.0/5	✅ item-management; tools: skill	✅ 0.13	❌ [15]
item-management	Diagnose cascading item and batching bugs in code generation pipeline	4.3/5 → 4.3/5	✅ item-management; tools: skill, edit, bash	✅ 0.13	❌ [16]
item-management	Fix item management anti-patterns	3.7/5 → 4.0/5 🟢	✅ item-management; tools: skill / ⚠️ NOT ACTIVATED	✅ 0.13	✅ [17]
binlog-generation	Build project with /bl flag	1.3/5 → 5.0/5 🟢	✅ binlog-generation; tools: skill / ⚠️ NOT ACTIVATED	🟡 0.34	✅ [18]
binlog-generation	Build with /bl in PowerShell	2.7/5 → 5.0/5 🟢	✅ binlog-generation; tools: skill	🟡 0.34	✅
binlog-generation	Build multiple configurations with unique binlogs	3.7/5 → 2.7/5 🔴	⚠️ NOT ACTIVATED	🟡 0.34	❌ [19]
eval-performance	Analyze MSBuild evaluation performance issues	5.0/5 → 5.0/5	✅ eval-performance; tools: skill	✅ 0.18	❌ [20]

[1] ⚠️ High run-to-run variance (CV=60%) — consider re-running with --runs 5
[2] ⚠️ High run-to-run variance (CV=202%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -3.6% due to: tokens (103181 → 184625)
[3] ⚠️ High run-to-run variance (CV=303%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -0.8% due to: tokens (82638 → 120475)
[4] ⚠️ High run-to-run variance (CV=68%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -4.6% due to: tokens (71229 → 127464), quality
[5] (Plugin) Quality unchanged but weighted score is -5.7% due to: tokens (70501 → 189534)
[6] ⚠️ High run-to-run variance (CV=1153%) — consider re-running with --runs 5
[7] ⚠️ High run-to-run variance (CV=60%) — consider re-running with --runs 5
[8] ⚠️ High run-to-run variance (CV=321%) — consider re-running with --runs 5
[9] ⚠️ High run-to-run variance (CV=1144%) — consider re-running with --runs 5. (Plugin) Quality improved but weighted score is -1.6% due to: tokens (84479 → 202194), time (47.1s → 64.2s), tool calls (10 → 12)
[10] ⚠️ High run-to-run variance (CV=179%) — consider re-running with --runs 5
[11] ⚠️ High run-to-run variance (CV=228%) — consider re-running with --runs 5
[12] ⚠️ High run-to-run variance (CV=92%) — consider re-running with --runs 5
[13] (Isolated) Quality unchanged but weighted score is -8.1% due to: tokens (56413 → 147016), quality
[14] ⚠️ High run-to-run variance (CV=109%) — consider re-running with --runs 5
[15] (Plugin) Quality unchanged but weighted score is -5.6% due to: tokens (26420 → 73385), tool calls (3 → 4), time (19.3s → 25.2s)
[16] ⚠️ High run-to-run variance (CV=76%) — consider re-running with --runs 5. (Isolated) Quality unchanged but weighted score is -12.3% due to: tokens (41636 → 161838), quality, tool calls (5 → 10), time (41.4s → 54.9s)
[17] ⚠️ High run-to-run variance (CV=206%) — consider re-running with --runs 5
[18] ⚠️ High run-to-run variance (CV=137%) — consider re-running with --runs 5
[19] ⚠️ High run-to-run variance (CV=169%) — consider re-running with --runs 5
[20] (Plugin) Quality unchanged but weighted score is -10.8% due to: quality, tokens (45525 → 97913)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 910 in dotnet/skills, download eval artifacts with gh run download 29562258763 --repo dotnet/skills --pattern "skill-validator-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/3cdaff8c37282773626a4718c0d83ce9864aa0ad/eng/skill-validator/src/docs/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

✅ Evaluation passed for 3cdaff8. cc @ViktorHofer @JanKrivanek @dotnet/skills-msbuild-reviewers — please review.