Move eight P0/P1 skills to staging

[y87feng]

Summary

Move eight cross-family P0/P1 skills out of main while retaining their complete pre-removal versions on the long-lived staging branch.

This PR removes each selected skill together with its eval/fixture directory, then cleans up only the associated CODEOWNERS, external-dependency allowlist, eval-quality allowlist, and dashboard sample-data entries.

Why these skills

The selection combines the latest corrected cross-family classifications (#893, #895, #896, #899, #901, including the #909 judge-infrastructure correction) with independently verified 30-day Copilot CLI usage from Kusto.

Telemetry window: 2026-07-02 11:11:59 through 2026-07-30 18:39:32. Automation accounts are accounts invoking at least 20 of the 92 skills; globally they account for 87.5% of invocations.

Skill	Plugin	Eval action	Non-automation users	Non-automation invocations	Automation share	Reason to stage
android-tombstone-symbolication	dotnet-diag	P0 FIX-REGRESSION	2	4	99.9%	Regresses on both frontier models and has almost no organic usage.
maui-collectionview	dotnet-maui	P0 FIX-REGRESSION	5	196	88.2%	Regresses about 30% of trials; usage is concentrated in a small audience.
msbuild-antipatterns	dotnet-msbuild	P0 FIX-DISCOVERY	31	82	78.5%	Negative impact and only 30% eval invocation; stage for priority repair because real demand exists.
eval-performance	dotnet-msbuild	P0 FIX-REGRESSION	5	9	97.4%	Regresses about 60% of trials with very little organic usage.
check-bin-obj-clash	dotnet-msbuild	P1 TRIM-COST	11	18	96.6%	Positive but thin evidence and high execution overhead; stage until it is cheaper and revalidated.
migrate-static-to-wrapper	dotnet-test	P0 FIX-REGRESSION	1	1	99.9%	Non-positive impact, frontier regressions, and effectively no organic adoption.
migrate-dotnet9-to-dotnet10	dotnet-upgrade	P0 FIX-REGRESSION	34	77	98.4%	Regresses about 39% of trials; stage for priority repair because real demand exists.
dotnet-aot-compat	dotnet-upgrade	P1 TRIM-COST	7	34	89.9%	Directionally useful but extremely expensive and based on thin eval evidence.

staging is a quarantine/repair branch, not a deletion of history. Skills with demonstrated demand can be fixed and revalidated there before being proposed back to main.

Explicitly retained on main

• All dotnet-blazor skills, including plan-ui-change and support-prerendering
• microbenchmarking, based on strong organic usage (31 users, 127 invocations, 7 users with 5+ invocations, 10 multi-day users)
• code-testing-agent, based on its exceptionally broad organic usage

Validation

• skill-validator check across all plugins: passed
• Validated 88 skills, 16 agents, and 16 plugins
• Reference scan: 0 errors
• git diff --check: passed
• Confirmed all eight removed skills remain present on origin/staging

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
❔	dotnet-diag	android-tombstone-symbolication	—	error
❔	dotnet-maui	maui-collectionview	—	error
❔	dotnet-msbuild	check-bin-obj-clash	—	error
❔	dotnet-msbuild	eval-performance	—	error
❔	dotnet-msbuild	msbuild-antipatterns	—	error
❔	dotnet-test	migrate-static-to-wrapper	—	error
❔	dotnet-upgrade	dotnet-aot-compat	—	error
❔	dotnet-upgrade	migrate-dotnet9-to-dotnet10	—	error

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[AbhitejJohn]

MMR verdict: 🟡 Request changes

The mechanical removal and config cleanup are correct and well-executed. The blocker is that 17 retained files still carry 31 references to the eight removed skills, including one dead markdown link and a retained agent eval that mounts a deleted skill. None of this is disclosed in the PR body, and the "Reference scan: 0 errors" line doesn't actually cover it.

What's proven correct ✅

• Config cleanup is complete for the 4 modified files — 0 leftover matches for any of the 8 skill names.
• All 8 skills retained on origin/staging — git cat-file -e origin/staging:<path> succeeded for all 8 SKILL.md.
• No new eval-quality errors — clean origin/main = 4 errors, PR HEAD = 3; the one that drops is maui-collectionview (removed here). The remaining 3 (maui-dependency-injection, maui-shell-navigation, maui-theming) are pre-existing on main and out of scope. This PR reduces the count.

Findings ⚠️

1. Retained agent eval mounts a deleted skill (highest severity). tests/dotnet-test/agent.testability-migration/eval.yaml:20 and :50 list ../../plugins/dotnet-test/skills/migrate-static-to-wrapper under environment.skills, but that directory is deleted. Probable/latent setup failure — if the harness skips missing mounts, the scenario is still semantically invalid since it no longer loads the skill it claims to test. (This file isn't in the diff, so it can't be inline-anchored.)
2. Dead markdown link in a shipped agent — plugins/dotnet-msbuild/agents/msbuild-code-review.agent.md:53 links to the deleted msbuild-antipatterns/SKILL.md.
3. Retained agents advertise removed skills — msbuild.agent.md / build-perf.agent.md list eval-performance / msbuild-antipatterns / check-bin-obj-clash as available; the review agent cites msbuild-antipatterns AP-13/14/21 by name.
4. Prose hand-offs dead-end — e.g. maui-data-binding/SKILL.md:38 → "use the maui-collectionview skill"; migrate-dotnet8-to-dotnet9/SKILL.md:30 → "use the migrate-dotnet9-to-dotnet10 skill". ~28 such references across the 17 files.
5. Misleading validation line — "Reference scan: 0 errors" is literally true but ReferenceScanner.cs only scans external URLs; it never validates cross-skill/relative links, so it gives no assurance about findings 1–4.

Not blockers

• The 3 remaining eval-quality errors are pre-existing maui exemptions — optional opportunistic cleanup.
• Scope is otherwise tight: only the 8 skills' trees + their exact config entries.

Verified empirically on a worktree at PR HEAD; cross-checked with a second model family. Inline notes below.

(Copilot, commenting on Abhitej's behalf.)

[AbhitejJohn]

Thanks for pulling all of this together, folks — the config cleanup itself is clean (I checked all four modified files and there are no leftover entries for the eight skills).

One gap though: we drop msbuild-antipatterns (and eval-performance / check-bin-obj-clash) here, but several retained msbuild artifacts still point at them. A few concrete ones:

• plugins/dotnet-msbuild/agents/msbuild-code-review.agent.md:53 has a hard markdown link [msbuild-antipatterns skill](../skills/msbuild-antipatterns/SKILL.md) — that target is deleted, so it's now a dead link.
• msbuild.agent.md and build-perf.agent.md still advertise eval-performance / msbuild-antipatterns / check-bin-obj-clash in their "available skills" lists, and the review agent leans on msbuild-antipatterns AP-13/AP-14/AP-21 by name.

Since these agents ship on main, they'll route the model to skills that no longer exist. The PR's "Reference scan: 0 errors" line only covers external URLs (that's all ReferenceScanner.cs checks), so it wouldn't catch any of this. Could we scrub or soften the inbound references in the same change? Happy to send a small pass against the branch if that helps.

(Copilot, commenting on Abhitej's behalf.)

[AbhitejJohn]

Heads-up while we're in this file: check_eval_quality.py is already red on main because these three (maui-dependency-injection, maui-shell-navigation, maui-theming) are stale underpowered exemptions that now clear the 5-trial floor. I verified it — clean origin/main reports 4 errors, this branch reports 3, and the one that drops is maui-collectionview (which this PR removes). So this PR isn't the cause and actually improves the count.

Since we're editing the allowlist anyway, want to drop these three stale lines too and get the gate green in the same change? None of them are in the eight-skill scope, so it's purely opportunistic — fine to skip if you'd rather keep this PR tightly scoped.

(Copilot, commenting on Abhitej's behalf.)

[AbhitejJohn]

Same inbound-reference nit on the upgrade side: plugins/dotnet-upgrade/skills/migrate-dotnet8-to-dotnet9/SKILL.md:30 still tells users to "use the migrate-dotnet9-to-dotnet10 skill as the next step", which we're removing here — so that hand-off now dead-ends on main. Curious whether we want to update it to point at migrate-dotnet10-to-dotnet11 or just drop the pointer while the skill is on staging.

(Copilot, commenting on Abhitej's behalf.)

[github-actions]

👋 @y87feng — this PR has changes requested,3 unresolved review thread(s),merge conflict. When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[steveisok]

I'd like to push back on staging android-tombstone-symbolication. I reproduced the eval and I think it's measuring the judge, not the skill.

The skill works. Against tombstone_sample.txt, the symbol server returns a real debug ELF and llvm-symbolizer resolves 0x222098 → ves_icall_System_Environment_FailFast (icall.c:6244). Verifiable ground truth.

Head-to-head on Opus 4.8:

• Baseline (no skill): resolved 0 of 7 frames. Tried three msdl URL shapes, all wrong, then asserted "public symbol servers returned 404 for that build-id" — which is false.
• Skilled: resolved 7 of 7, plus runtime version, commit SHA, and root cause.

A blind cross-family judge scored that as much-better for the failing baseline, on the grounds that the correct answer was "highly specific, unsupported." It confirmed it could not verify the symbols, could not check the false 404 claim, and could not detect fabrication. That's a −1.0 on the one stimulus where the skill is unambiguously right, and it explains the frontier-regressed flag on its own.

Eval defects:

• 6 of 8 fixture BuildIds are fabricated and 404 on msdl — only 2 stimuli are completable end-to-end.
• Skill-free Opus scores 3/3 on no_dotnet, no_buildid, and crashlog_ios in 38s, matching the skilled rubric verbatim. Those are coin flips, not evidence.
• reject_skills: ["*"] on the iOS stimulus is unfailable for baseline, failable only for skilled.
• runs is unset, so the sign test pools 8 different tasks as if they were repeated measures of one effect.
• The 6 deterministic output-matches graders separate skilled from baseline perfectly, but adapt.mjs derives wins/losses solely from the blind preference judge.

The label is also inconsistent with the harness. From #893's own numbers (~33 trials, 13L, 2T) that's ~18 wins, and adapt.mjs sets regressed only when losses > wins. The P0 rests on Impact 0.16 vs a 0.4 bar — but InvestigatingResults.md says "Do not read meanScore here… a verdict never turns on it." The artifacts from run 29228914412 have expired, so the judge reasoning behind those losses can't be re-read.

On usage: 30-day invocation counts are the wrong instrument for incident-driven crash triage. Its sibling apple-crash-symbolication is already on the underpowered allowlist — the whole symbolication category is being measured by something that can't see it.

Suggest dropping this skill from the PR and fixing the eval instead: real BuildIds in every fixture, defaults.runs >= 3, reweight or remove the guardrail stimuli baseline aces, and let the judge see the deterministic grader outcomes.

[AbhitejJohn]

Re: @steveisok's point that android-tombstone-symbolication measures the judge rather than the skill — I re-ran it cross-family on the latest main (5 executors, judge ≠ executor family) to regenerate the evidence, since the original run's artifacts had expired. The concern holds up.

Run: https://github.com/dotnet/skills/actions/runs/30862999210

Executor	Judge	W/T/L	Mean pref	Result
opus	claude-opus-4.8→gpt-5.5	3/1/3	+17.1%	inconclusive (1 unmatched trajectory)
gpt	gpt-5.5→claude-opus-4.8	3/0/5	−2.5%	no improvement (flat, p=0.36)
sonnet46	→claude-opus-4.8	5/1/1	+31.4%	inconclusive (1 unmatched trajectory)
haiku	→claude-opus-4.8	6/2/0	+52.5%	credibly better
mai	→claude-opus-4.8	5/0/2	+25.7%	inconclusive (1 unmatched trajectory)

Takeaways:

• No credible regression on current main. 4/5 executors are directionally positive (+17% to +52%); the only non-positive one (gpt) is statistically flat (−2.5%, sign-test p=0.36, 95% CI spans 0), and regressed=false on all five. The original "P0 FIX-REGRESSION" verdict does not reproduce — consistent with the gate fix in Eval gate is underpowered: 27% of verdicts flip on byte-identical inputs (A/A test); 53% of evals run at n<=3 #952/Fix the underpowered eval gate: score direction, not magnitude #965 ("score direction, not magnitude").
• The apparent fails are harness artifacts. opus/sonnet46/mai each flip to inconclusive solely because one trajectory couldn't be paired (unmatched), not because the skill lost. The sign test's unit is the stimulus (~8 of them), so it needs near-unanimity — a single dropped trajectory (8→7 pairs) decides the verdict. That's fragility in the measurement, not signal about the skill.
• There is a real, fixable eval bug. In the "no BuildId" stimulus the skilled run fabricated a BuildId borrowed from an unrelated fixture in the shared directory; the judge correctly flagged it as a dangerous hallucination. That's fixture cross-contamination, not a stable skill regression.
• In fairness to the eval: where the judge can verify ground truth, it does reward the skill — on the real symbolication tasks the skilled run downloaded correct symbols by BuildId and produced full function/line tables while baseline used the wrong URL, 404'd, and gave up. So the judge isn't blind; the verdict is just dominated by unmatched-trajectory fragility + fixture contamination + marginal presentation nitpicks (the judge repeatedly notes "both correct, gap is small").

Recommendation: drop android-tombstone-symbolication from this staging PR and fix the eval instead — dedupe fixtures/BuildIds so they can't cross-contaminate, drop the baseline-unfailable iOS guardrail stimulus, and raise power via more distinct stimuli + runs ≥ 3. Precision on a biased fixture just yields a precisely-wrong number, so the fixes should come before any re-measurement.

I'm re-running the other skills being staged in this PR to check whether any of them fail for the same measurement reasons, and will follow up with those results.

[AbhitejJohn]

Cross-family re-validation of the staged skills (latest main)

Re-ran the staged skills cross-family on latest main (5 executors — opus/gpt/sonnet46/haiku/mai — judge ≠ executor family). Runs: android-tombstone, other 6. Verified these still reflect current main (only intervening commit is #969, which touches none of these skills/evals).

Finding: most staging verdicts measure the eval instrument, not the skill. Tracking issue: #986.

Skill	Original	Re-run verdict	Re-category
android-tombstone-symbolication	P0 FIX-REGRESSION	4/5 positive, regressed=false	eval-defect (biased + underpowered) — fix in flight
check-bin-obj-clash	P1 TRIM-COST	n=1 → can't pass by construction	eval-defect (underpowered)
eval-performance	strengthen	n=1 → can't pass by construction	eval-defect (underpowered)
msbuild-antipatterns	P0	neutral; 3/4 stimuli off-target (F#)	eval-defect (construct-invalid)
migrate-static-to-wrapper	P0 FIX-REGRESSION	positive; sonnet46 credibly better, 0 regressions	keep — skill is helpful
migrate-dotnet9-to-dotnet10	P0 FIX-REGRESSION	helps haiku (+0.66); gpt dip is polish-only (both correct)	real but overstated → P2 rubric fix
dotnet-aot-compat	P1 TRIM-COST	n=1 → can't pass by construction	eval-defect (underpowered)
maui-collectionview	P0	fixed + revalidated in merged #947	stale — drop

Why the "fails" aren't regressions: three evals have a single stimulus (one trial → sign test p=0.5 always, can never pass); one amplifies marginal "both-correct" losses into a "regression" via direction-based sign test; one has fixture/grader/guardrail bias; one tests off-target scenarios. Details per plugin in #896 / #899 / #901 / #895.

Suggested changes to this PR

• Reframe from "shelve 8 P0/P1 skills" → "fix these evals and re-baseline." The defect is mostly the instrument.
• Drop: maui-collectionview (stale) and migrate-static-to-wrapper (skill is helpful).
• Keep skill, fix eval (don't stage on pass-rate): android-tombstone (in flight), check-bin-obj-clash, eval-performance, msbuild-antipatterns, dotnet-aot-compat.
• Only genuine follow-up: migrate-dotnet9-to-dotnet10 — and that's a rubric-tightening ("both correct ⇒ tie"), not a P0 shelve.
• If cost is the real driver for the TRIM-COST skills, state it explicitly and separately from the (invalid) pass-rate argument.

Net: of the 8, 0–1 warrant staging on the original premise; the rest are eval repairs tracked in #986.