Mature skill evaluation infrastructure

[PureWeen]

Why

ASP.NET Core is starting to ship repository-specific Copilot skills, but we do not yet have a shared way to prove that a skill improves agent behavior or continues working as the repository changes. The existing evals are useful, but their layout, discovery rules, tool version, comparison behavior, and local validation path are implicit. Each new skill would otherwise need to invent those decisions again, and results could silently depend on different tooling or mix model infrastructure failures with actual skill regressions.

This PR makes skill evaluation a small repository-level contract. It defines one standard baseline-vs-skilled lane, keeps specialized suites explicit, pins the Vally version used to interpret the specs, and provides a safe local validation path that does not call a model. It also keeps eval-only fixtures and tooling separate from the runtime skills agents actually load.

That gives maintainers a repeatable way to review and evolve repository skills, and gives future agentic work a common foundation instead of a collection of one-off harnesses.

Relationship to #68625

Draft PR #68625 publishes fix-challenge/try-fix reviewer skills and specialized reviewer evaluations. This PR deliberately does not cherry-pick, modify, or depend on that draft. It is based on the existing evals on current main and defines the general repository convention first.

If this structure proves useful, #68625 can conform to it later:

• one-skill generic A/B coverage lives at eng/skill-evals/<skill>/eval.vally.yaml and is auto-discovered by the repository experiment and runner;
• reviewer/fix-challenge suites with different orchestration or scoring remain specialized *.vally.yaml files invoked explicitly;
• fixtures stay beside the eval specs that consume them;
• eval-only runners and checks stay under eng/skill-evals, not in runtime skill directories;
• specialized suites own case promotion, consolidation, retirement, and held-out refresh so their active corpora remain representative and bounded;
• the same pinned Vally entry point and repository-layout checks apply without copying a reviewer-specific harness.

Keeping these changes separate lets the general infrastructure be evaluated on its own merits and avoids making the broader reviewer-skill draft the source of repository-wide conventions. This PR also does not claim to validate the quality of the reviewer skills in #68625; it only provides the topology and local tooling those evals can later use.

Summary

• document the general eng/skill-evals/<skill>/eval.vally.yaml convention and specialized-suite boundary
• add a cross-platform PowerShell entry point with exact @microsoft/vally-cli@0.13.0 identity
• add parser-free repository layout checks and deterministic failure-injection/runner self-tests
• validate experiment configuration with an isolated, model-free --compare --dry-run
• migrate removed per-stimulus pairwise graders to Vally 0.13 experiment-level comparison
• preserve each existing eval's effective 5 runs × 5 stimuli = 25 trials
• keep specialized eval corpora bounded through suite-owned case lifecycle policy

Validation

• pwsh -NoLogo -NoProfile -File eng/skill-evals/run.ps1 Test
• pwsh -NoLogo -NoProfile -File eng/skill-evals/run.ps1 Validate
• PowerShell parser checks
• git diff --check

No hosted model or judge calls were made. Parser-dependent policy checks are explicitly documented as follow-ups rather than approximated with regex or prose heuristics.

[Copilot]

Pull request overview

Establishes a repository-level contract and tooling for evaluating ASP.NET Core Copilot skills using a standard baseline-vs-skilled Vally experiment lane, with a pinned Vally CLI version and a model-free local validation workflow.

Changes:

• Standardizes discovery around eng/skill-evals/<skill>/eval.vally.yaml and a shared skills-vs-baseline.experiment.yaml A/B experiment.
• Adds a cross-platform PowerShell runner (run.ps1) plus self-tests for layout validation and runner dispatch without model calls.
• Updates existing eval specs to remove the deprecated per-stimulus pairwise grader (now owned by experiment-level --compare).

Show a summary per file

File	Description
eng/skill-evals/validate-blazor-feature/eval.vally.yaml	Removes per-stimulus pairwise grader to align with experiment-level comparison.
eng/skill-evals/review-public-api/eval.vally.yaml	Removes per-stimulus pairwise grader to align with experiment-level comparison.
eng/skill-evals/skills-vs-baseline.experiment.yaml	Keeps comparison centralized; removes experiment-level runs override so specs own trial count.
eng/skill-evals/run.ps1	Adds the pinned Vally entry point, layout validation, linting, dry-run experiment validation, and run dispatch.
eng/skill-evals/README.md	Documents the eval layout contract, standard vs specialized suite boundary, and local validation/run workflow.
eng/skill-evals/test_validate.ps1	Adds deterministic layout validation self-tests via failure injection.
eng/skill-evals/test_run.ps1	Adds runner isolation/dispatch self-test using a fake Vally command.

Review details

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 7/7 changed files
• Comments generated: 1
• Review effort level: Lite

[maraf]

Looks good to me 👍