Skip to content

Mature skill evaluation infrastructure - #21

Closed
PureWeen wants to merge 2 commits into
mainfrom
pureween-mature-skill-evaluations
Closed

Mature skill evaluation infrastructure#21
PureWeen wants to merge 2 commits into
mainfrom
pureween-mature-skill-evaluations

Conversation

@PureWeen

@PureWeen PureWeen commented Aug 18, 2026

Copy link
Copy Markdown
Owner

Why

ASP.NET Core already has useful skill evaluations, but the repository contract around them is implicit: contributors have to infer which files are auto-discovered, where fixtures belong, how to run validation locally, and which failures represent skill quality versus model/tool infrastructure.

This PR turns that implicit arrangement into a small, repeatable repository convention. The goal is not to add another skill or expand evaluation policy. It establishes the shared foundation that current and future skill evals can use: a standard one-skill baseline-vs-skilled lane, an explicit boundary for specialized suites, one pinned local entry point, and deterministic checks that do not call a model.

Relationship to dotnet#68625

Draft PR dotnet#68625 publishes fix-challenge/try-fix reviewer skills and specialized reviewer evaluations. This PR deliberately does not cherry-pick, modify, or depend on that draft. It is based on the existing evals on current main and defines the general repository convention first.

If this structure proves useful, dotnet#68625 can conform to it later:

  • one-skill generic A/B coverage lives at eng/skill-evals/<skill>/eval.vally.yaml and is auto-discovered;
  • reviewer/fix-challenge suites with different orchestration or scoring remain specialized *.vally.yaml files invoked explicitly;
  • fixtures stay beside the eval specs that consume them;
  • eval-only runners and checks stay under eng/skill-evals, not in runtime skill directories;
  • the same pinned Vally entry point and repository-layout checks apply without copying a reviewer-specific harness.

Keeping these changes separate lets the general infrastructure be evaluated on its own merits and avoids making the broader reviewer-skill draft the source of repository-wide conventions. This PR also does not claim to validate the quality of the reviewer skills in dotnet#68625; it only provides the topology and local tooling those evals can later use.

Summary

  • document the general eng/skill-evals/<skill>/eval.vally.yaml convention and specialized-suite boundary
  • add a cross-platform PowerShell entry point with exact @microsoft/vally-cli@0.13.0 identity
  • add parser-free repository layout checks and deterministic failure-injection/runner self-tests
  • validate experiment configuration with an isolated, model-free --compare --dry-run
  • migrate removed per-stimulus pairwise graders to Vally 0.13 experiment-level comparison
  • preserve each existing eval's effective 5 runs × 5 stimuli = 25 trials

Validation

  • pwsh -NoLogo -NoProfile -File eng/skill-evals/run.ps1 Test
  • pwsh -NoLogo -NoProfile -File eng/skill-evals/run.ps1 Validate
  • PowerShell parser checks
  • git diff --check

No hosted model or judge calls were made. Parser-dependent policy checks are explicitly documented as follow-ups rather than approximated with regex or prose heuristics.

Copilot AI added 2 commits August 18, 2026 12:44
Establish a pinned Vally entry point, parser-free repository validation, and deterministic runner self-tests while preserving the generic baseline experiment.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@PureWeen PureWeen closed this Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants