Mature skill evaluation infrastructure - #21
Closed
PureWeen wants to merge 2 commits into
Closed
Conversation
Establish a pinned Vally entry point, parser-free repository validation, and deterministic runner self-tests while preserving the generic baseline experiment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
ASP.NET Core already has useful skill evaluations, but the repository contract around them is implicit: contributors have to infer which files are auto-discovered, where fixtures belong, how to run validation locally, and which failures represent skill quality versus model/tool infrastructure.
This PR turns that implicit arrangement into a small, repeatable repository convention. The goal is not to add another skill or expand evaluation policy. It establishes the shared foundation that current and future skill evals can use: a standard one-skill baseline-vs-skilled lane, an explicit boundary for specialized suites, one pinned local entry point, and deterministic checks that do not call a model.
Relationship to dotnet#68625
Draft PR dotnet#68625 publishes fix-challenge/try-fix reviewer skills and specialized reviewer evaluations. This PR deliberately does not cherry-pick, modify, or depend on that draft. It is based on the existing evals on current main and defines the general repository convention first.
If this structure proves useful, dotnet#68625 can conform to it later:
eng/skill-evals/<skill>/eval.vally.yamland is auto-discovered;*.vally.yamlfiles invoked explicitly;eng/skill-evals, not in runtime skill directories;Keeping these changes separate lets the general infrastructure be evaluated on its own merits and avoids making the broader reviewer-skill draft the source of repository-wide conventions. This PR also does not claim to validate the quality of the reviewer skills in dotnet#68625; it only provides the topology and local tooling those evals can later use.
Summary
eng/skill-evals/<skill>/eval.vally.yamlconvention and specialized-suite boundary@microsoft/vally-cli@0.13.0identity--compare --dry-runpairwisegraders to Vally 0.13 experiment-level comparisonValidation
pwsh -NoLogo -NoProfile -File eng/skill-evals/run.ps1 Testpwsh -NoLogo -NoProfile -File eng/skill-evals/run.ps1 Validategit diff --checkNo hosted model or judge calls were made. Parser-dependent policy checks are explicitly documented as follow-ups rather than approximated with regex or prose heuristics.