diff --git a/eng/skill-evals/README.md b/eng/skill-evals/README.md new file mode 100644 index 000000000000..9c662718d230 --- /dev/null +++ b/eng/skill-evals/README.md @@ -0,0 +1,109 @@ +# Skill evaluations + +This directory contains evaluation-only assets for skills shipped from +`.github/skills`. Runtime skill instructions and references stay under +`.github/skills`; eval specifications, their fixtures, runners, and validators +stay here. + +## Layout and discovery + +`eng/skill-evals//eval.vally.yaml` is the standard one-skill lane, +auto-discovered by this repository's experiment and runner. +`skills-vs-baseline.experiment.yaml` runs each of those specs twice with the +same stimuli: once without a skill and once with exactly +`.github/skills/` loaded. The experiment owns `environment.skills`; +standard specs must not set it themselves. + +Any other `*.vally.yaml` file is a specialized suite. Specialized suites are not +auto-discovered and must be passed explicitly to the runner. Keep fixture files +beside the spec that consumes them, normally under a `fixtures` directory. Do +not place eval specs, `evals` directories, or eval runners in runtime skill +directories. A runtime skill may use a `fixtures` directory for non-eval assets. + +The experiment deliberately does not override `runs`. A standard spec owns its +trial count through `defaults.runs`. The existing specs retain five runs and 25 +trials each. The dotnet/skills quality gate uses five trials as the minimum at +which a clean sweep can support a one-sided sign test at 5%; enforcing a +trial-count policy here remains a documented follow-up. + +## Local entry point + +Run these commands from any directory: + +```powershell +# Safe default: deterministic checks with no model or judge calls +./eng/skill-evals/run.ps1 + +# Prove every validator rule and runner isolation with self-tests +./eng/skill-evals/run.ps1 Test + +# Strict Vally parsing/schema lint; does not call a model +./eng/skill-evals/run.ps1 Lint + +# Model-bearing operations are explicit and are not part of validation +./eng/skill-evals/run.ps1 Run +./eng/skill-evals/run.ps1 Run -Eval eng/skill-evals/review-public-api/eval.vally.yaml +./eng/skill-evals/run.ps1 Run -Eval eng/skill-evals//.vally.yaml +``` + +`Validate`, `Lint`, and `Run` use the exact +`@microsoft/vally-cli@0.13.0` package through `npx` and the Microsoft package +feed proxy. Pass `-Vally -VallyPrefix ` only to +intentionally override that invocation. The runner prints the resolved command +and reported version for provenance. Additional Vally arguments can be appended +to the command. If the package is not already cached, `npx` downloads that +exact version from the proxy; validation is model-free, not offline. It does not +install a package into the repository or modify dependency manifests. Run +output defaults to `artifacts/skill-evals`. + +Standard runs use Vally's experiment `--compare` mode. Vally 0.13 removed the +old per-stimulus `pairwise` grader, so comparison is owned by the experiment +rather than repeated in each eval spec. + +## Result interpretation and provenance + +An incomplete run, an unavailable model or judge, a timeout, an authentication +failure, or too few completed trials is an infrastructure/inconclusive result, +not evidence that the skill failed. A quality conclusion requires completed +baseline and skilled trials under the same inputs and identities. + +Retain the raw Vally output and enough provenance to reproduce a conclusion: + +- repository commit and whether the worktree was dirty; +- eval and experiment file paths and revisions; +- Vally version and full invocation; +- executor/tool identity and version; +- model and judge model identities; +- timestamps, run counts, retries, and incomplete trials. + +Defaults in each standard spec identify its model and judge. CLI overrides are +allowed for an intentional run, but the override and resulting identities must +remain in the saved provenance. Do not compare runs whose relevant identities +or inputs differ without calling out that difference. + +## Validation boundaries + +Pinned Vally owns YAML parsing, duplicate-key rejection, and eval/grader schema +validation. Default `Validate` also resolves the experiment with Vally +`--dry-run`, so experiment YAML, variants, and eval discovery are checked +without model or judge calls. The parser-free repository checks cover standard +eval-to-skill mapping, git-tracked eval specs and fixture trees, symlink-free +fixtures, and separation of eval specs and `evals` directories from runtime +skills. The self-tests inject every repository-layout failure class and +exercise runner dispatch without model calls. + +Checks that require interpreting eval YAML remain deferred until Vally exposes a +stable machine-readable validation contract or this repository has enough +demonstrated failures to justify a repo-native parser. These include +reference-specific fixture/path validation, trial-count policy, standard eval +skill-selection ownership, model/judge policy, and answer-material staging +rules. Do not approximate those checks with prose matching. + +Specialized suites own case promotion, consolidation, retirement, and held-out +refresh to keep coverage representative and bounded. + +Validation does not judge prompt or rubric quality, run a model, validate +runtime skill behavior, or decide whether a specialized suite is statistically +persuasive. Those concerns belong in skill-specific review and runtime +validation. Hosted execution, result publication, PR automation, and +cross-repository comparison adapters are deliberate follow-ups. diff --git a/eng/skill-evals/review-public-api/eval.vally.yaml b/eng/skill-evals/review-public-api/eval.vally.yaml index ffd599df0464..e5b6cb962a5b 100644 --- a/eng/skill-evals/review-public-api/eval.vally.yaml +++ b/eng/skill-evals/review-public-api/eval.vally.yaml @@ -46,7 +46,6 @@ stimuli: config: pattern: (?i)(parameter|cancellationtoken|IReadOnly|IDictionary|cachekey|\bid\b) - type: prompt - - type: pairwise rubric: - Recommends sealing the concrete component/options/DTO/context types unless extensibility is justified - 'Reasons about where the input data and the cacheKey actually come from (e.g. blob storage / database / uploads usually already provide a stable id), and concludes the required key is not real friction: keep it, make it easy to pass, and do NOT synthesize one expensively (e.g. hashing the whole stream)' @@ -89,7 +88,6 @@ stimuli: config: pattern: (?i)(seal|sealed|TimeSpan|init|IReadOnly|parameter|namespace) - type: prompt - - type: pairwise rubric: - Flags that LogLevel typed as int? lets callers pass magic numbers and should be a strongly-typed enum (the existing LogLevel enum) - Recommends sealing the configuration/options classes and renaming toward the Options convention (e.g. BrowserOptions) consistent with the rest of ASP.NET Core @@ -128,7 +126,6 @@ stimuli: config: pattern: (?i)(breaking|override|bookmark|stable) - type: prompt - - type: pairwise rubric: - Concludes the empty-string QueryParameterNamePrefix default is acceptable because one grid per page is the common case and multi-grid is a rarer opt-in scenario, and does NOT recommend auto-generating a prefix (which would defeat the bookmarkable/stable-URL goal) - Identifies that keying the sort query value on the column's display Title is fragile (renames and localization break shared/bookmarked URLs; title-less columns can't sort) and proposes a stable, opt-in column identifier with Title as fallback @@ -146,7 +143,6 @@ stimuli: config: pattern: (?i)(breaking|default|custom|convention) - type: prompt - - type: pairwise rubric: - Identifies that changing a shipped, on-by-default convention is a breaking change to existing behavior for every consumer - Recommends against modifying the built-in default convention @@ -173,7 +169,6 @@ stimuli: config: pattern: (?i)(default interface|default implementation|overload|breaking|binary) - type: prompt - - type: pairwise rubric: - Supports adding CancellationToken support to the async methods - Recognizes ITicketStore is a public interface with external implementers, so adding members outright is a binary-breaking change diff --git a/eng/skill-evals/run.ps1 b/eng/skill-evals/run.ps1 new file mode 100644 index 000000000000..f26a319289f0 --- /dev/null +++ b/eng/skill-evals/run.ps1 @@ -0,0 +1,281 @@ +#requires -Version 7.0 +param( + [Parameter(Position = 0)] + [ValidateSet('Validate', 'Test', 'Lint', 'Run')] + [string]$Action = 'Validate', + + [string]$Eval, + + [string]$OutputDirectory, + + [string]$Root, + + [string]$Vally = 'npx', + + [string[]]$VallyPrefix = @( + '--yes', + '--registry=https://packagefeedproxy.microsoft.io/npm/', + '@microsoft/vally-cli@0.13.0' + ), + + [Parameter(ValueFromRemainingArguments = $true)] + [string[]]$Arguments +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version 3 + +$repoRoot = if ($Root) { + (Resolve-Path $Root).Path +} else { + (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path +} +$evalRoot = Join-Path $repoRoot 'eng/skill-evals' +$runtimeRoot = Join-Path $repoRoot '.github/skills' +$experiment = Join-Path $evalRoot 'skills-vs-baseline.experiment.yaml' +$script:vallyInitialized = $false + +function Resolve-Eval { + if (-not $Eval) { + return $null + } + + $candidate = if ([IO.Path]::IsPathRooted($Eval)) { + $Eval + } else { + Join-Path $repoRoot $Eval + } + return (Resolve-Path $candidate).Path +} + +function Get-EvalSpecs { + $resolvedEval = Resolve-Eval + if ($resolvedEval) { + return @($resolvedEval) + } + return @( + Get-ChildItem $evalRoot -Recurse -File -Filter '*.vally.yaml' | + ForEach-Object FullName + ) +} + +function Invoke-Vally { + param([string[]]$VallyArguments) + + $command = Get-Command $Vally -ErrorAction SilentlyContinue + if (-not $command) { + throw "Vally command '$Vally' was not found." + } + + if (-not $script:vallyInitialized) { + $global:LASTEXITCODE = 0 + $versionOutput = & $Vally @VallyPrefix --version + $versionSucceeded = $? + $versionExitCode = $LASTEXITCODE + if (-not $versionSucceeded -or $versionExitCode -ne 0) { + throw "Vally version probe failed with exit code $versionExitCode." + } + + $identity = ($versionOutput | Out-String).Trim() + $commandPath = if ($command.Path) { $command.Path } else { $Vally } + $invocation = (@($commandPath) + $VallyPrefix) -join ' ' + Write-Host "Using Vally: $invocation (reported version: $identity)" + $script:vallyInitialized = $true + } + + $global:LASTEXITCODE = 0 + & $Vally @VallyPrefix @VallyArguments + $invocationSucceeded = $? + $invocationExitCode = $LASTEXITCODE + if (-not $invocationSucceeded -or $invocationExitCode -ne 0) { + if ($invocationExitCode -eq 0) { + $invocationExitCode = 1 + } + exit $invocationExitCode + } +} + +function Invoke-VallyLint { + $specs = @(Get-EvalSpecs) + if ($specs.Count -eq 0) { + throw "No Vally eval specifications were found under '$evalRoot'." + } + + foreach ($spec in $specs) { + $vallyArguments = @('lint', '--strict', $runtimeRoot, '--eval-spec', $spec) + $vallyArguments += $Arguments + Invoke-Vally $vallyArguments + } +} + +function Invoke-VallyIsolated { + param([string[]]$VallyArguments) + + $workDirectory = Join-Path ([IO.Path]::GetTempPath()) ( + 'aspnetcore-skill-evals-' + [guid]::NewGuid().ToString('N') + ) + New-Item -ItemType Directory -Path $workDirectory | Out-Null + try { + Push-Location $workDirectory + Invoke-Vally $VallyArguments + } finally { + Pop-Location + Remove-Item -Recurse -Force $workDirectory + } +} + +function Invoke-VallyExperimentDryRun { + Invoke-VallyIsolated @( + 'experiment', + 'run', $experiment, + '--compare', + '--dry-run', + '--output-dir', 'plan-results' + ) +} + +function Get-TrackedFiles { + $global:LASTEXITCODE = 0 + $files = @(& git -C $repoRoot -c core.quotepath=false ls-files) + if ($LASTEXITCODE -ne 0) { + throw "Could not read tracked files from '$repoRoot'." + } + return [Collections.Generic.HashSet[string]]::new( + [string[]]($files | ForEach-Object { $_.Replace('\', '/') }) + ) +} + +function Test-IsLink { + param([IO.FileSystemInfo]$Item) + + return $null -ne $Item.LinkType -or ( + ($Item.Attributes -band [IO.FileAttributes]::ReparsePoint) -ne 0 + ) +} + +function Get-LayoutErrors { + $errors = [Collections.Generic.List[string]]::new() + if (-not (Test-Path $experiment -PathType Leaf)) { + $errors.Add("Missing standard experiment: $experiment") + } + + $standardSpecs = @( + Get-ChildItem $evalRoot -Directory -ErrorAction SilentlyContinue | + ForEach-Object { Join-Path $_.FullName 'eval.vally.yaml' } | + Where-Object { Test-Path $_ -PathType Leaf } + ) + if ($standardSpecs.Count -eq 0) { + $errors.Add("No standard */eval.vally.yaml specs were found under '$evalRoot'.") + } + foreach ($spec in $standardSpecs) { + $skillName = Split-Path (Split-Path $spec -Parent) -Leaf + $skillFile = Join-Path $runtimeRoot "$skillName/SKILL.md" + if (-not (Test-Path $skillFile -PathType Leaf)) { + $errors.Add("$spec has no matching runtime skill at '$skillFile'.") + } + } + + if (Test-Path $runtimeRoot -PathType Container) { + foreach ($spec in Get-ChildItem $runtimeRoot -Recurse -File -Filter '*.vally.yaml') { + $errors.Add("$($spec.FullName) is eval-only and belongs under '$evalRoot'.") + } + foreach ($directory in Get-ChildItem $runtimeRoot -Recurse -Directory) { + if ($directory.Name -eq 'evals') { + $errors.Add("$($directory.FullName) is eval-only and belongs under '$evalRoot'.") + } + } + } + + $tracked = Get-TrackedFiles + foreach ($spec in Get-ChildItem $evalRoot -Recurse -File -Filter '*.vally.yaml') { + $relative = [IO.Path]::GetRelativePath($repoRoot, $spec.FullName).Replace('\', '/') + if (-not $tracked.Contains($relative)) { + $errors.Add("$relative is not tracked by git and will not exist in CI.") + } + } + + foreach ($fixtures in Get-ChildItem $evalRoot -Recurse -Directory | + Where-Object Name -eq 'fixtures') { + $items = @(Get-ChildItem $fixtures.FullName -Recurse -Force) + $fixtureFiles = @($items | Where-Object { -not $_.PSIsContainer }) + if ($fixtureFiles.Count -eq 0) { + $errors.Add("$($fixtures.FullName) is empty and cannot be represented in git.") + continue + } + foreach ($item in @($fixtures) + $items) { + if (Test-IsLink $item) { + $errors.Add("$($item.FullName) is a symlink; eval fixtures must be self-contained.") + } + if (-not $item.PSIsContainer) { + $relative = [IO.Path]::GetRelativePath( + $repoRoot, + $item.FullName + ).Replace('\', '/') + if (-not $tracked.Contains($relative)) { + $errors.Add("$relative is not tracked by git and will not exist in CI.") + } + } + } + } + return $errors +} + +function Invoke-LayoutValidation { + $errors = @(Get-LayoutErrors) + if ($errors.Count -gt 0) { + throw "Skill-eval layout validation failed:`n - $($errors -join "`n - ")" + } + Write-Host 'Skill-eval layout validation passed.' +} + +switch ($Action) { + 'Validate' { + Invoke-LayoutValidation + Invoke-VallyLint + Invoke-VallyExperimentDryRun + } + 'Test' { + & (Join-Path $PSScriptRoot 'test_validate.ps1') + & (Join-Path $PSScriptRoot 'test_run.ps1') + } + 'Lint' { + Invoke-VallyLint + } + 'Run' { + $resolvedEval = Resolve-Eval + $output = if ($OutputDirectory) { + if ([IO.Path]::IsPathRooted($OutputDirectory)) { + $OutputDirectory + } else { + Join-Path $repoRoot $OutputDirectory + } + } else { + Join-Path $repoRoot 'artifacts/skill-evals' + } + + if (-not $resolvedEval -or (Split-Path $resolvedEval -Leaf) -eq 'eval.vally.yaml') { + $vallyArguments = @( + 'experiment', + 'run', $experiment, + '--compare', + '--output-dir', $output + ) + if ($resolvedEval) { + $relativeEval = [IO.Path]::GetRelativePath( + $evalRoot, + $resolvedEval + ).Replace('\', '/') + $vallyArguments += @('--eval-filter', $relativeEval) + } + } else { + $vallyArguments = @( + 'eval', + '--eval-spec', $resolvedEval, + '--output-dir', $output + ) + } + + $vallyArguments += $Arguments + Invoke-VallyIsolated $vallyArguments + } +} diff --git a/eng/skill-evals/skills-vs-baseline.experiment.yaml b/eng/skill-evals/skills-vs-baseline.experiment.yaml index 08f3453b26cb..f7dfbed8a6fa 100644 --- a/eng/skill-evals/skills-vs-baseline.experiment.yaml +++ b/eng/skill-evals/skills-vs-baseline.experiment.yaml @@ -15,9 +15,6 @@ name: skills-vs-baseline evals: - "*/eval.vally.yaml" -overrides: - runs: 5 - vary: - /environment/skills diff --git a/eng/skill-evals/test_run.ps1 b/eng/skill-evals/test_run.ps1 new file mode 100644 index 000000000000..0b1554a973e6 --- /dev/null +++ b/eng/skill-evals/test_run.ps1 @@ -0,0 +1,114 @@ +#requires -Version 7.0 + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version 3 + +$runner = Join-Path $PSScriptRoot 'run.ps1' +$repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path +$testRoot = Join-Path ([IO.Path]::GetTempPath()) ( + 'aspnetcore-skill-eval-runner-test-' + [guid]::NewGuid().ToString('N') +) +$fakeVally = Join-Path $testRoot 'fake-vally.ps1' +$record = Join-Path $testRoot 'invocation.txt' +$specialized = Join-Path $testRoot 'specialized.vally.yaml' +$output = Join-Path $testRoot 'results' +$relativeOutput = 'artifacts/skill-eval-runner-selftest' + +function Assert-True { + param([bool]$Condition, [string]$Message) + + if (-not $Condition) { + throw $Message + } +} + +function Read-Invocation { + $lines = @(Get-Content $record) + Assert-True ($lines.Count -gt 1) 'Fake Vally did not capture an invocation.' + Assert-True ($lines[0] -ne $repoRoot) 'Run used the repository as its working directory.' + Assert-True (-not (Test-Path $lines[0])) 'The isolated working directory was not removed.' + return $lines +} + +New-Item -ItemType Directory -Path $testRoot | Out-Null +try { + @' +if ($args -contains '--version') { + Write-Output 'fake-vally 1.0' + return +} +if ($env:SKILL_EVAL_FAKE_FAILURE) { + throw 'fake Vally failure' +} +[IO.File]::WriteAllLines( + $env:SKILL_EVAL_RUNNER_RECORD, + @((Get-Location).Path) + [string[]]$args +) +'@ | Set-Content $fakeVally + Set-Content $specialized "name: specialized`n" + $env:SKILL_EVAL_RUNNER_RECORD = $record + + Push-Location $testRoot + try { + & $runner -Vally $fakeVally -VallyPrefix @() + } finally { + Pop-Location + } + $validateInvocation = Read-Invocation + Assert-True ($validateInvocation[1] -eq 'experiment') ( + 'Default validation did not resolve the experiment.' + ) + Assert-True ($validateInvocation[2] -eq 'run') ( + 'Default validation did not use Vally experiment run.' + ) + Assert-True ($validateInvocation -contains '--dry-run') ( + 'Default validation could invoke models.' + ) + Assert-True ($validateInvocation -contains '--compare') ( + 'Default validation did not resolve comparison mode.' + ) + + & $runner Run ` + -Eval eng/skill-evals/review-public-api/eval.vally.yaml ` + -Vally $fakeVally ` + -VallyPrefix @() ` + -OutputDirectory $relativeOutput ` + '--workers' '2' + $standardInvocation = Read-Invocation + Assert-True ($standardInvocation[1] -eq 'experiment') 'Standard run did not use Vally experiment.' + Assert-True ($standardInvocation[2] -eq 'run') 'Standard run did not use Vally experiment run.' + Assert-True ($standardInvocation -contains '--compare') 'Standard run omitted A/B comparison.' + Assert-True ($standardInvocation -contains '--eval-filter') 'Standard run omitted --eval-filter.' + Assert-True ($standardInvocation -contains 'review-public-api/eval.vally.yaml') ( + 'Standard run did not resolve the repository-relative eval path.' + ) + Assert-True ($standardInvocation -contains '--workers') 'Run did not forward additional arguments.' + Assert-True ($standardInvocation -contains '2') 'Run did not forward the workers value.' + Assert-True ($standardInvocation -contains (Join-Path $repoRoot $relativeOutput)) ( + 'Run did not resolve the repository-relative output path.' + ) + + & $runner Run ` + -Eval $specialized ` + -Vally $fakeVally ` + -VallyPrefix @() ` + -OutputDirectory $output + $specializedInvocation = Read-Invocation + Assert-True ($specializedInvocation[1] -eq 'eval') 'Specialized run did not use Vally eval.' + Assert-True ($specializedInvocation -contains '--eval-spec') 'Specialized run omitted --eval-spec.' + + $env:SKILL_EVAL_FAKE_FAILURE = 'true' + $failurePropagated = $false + try { + & $runner Run -Vally $fakeVally -VallyPrefix @() -OutputDirectory $output + } catch { + $failurePropagated = $_.Exception.Message -like '*fake Vally failure*' + } + Assert-True $failurePropagated 'Vally invocation failure was not propagated.' + + Write-Host 'Skill-eval runner self-test passed.' +} finally { + Remove-Item Env:SKILL_EVAL_RUNNER_RECORD -ErrorAction SilentlyContinue + Remove-Item Env:SKILL_EVAL_FAKE_FAILURE -ErrorAction SilentlyContinue + Remove-Item -Recurse -Force $testRoot +} diff --git a/eng/skill-evals/test_validate.ps1 b/eng/skill-evals/test_validate.ps1 new file mode 100644 index 000000000000..9463b7f80788 --- /dev/null +++ b/eng/skill-evals/test_validate.ps1 @@ -0,0 +1,141 @@ +#requires -Version 7.0 + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version 3 + +$runner = Join-Path $PSScriptRoot 'run.ps1' +$testRoot = Join-Path ([IO.Path]::GetTempPath()) ( + 'aspnetcore-skill-eval-layout-test-' + [guid]::NewGuid().ToString('N') +) + +function Write-TestRepository { + param([string]$Root) + + $eval = Join-Path $Root 'eng/skill-evals/widget' + $skill = Join-Path $Root '.github/skills/widget' + New-Item -ItemType Directory -Path "$eval/fixtures/sample" -Force | Out-Null + New-Item -ItemType Directory -Path $skill -Force | Out-Null + Set-Content "$skill/SKILL.md" "---`nname: widget`ndescription: Widget.`n---`n" + Set-Content "$eval/fixtures/sample/input.txt" "input" + Set-Content "$eval/eval.vally.yaml" @' +name: widget +type: capability +defaults: + runs: 5 + model: test-model + judge_model: test-judge +stimuli: + - name: widget + prompt: Explain the widget. + rubric: + - Explains the widget +'@ + Set-Content (Join-Path $Root 'eng/skill-evals/skills-vs-baseline.experiment.yaml') @' +name: skills-vs-baseline +evals: + - "*/eval.vally.yaml" +vary: + - /environment/skills +baseline: baseline +variants: + baseline: + environment: + skills: [] + skilled: + environment: + skills: + - "../../.github/skills/${eval.parent}" +'@ + & git -C $Root init -q + & git -C $Root config user.email skill-evals@example.invalid + & git -C $Root config user.name 'Skill eval self-test' + & git -C $Root add -A + & git -C $Root commit -qm baseline +} + +function Invoke-Case { + param( + [string]$Name, + [scriptblock]$Mutation, + [string]$Expected, + [bool]$ShouldFail = $true + ) + + $root = Join-Path $testRoot ([guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $root | Out-Null + Write-TestRepository $root + & $Mutation $root + $failed = $false + $message = '' + try { + & $runner Validate -Root $root -Vally $fakeVally -VallyPrefix @() + } catch { + $failed = $true + $message = $_.Exception.Message + } + $passed = $failed -eq $ShouldFail -and ( + -not $Expected -or $message -like "*$Expected*" + ) + if (-not $passed) { + throw "$Name failed. Expected failure=$ShouldFail containing '$Expected'; got '$message'." + } + Write-Host " [OK] $Name" +} + +New-Item -ItemType Directory -Path $testRoot | Out-Null +$fakeVally = Join-Path $testRoot 'fake-vally.ps1' +@' +if ($args -contains '--version') { + Write-Output 'fake-vally 1.0' +} +'@ | Set-Content $fakeVally + +try { + Invoke-Case 'clean repository' {} '' $false + Invoke-Case 'missing experiment' { + param($root) + Remove-Item "$root/eng/skill-evals/skills-vs-baseline.experiment.yaml" + } 'Missing standard experiment' + Invoke-Case 'missing runtime skill' { + param($root) + Remove-Item -Recurse "$root/.github/skills/widget" + } 'no matching runtime skill' + Invoke-Case 'eval spec in runtime skill' { + param($root) + Set-Content "$root/.github/skills/widget/eval.vally.yaml" 'name: wrong' + } 'is eval-only' + Invoke-Case 'evals directory in runtime skill' { + param($root) + New-Item -ItemType Directory "$root/.github/skills/widget/evals" | Out-Null + } 'is eval-only' + Invoke-Case 'runtime fixture directory is allowed' { + param($root) + New-Item -ItemType Directory "$root/.github/skills/widget/fixtures" | Out-Null + Set-Content "$root/.github/skills/widget/fixtures/runtime.txt" 'runtime asset' + } '' $false + Invoke-Case 'untracked eval spec' { + param($root) + Set-Content "$root/eng/skill-evals/widget/specialized.vally.yaml" 'name: specialized' + } 'is not tracked by git' + Invoke-Case 'untracked fixture' { + param($root) + Set-Content "$root/eng/skill-evals/widget/fixtures/sample/untracked.txt" 'input' + } 'is not tracked by git' + Invoke-Case 'empty fixture directory' { + param($root) + Remove-Item "$root/eng/skill-evals/widget/fixtures/sample/input.txt" + } 'is empty' + + if (-not $IsWindows) { + Invoke-Case 'symlinked fixture' { + param($root) + New-Item -ItemType SymbolicLink ` + -Path "$root/eng/skill-evals/widget/fixtures/sample/link.txt" ` + -Target "$root/eng/skill-evals/widget/fixtures/sample/input.txt" | Out-Null + } 'is a symlink' + } + + Write-Host 'Skill-eval layout self-tests passed.' +} finally { + Remove-Item -Recurse -Force $testRoot +} diff --git a/eng/skill-evals/validate-blazor-feature/eval.vally.yaml b/eng/skill-evals/validate-blazor-feature/eval.vally.yaml index 5a5b723160a9..68a94678cc22 100644 --- a/eng/skill-evals/validate-blazor-feature/eval.vally.yaml +++ b/eng/skill-evals/validate-blazor-feature/eval.vally.yaml @@ -29,7 +29,6 @@ stimuli: config: pattern: '(?i)@rendermode' - type: prompt - - type: pairwise rubric: - "Directs the user to one of the in-repo Blazor Web App samples (BlazorWebAppPerPage or BlazorWebAppGlobal), not a freshly scaffolded app" - "Says a WebAssembly (or Auto) component must live in the .Client project so it compiles into the client assembly" @@ -49,7 +48,6 @@ stimuli: config: pattern: '(?i)(App\.razor|Routes|HeadOutlet|rendermode)' - type: prompt - - type: pairwise rubric: - "Explains that a global app sets the render mode in the host App.razor on the Routes (and HeadOutlet) components" - "Says to change the single @rendermode value from InteractiveAuto to InteractiveServer" @@ -69,7 +67,6 @@ stimuli: config: pattern: '(?i)(static|SSR|circuit|websocket|_blazor|webassembly)' - type: prompt - - type: pairwise rubric: - "Says that rendered markup alone does not prove interactivity because static SSR emits the same HTML" - "Requires a behavioral check: interact (click the button) and assert the displayed state actually changes (e.g. the count increments)" @@ -87,7 +84,6 @@ stimuli: config: pattern: '(?i)(stopped|previous|other port|not.*running|refused)' - type: prompt - - type: pairwise rubric: - "Identifies the errors as stale/false positives from a previously-stopped server on a different port, not a defect in the app under test" - "Advises scoping console checks to the current page rather than dumping the whole session history" @@ -104,7 +100,6 @@ stimuli: config: pattern: '(?i)(rendermode|render mode|interactiv)' - type: prompt - - type: pairwise rubric: - "Identifies that the component is rendering as static SSR, i.e. no interactive render mode is actually applied" - "Says to add/verify the @rendermode (per page, or on Routes/HeadOutlet for a global app)"