Skip to content

[Infra] Updates to eval pipeline and instructions #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ViktorHofer merged 2 commits into from
Feb 23, 2026
Merged

[Infra] Updates to eval pipeline and instructions #82

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
da62d8e
Update to eval pipeline and instructions
ViktorHofer Feb 23, 2026
2dbd61c
Update RUNS to 5 for main but 3 for PRs
ViktorHofer Feb 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions .github/workflows/evaluation.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,11 @@ on:
model:
description: 'Copilot model to use'
required: false
default: 'claude-opus-4.5'
default: 'claude-opus-4.6'
judge-model:
description: 'Judge model to use for evaluation'
required: false
default: 'claude-opus-4.5'
default: 'claude-opus-4.6'
runs:
description: 'Number of runs per test'
required: false
Expand Down Expand Up @@ -51,9 +51,10 @@ concurrency:
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DOTNET_VERSION: '10.0.x'
DEFAULT_MODEL: 'claude-opus-4.5'
DEFAULT_JUDGE_MODEL: 'claude-opus-4.5'
DEFAULT_RUNS: '3'
DEFAULT_MODEL: 'claude-opus-4.6'
DEFAULT_JUDGE_MODEL: 'claude-opus-4.6'
# Use more runs for main branch to get more stable benchmark data, but fewer runs for PRs to reduce quota pressure.
DEFAULT_RUNS: ${{ github.ref == 'refs/heads/main' && '5' || '3' }}
DEFAULT_PARALLEL_SKILLS: '3'
DEFAULT_PARALLEL_RUNS: '3'

Expand Down
7 changes: 5 additions & 2 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -224,12 +224,15 @@ node dist/index.js --tests-dir ./src/dotnet-msbuild/tests ./src/dotnet-msbuild/s
node dist/index.js --tests-dir ./src/dotnet-msbuild/tests ./src/dotnet-msbuild/skills/common-build-errors

# Fewer runs for faster iteration (default is 5)
node dist/index.js --runs 1 --tests-dir ./src/dotnet-msbuild/tests ./src/dotnet-msbuild/skills
node dist/index.js --runs 3 --tests-dir ./src/dotnet-msbuild/tests ./src/dotnet-msbuild/skills

# Use a specific model
node dist/index.js --model claude-sonnet-4.5 --tests-dir ./src/dotnet-msbuild/tests ./src/dotnet-msbuild/skills
node dist/index.js --model claude-opus-4.6 --tests-dir ./src/dotnet-msbuild/tests ./src/dotnet-msbuild/skills
```

> [!WARNING]
> If you share the results in a Pull Request, make sure to have `--runs` configured to at least 3 but better 5 for reliable results.

### CI evaluation

Tests run automatically on pull requests that modify files under `src/`. The evaluation workflow discovers changed components and runs the skill-validator for each one. Results are posted as a PR comment and uploaded as build artifacts.
Expand Down