Skip to content

Only run evals etc when relevant #1589

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
aantn wants to merge 5 commits into from
Closed

Only run evals etc when relevant #1589

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5f5d156
Skip evals, CLI benchmark, and Docker build for docs-only PRs
claude Feb 19, 2026
895f57f
Use check-changes job pattern instead of paths-ignore for docs-only skip
claude Feb 19, 2026
c369d52
Also exclude .github/** from code change detection
claude Feb 19, 2026
65f8336
Replace dorny/paths-filter with actions/github-script
claude Feb 19, 2026
b42d416
Add paths-ignore for .github/** to prevent triggering on CI-only changes
claude Feb 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 32 additions & 1 deletion .github/workflows/cli-performance.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,37 @@ name: CLI Performance Benchmark
on:
pull_request:
branches: [master]
paths-ignore:
- '.github/**'

jobs:
check-changes:
runs-on: ubuntu-latest
permissions:
pull-requests: read
outputs:
has_code_changes: ${{ steps.check.outputs.has_code_changes }}
steps:
- id: check
uses: actions/github-script@v7
with:
script: |
const files = await github.paginate(github.rest.pulls.listFiles, {
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.payload.pull_request.number,
});
const hasCodeChanges = files.some(f =>
!f.filename.startsWith('docs/') &&
!f.filename.endsWith('.md') &&
f.filename !== 'mkdocs.yml' &&
!f.filename.startsWith('.github/')
);
Comment on lines +29 to +34

@coderabbitai coderabbitai Bot Feb 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Excluding .github/ means changes to this workflow itself won't trigger the benchmark.

The filter on line 31 (!f.filename.startsWith('.github/')) causes modifications to this very workflow file to be treated as "no code changes," so the benchmark won't run. If someone refactors the benchmark script invocation or changes environment variables in this YAML, the workflow would silently skip itself.

If the intent is to only skip CI-config files unrelated to benchmarks, consider narrowing the exclusion (e.g., only skip .github/workflows/ files that aren't this one, or remove the .github/ exclusion from this workflow entirely).

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/cli-performance.yaml around lines 27 - 32, The current
hasCodeChanges predicate excludes any file under '.github/', so edits to this
workflow itself will be treated as "no code changes" and skip the benchmark;
update the files.some condition (the line using
!f.filename.startsWith('.github/')) so that either (a) you remove that exclusion
entirely in this workflow so changes to workflows trigger the benchmark, or (b)
narrow it to only skip unrelated CI config by replacing the broad '.github/'
check with a more specific check (e.g., only skip other workflow files by
checking for the workflows subdirectory and excluding a specific filename
exception) so this workflow file is not ignored by hasCodeChanges.

core.setOutput('has_code_changes', hasCodeChanges.toString());

benchmark-pr:
needs: check-changes
if: needs.check-changes.outputs.has_code_changes == 'true'
name: "Benchmark PR"
runs-on: ubuntu-latest
timeout-minutes: 15
Expand Down Expand Up @@ -59,6 +87,8 @@ jobs:
echo "startup_json=$(cat startup.json | jq -c)" >> "$GITHUB_OUTPUT"

benchmark-master:
needs: check-changes
if: needs.check-changes.outputs.has_code_changes == 'true'
name: "Benchmark Master"
runs-on: ubuntu-latest
timeout-minutes: 15
Expand Down Expand Up @@ -122,7 +152,8 @@ jobs:
compare:
name: "Compare & Report"
runs-on: ubuntu-latest
needs: [benchmark-pr, benchmark-master]
needs: [check-changes, benchmark-pr, benchmark-master]
if: needs.check-changes.outputs.has_code_changes == 'true'
permissions:
pull-requests: write

Expand Down
28 changes: 28 additions & 0 deletions .github/workflows/docker-dev-images.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,42 @@ name: Docker Build on PR
on:
pull_request:
types: [opened, synchronize, reopened]
paths-ignore:
- '.github/**'

# Cancel in-progress runs for the same PR
concurrency:
group: docker-build-${{ github.head_ref }}
cancel-in-progress: true

jobs:
check-changes:
runs-on: ubuntu-latest
permissions:
pull-requests: read
outputs:
has_code_changes: ${{ steps.check.outputs.has_code_changes }}
steps:
- id: check
uses: actions/github-script@v7
with:
script: |
const files = await github.paginate(github.rest.pulls.listFiles, {
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.payload.pull_request.number,
});
const hasCodeChanges = files.some(f =>
!f.filename.startsWith('docs/') &&
!f.filename.endsWith('.md') &&
f.filename !== 'mkdocs.yml' &&
!f.filename.startsWith('.github/')
);
core.setOutput('has_code_changes', hasCodeChanges.toString());

build:
needs: check-changes
if: needs.check-changes.outputs.has_code_changes == 'true'
runs-on: ubuntu-latest

permissions:
Expand Down
41 changes: 37 additions & 4 deletions .github/workflows/eval-regression.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ name: Run eval regression tests
on:
pull_request:
branches: ["*"]
paths-ignore:
- '.github/**'
push:
branches: [master]
workflow_dispatch:
Expand Down Expand Up @@ -41,6 +43,32 @@ permissions:
issues: write

jobs:
# Detect docs-only PRs to skip expensive eval jobs
check-changes:
if: github.event_name == 'pull_request'
runs-on: ubuntu-latest
permissions:
pull-requests: read
outputs:
has_code_changes: ${{ steps.check.outputs.has_code_changes }}
steps:
- id: check
uses: actions/github-script@v7
with:
script: |
const files = await github.paginate(github.rest.pulls.listFiles, {
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.payload.pull_request.number,
});
const hasCodeChanges = files.some(f =>
!f.filename.startsWith('docs/') &&
!f.filename.endsWith('.md') &&
f.filename !== 'mkdocs.yml' &&
!f.filename.startsWith('.github/')
);
core.setOutput('has_code_changes', hasCodeChanges.toString());

# Handle /list command - outputs available eval names
list_evals:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -118,12 +146,17 @@ jobs:
});

llm_evals:
needs: [check-changes]
runs-on: ubuntu-latest
if: |
github.event_name == 'pull_request' ||
github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'issue_comment' && github.event.issue.pull_request && (startsWith(github.event.comment.body, '/eval') || startsWith(github.event.comment.body, '/rerun')))
!cancelled() &&
needs.check-changes.outputs.has_code_changes != 'false' &&
(
github.event_name == 'pull_request' ||
github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'issue_comment' && github.event.issue.pull_request && (startsWith(github.event.comment.body, '/eval') || startsWith(github.event.comment.body, '/rerun')))
)

# Serialize automated regression runs per PR to avoid race conditions on persistent comment
# Manual runs (/eval, workflow_dispatch) get unique groups so they can run in parallel
Expand Down