Skip to content

ci(bench): validate /benchmark suite against benchmarks main, not a stale pin #4947

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pranavraja99 merged 1 commit into from
Jun 16, 2026
Merged

ci(bench): validate /benchmark suite against benchmarks main, not a stale pin #4947

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions .github/workflows/nearai-bench.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,11 +93,14 @@ jobs:
PR: ${{ github.event.issue.number }}
REPO: ${{ github.repository }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Pinned reviewed commit on nearai/benchmarks main. Must match
# the SHA in the `bench` job's `uses:` below — bump in both
# places together. Used here to validate the requested suite
# exists at the same SHA we'd actually run against.
BENCH_PIN: 67effacd8c8a7f6f43b422e30a1810822b9d6d5f
# Ref on nearai/benchmarks used to validate the requested suite
# exists. Must match the ref the `bench` job actually runs against
# (its `uses:` below) so the pre-dispatch check gives the same
# answer as the dispatched workflow. That `uses:` is `@main`, so a
# pinned SHA here drifts stale and rejects suites that already exist
# on main (e.g. pinchbench26, officeqa, terminal-bench-2). Track
# `main` so the two stay in lock-step.
BENCH_PIN: main
Comment on lines +96 to +103

@coderabbitai coderabbitai Bot Jun 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Use an immutable benchmarks ref in this privileged dispatcher path (Line 103).

BENCH_PIN: main makes pre-dispatch authorization/validation depend on a moving target, and it compounds with the mutable reusable-workflow ref at Line 219 (@main). In a privileged issue_comment workflow, this breaks immutability/provenance and reintroduces TOCTOU drift. Prefer a full commit SHA (or a bot-managed SHA bump process) and keep both validation and execution pinned to the same immutable revision.

As per coding guidelines, “.github/workflows/**: GitHub Actions hygiene… flag privileged workflows… unpinned third-party actions (pin full SHAs) …”.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/nearai-bench.yml around lines 96 - 103, The BENCH_PIN
environment variable set to main creates a mutable reference that causes TOCTOU
drift between pre-dispatch validation and actual workflow execution in this
privileged issue_comment dispatcher. Replace BENCH_PIN from main with a full
immutable commit SHA. Additionally, update the reusable-workflow reference (the
uses directive that currently specifies `@main`) to use the same immutable commit
SHA so that both the pre-dispatch authorization check and the dispatched
workflow execution are pinned to the identical revision, eliminating the drift
vulnerability.

Source: Coding guidelines

run: |
set -euo pipefail
# Grammar: /benchmark <suite> [--model <model-id>] [--framework <ironclaw|ironclaw-reborn>]
Expand Down
Loading