ci(bench): validate /benchmark suite against benchmarks main, not a stale pin

[pranavraja99]

Problem

/benchmark pinchbench26 --framework ironclaw-reborn is rejected with:

Unknown suite pinchbench26. Available: ironclaw, ironclaw-smoke, pinchbench, spot, swe-bench, terminal-bench, trajectory, zclaw-security-chn, zclaw-security-eng

…even though suites/pinchbench26.toml is on nearai/benchmarks main and the run would find it.

Root cause

The pre-dispatch suite validation in .github/workflows/nearai-bench.yml looks up the suite TOML at a hardcoded BENCH_PIN SHA:

BENCH_PIN: 67effacd8c8a7f6f43b422e30a1810822b9d6d5f
gh api "repos/nearai/benchmarks/contents/suites/${suite}.toml?ref=${BENCH_PIN}"

But the bench job that actually runs uses nearai/benchmarks/.github/workflows/bench-pr-reusable.yml@main — and its own comment says it intentionally tracks benchmarks main. The pin had drifted behind main, so the validation list is stale and rejects suites that already exist there: pinchbench26, officeqa, terminal-bench-2, etc. The check is supposed to "give the same answer the dispatched workflow would" — but it wasn't.

Fix

Point BENCH_PIN at main so the validation ref matches the ref we run against. They stay in lock-step, and new suites work the moment they land on benchmarks main — no manual pin bump per suite. Same trust boundary the run already accepts (uses: …@main), so no new supply-chain delta.

Verification

• pinchbench26, officeqa, terminal-bench-2 (all on benchmarks main) now pass the existence check.
• Unknown names still fail and list the current suites/ contents.

🤖 Generated with Claude Code

[gemini-code-assist]

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

• Chores
  • Updated benchmark workflow configuration to ensure suite validation uses the same reference as the actual benchmark execution, improving consistency in the validation process.

Walkthrough

BENCH_PIN in .github/workflows/nearai-bench.yml is changed from a hard-coded commit SHA to main. This aligns the pre-dispatch suite-existence gh api ref query against nearai/benchmarks@main with the ref used by the downstream bench-pr-reusable.yml@main.

Changes

BENCH_PIN Workflow Ref Update

Layer / File(s)	Summary
Suite validation ref alignment .github/workflows/nearai-bench.yml	BENCH_PIN changed from a pinned commit SHA to main; the gh api suite-contents lookup now queries nearai/benchmarks at main instead of a stale SHA, matching the ref used by the downstream reusable workflow.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

A SHA once stood guard, rigid and old,
Now main takes the watch, dynamic and bold.
The suite check and workflow now speak the same tongue,
No stale ref mismatch leaves validation unsung.
🦀 Aligned at last, the CI sings its song.

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	Description is detailed with Problem, Root cause, Fix, and Verification sections. However, required template fields (Summary bullets, Change Type checkbox, Linked Issue, Validation checklist, Security Impact) are entirely missing.	Add required template sections: Summary bullets, Change Type (check CI/Infrastructure), Linked Issue, Validation checklist items, and Security Impact statement. Consider filling Blast Radius and Review track fields.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	Title follows Conventional Commits style (type(scope): summary) and clearly summarizes the main change: updating BENCH_PIN from a stale hardcoded SHA to main.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/nearai-bench.yml:
- Around line 96-103: The BENCH_PIN environment variable set to main creates a
mutable reference that causes TOCTOU drift between pre-dispatch validation and
actual workflow execution in this privileged issue_comment dispatcher. Replace
BENCH_PIN from main with a full immutable commit SHA. Additionally, update the
reusable-workflow reference (the uses directive that currently specifies `@main`)
to use the same immutable commit SHA so that both the pre-dispatch authorization
check and the dispatched workflow execution are pinned to the identical
revision, eliminating the drift vulnerability.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: bcdb36b7-fea2-40cf-8b46-43e193dd4cd8

📥 Commits

Reviewing files that changed from the base of the PR and between 3c1ae10 and f185ecc.

📒 Files selected for processing (1)

• .github/workflows/nearai-bench.yml

[coderabbitai]

⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Use an immutable benchmarks ref in this privileged dispatcher path (Line 103).

BENCH_PIN: main makes pre-dispatch authorization/validation depend on a moving target, and it compounds with the mutable reusable-workflow ref at Line 219 (@main). In a privileged issue_comment workflow, this breaks immutability/provenance and reintroduces TOCTOU drift. Prefer a full commit SHA (or a bot-managed SHA bump process) and keep both validation and execution pinned to the same immutable revision.

As per coding guidelines, “.github/workflows/**: GitHub Actions hygiene… flag privileged workflows… unpinned third-party actions (pin full SHAs) …”.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/nearai-bench.yml around lines 96 - 103, The BENCH_PIN
environment variable set to main creates a mutable reference that causes TOCTOU
drift between pre-dispatch validation and actual workflow execution in this
privileged issue_comment dispatcher. Replace BENCH_PIN from main with a full
immutable commit SHA. Additionally, update the reusable-workflow reference (the
uses directive that currently specifies `@main`) to use the same immutable commit
SHA so that both the pre-dispatch authorization check and the dispatched
workflow execution are pinned to the identical revision, eliminating the drift
vulnerability.

Source: Coding guidelines