ci(bench-gate): run the tier at the release cut, and report it per module (#1477)

[robotrocketscience]

Closes #1477. Refs #1456 (AC3).

What was actually wrong

#1456 records that the bench-gate quality tier "runs lab-side only". It does not
run anywhere.

• The private lab repository has no .github/workflows directory at all. So
  "add a lab-side job" had no host to be added to — it is standing new
  infrastructure, not a job.
• scripts/run_bench_gate.sh exists on main, defaults AELFRICE_CORPUS_ROOT
  to the lab corpus, and is invoked by nothing.
• The corpus covers a minority of the modules scaffolded under tests/corpus/,
  so even a job built today would leave most of the tier skipping for want of
  rows.

Several defaults are held OFF pending exactly these verdicts. A gate that never
runs cannot deliver one, so those defaults are parked on a measurement that was
not scheduled to happen.

Where it runs now

Operator ruling 2026-08-11: at the release cut, and nowhere else.
docs/concepts/RELEASING.md step 7 makes scripts/run_bench_gate.sh mandatory
and its output part of the release PR body.

That is the only moment the verdicts are consumed, and a checklist step that
blocks the cut is the one schedule that cannot die unnoticed — which is exactly
the #1423 failure mode a weekly cron on one machine would have reproduced. The
corpus is in any case too thin for a weekly cadence to produce a new verdict.

CONTRIBUTING.md's bench-gate section said the tier runs lab-side. It now says
where it actually runs.

Why the aggregate count was not enough (#1477 AC3)

#1456's aggregate is the right answer while the corpus is absent: with nothing
to run against, "36 skipped" is the whole story. It is the wrong answer the
moment a corpus exists — a run reports a healthy "N passed" while most of the
tier skipped for want of rows, which is the same misreading #1456 closed, one
level in.

The bench-gate tier block now separates three states and names them:

• tests that executed against the corpus — the only verdict;
• tests skipped because a named corpus module is missing or empty, listed
  per module with which of the two it was (they are different failures: no rows
  delivered, versus a module that exists and is dead);
• the whole tier skipped for want of a corpus root.

Classification is by skip reason, not by re-deriving which tests carry the
marker, so an unrelated skip inside a bench-gated module is not folded into the
missing-rows figure — inflating it is the direction that makes the tier look
more blocked than it is, so nobody would chase it. Executed tests are counted
off the marker, which is the only place that signal survives to summary time.

How it is tested

Driven against the hook with constructed reports rather than by running the tier
under a fixture corpus. The states that need pinning include ones this
repository cannot produce — a module present and non-empty, so its gate actually
ran — and a test that could only assert the locally reachable states would pin
exactly the case that was never in doubt.

Verified against a real run as well: with a stub corpus holding one module, the
block names every other module as having no verdict.

mutation	tests that go red
drop the per-module lines (aggregate only)	3
count executed tests without reading the marker	2
collapse missing and empty into one bucket	2

Out of scope

AC1 and AC2 as literally worded — a lab-side job on a stated cadence, and where
a public reader is pointed — are discharged by the ruling rather than by code:
the run is at the cut, and its output lands in the release PR, which is public.
Funding corpus rows for the modules that have none is a separate decision and is
not taken here.

Summary by Sourcery

Run the bench-gate quality tier at release time and expand its reporting so release PRs clearly show which modules did and did not produce corpus-backed verdicts.

Enhancements:

• Extend the pytest terminal summary hook to classify bench-gate outcomes by skip reason, counting executed tests separately and listing per-module corpus issues instead of only an aggregate skip count.

CI:

• Make running scripts/run_bench_gate.sh a mandatory step in the release process and require pasting its bench-gate summary block into the release PR body.

Documentation:

• Clarify CONTRIBUTING and releasing docs to state that the bench-gate tier runs only at the release cut, and explain how to interpret its per-module summary block.

Tests:

• Add tests for the pytest terminal summary hook to ensure bench-gate reporting distinguishes executed tests from skips, and reports per-module missing/empty corpus states.

Summary by CodeRabbit

• New Features

  • Added bench-gate reporting for release validation, covering executed tests, XPASS results, skipped modules, skip reasons, and missing or empty corpus data.

• Documentation

  • Updated release guidance to require complete bench-gate summaries and skip reasons in release pull requests.
  • Clarified that bench-gate checks run only at release cut.

• Tests

  • Added coverage for aggregate reporting and skip classification.

• Changelog

  • Documented the updated bench-gate reporting requirements.

[sourcery-ai]

Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[coderabbitai]

Warning

Review limit reached

@robotrocketscience, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 47 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: d6ab3693-7d5f-4435-b0ed-264e31d146b0

📥 Commits

Reviewing files that changed from the base of the PR and between 4be187c and c6f0c0c.

📒 Files selected for processing (2)

• tests/conftest.py
• tests/test_bench_gate_module_report_1477.py

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: fca7924e-25f0-4446-b804-875e7d471b2d

📥 Commits

Reviewing files that changed from the base of the PR and between 8c7d142 and 4be187c.

📒 Files selected for processing (3)

• CONTRIBUTING.md
• tests/conftest.py
• tests/test_bench_gate_module_report_1477.py

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/conftest.py

---

📝 Walkthrough

Walkthrough

The change classifies bench-gate results in pytest terminal summaries and documents release-cut execution, result interpretation, and mandatory inclusion of the complete summary in release PRs. Tests cover executed tests, module skips, whole-tier skips, and unrelated or malformed skip reasons.

Changes

Bench-gate release reporting

Layer / File(s)	Summary
Classify bench-gate summary results tests/conftest.py	The terminal summary classifies executed tests, missing or empty corpus modules, and whole-tier skips.
Validate bench-gate summary classification tests/test_bench_gate_module_report_1477.py	Tests validate report parsing, skip classification, executed counts, XPASS handling, and exclusion of unrelated or malformed skips.
Document release-cut execution CONTRIBUTING.md, docs/concepts/RELEASING.md, CHANGELOG/unreleased/...	Documentation requires bench-gate execution at release cut, interpretation of each result category, and verbatim inclusion of the summary in release PRs.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• robotrocketscience/aelfrice#320: Introduced the bench-gate harness extended by this reporting change.

Suggested labels: attn:review

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	The PR implements per-module reporting, but it does not provide the linked issue's required lab-side job or explicit corpus-coverage quantification.	Either implement the linked issue's lab-side job and AC4 coverage report, or update the linked issue and acceptance criteria to approve the release-cut approach.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the release-cut execution and per-module reporting changes.
Description check	✅ Passed	The description provides detailed rationale, linked issue references, scope, implementation summary, and test coverage, despite omitting template headings and checkboxes.
Out of Scope Changes check	✅ Passed	The code, tests, changelog, and documentation changes directly support release-cut execution and bench-gate reporting objectives.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ci/issue-1477-bench-gate-at-release-cut

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

PR-size soft cap

This PR is over the advisory size threshold:

• 308 changed lines (limit: 200)
• 5 changed files (limit: 3)

Bigger PRs collide with more open work, which under the parallel-session workflow tends to produce repeated attn:merge-conflict cycles (see #602). When practical, split into smaller PRs that each touch a focused surface.

This is advisory only — nothing is blocked. If the size is intentional (large refactor, module removal, generated code), apply the size:override label and this comment will be removed on the next push.

[sourcery-ai]

Reviewer's Guide

Implements per-module bench-gate reporting in pytest, formalizes when the bench-gate tier is run (at release cut only), documents the new process in RELEASING and CONTRIBUTING, and adds focused tests and changelog entry for the new reporting behavior.

Sequence diagram for running bench-gate at release cut and reporting per module

sequenceDiagram
    actor ReleaseEngineer
    participant PublicRepo
    participant BenchGateScript as scripts_run_bench_gate_sh
    participant PytestBenchGateTier as pytest_bench_gate_tier
    participant LabCorpus as lab_corpus
    participant ReleasePR as release_pr

    ReleaseEngineer->>PublicRepo: follow RELEASING_step_7
    ReleaseEngineer->>BenchGateScript: run scripts/run_bench_gate_sh
    BenchGateScript->>LabCorpus: read AELFRICE_CORPUS_ROOT (default lab corpus)
    BenchGateScript->>PytestBenchGateTier: invoke pytest bench-gate tier

    PytestBenchGateTier->>LabCorpus: load corpus modules
    PytestBenchGateTier-->>BenchGateScript: bench_gate_tier summary block
    note over PytestBenchGateTier: block separates executed, missing_module, empty_module, tier_skipped

    BenchGateScript-->>ReleaseEngineer: display bench_gate_tier block
    ReleaseEngineer->>ReleasePR: paste bench_gate_tier block into PR body

    PublicRepo->>ReleasePR: enforce presence of bench_gate_tier block before merge

Loading

File-Level Changes

Change	Details	Files
Enhance pytest terminal summary to report bench-gate results per module and distinguish executed tests from different skip reasons.	Introduce a regex classifier to parse corpus-module skip reasons into module name and missing/empty status. Add helper to safely extract skip reason text from pytest reports. Rewrite pytest_terminal_summary to count tier-wide corpus-root skips, per-module skips, and executed bench-gated tests, and to emit a structured summary block only when any bench-gate activity occurs. Ensure executed tests are counted via the bench_gated marker in passed/failed reports rather than via skip data, and that unrelated skips are excluded.	tests/conftest.py
Make running the bench-gate tier a mandatory step at release time and require its output in the release PR body.	Update release checklist to run scripts/run_bench_gate.sh with default corpus root and to paste the bench-gate summary block verbatim into the release PR. Clarify that the bench-gate summary’s three states (executed, module missing/empty, whole tier skipped) must be read rather than the raw pass count. Require explicit mention in the PR body when the tier could not run, forbidding silent skips. Adjust release step numbering to insert the new bench-gate step.	docs/concepts/RELEASING.md
Document where and how the bench-gate tier actually runs and how to interpret its summary.	Explain that the bench-gate tier runs only at the release cut, not in lab-side CI, and why the previous description was incorrect. Describe the three bench-gate states (executed, module-level skips, whole-tier skip) and that only executed tests provide a verdict. Emphasize that a run with many modules skipped is currently normal and must be interpreted via the summary block. Align contributor guidance with the new release process and corpus-based evidence expectations.	CONTRIBUTING.md
Add tests that drive pytest_terminal_summary directly with synthetic reports to pin the new per-module reporting behavior.	Dynamically import tests/conftest.py into a dedicated test module to exercise pytest_terminal_summary without running real tests. Define lightweight _Report and _Terminal helpers to model pytest reports and terminal reporter behavior in isolation. Add tests for: no bench-gate activity producing no output; aggregate tier skip behavior unchanged; per-module naming/counting of missing/empty modules; separate treatment of missing vs empty; counting executed gates via bench_gated marker; ignoring unrelated skips; and ignoring malformed/partial skip reasons. Use helpers to generate skip reasons via the same message format as load_corpus_module to ensure classifier stays in sync with message strings.	tests/test_bench_gate_module_report_1477.py
Record the bench-gate-at-release-cut and per-module reporting change in the unreleased changelog.	Add a CI changelog entry describing that the bench-gate quality tier now runs at release cut and that its report names modules with no verdict. Explain why prior "lab-side" assumptions were wrong and how the new three-state reporting works. Note that classification is by skip reason and executed tests are counted off the bench_gated marker, and that the tests are mutation-checked for robustness.	CHANGELOG/unreleased/1477-bench-gate-at-release-cut.md

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#1477	Establish a reproducible process so the bench-gate tier actually runs with AELFRICE_CORPUS_ROOT set on a defined cadence, instead of never running.	✅
#1477	Ensure the bench-gate results are visible to public-repo readers and update documentation (e.g. CONTRIBUTING) to point to where those results can be seen.	✅
#1477	Enhance the bench-gate reporting to give per-module execution/skip status and quantify the corpus-coverage gap, including which modules have corpus rows, which do not, and which parked defaults depend on the missing modules.	❌	The PR updates pytest_terminal_summary to report per-module skips (missing vs empty) and counts executed bench-gated tests, and documents reading this block at release, which satisfies per-module reporting (AC3) and partially surfaces coverage. However, it does not provide a separate explicit quantification of which modules have corpus rows vs not, nor does it document which parked defaults depend on each missing module (AC4).

Possibly linked issues

• ci(bench-gate): record that the 36 corpus-gated quality tests run lab-side only (#1420 §3) #1456: They address the same gap: bench-gate never actually ran; PR wires it into release cuts with per-module reporting.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/conftest.py`:
- Around line 76-124: Update _MODULE_SKIP_RE to include the expected corpus-root
prefix and use fullmatch() when classifying reasons in pytest_terminal_summary.
Replace the broad CORPUS_ENV_VAR substring check with an exact
BENCH_GATE_SKIP_REASON match, so only bench-gate skip messages contribute to
tier or module counts. Add regression coverage for unrelated skip reasons
containing the environment-variable name or an embedded module phrase.
- Around line 126-131: Update the executed-count calculation in the pytest
reporting logic to include the "xpassed" entry from TerminalReporter.stats, so
non-strict XPASS reports with the "bench_gated" keyword are counted. Add a
constructed xpassed report in test_bench_gate_module_report_1477.py to cover
this behavior.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: aa537944-4224-4771-9f77-e108f30615be

📥 Commits

Reviewing files that changed from the base of the PR and between 05c6195 and 8c7d142.

📒 Files selected for processing (5)

• CHANGELOG/unreleased/1477-bench-gate-at-release-cut.md
• CONTRIBUTING.md
• docs/concepts/RELEASING.md
• tests/conftest.py
• tests/test_bench_gate_module_report_1477.py

[github-actions]

This PR is now behind main. Rebase locally so your commit signatures stay intact:

git fetch origin && git checkout 'ci/issue-1477-bench-gate-at-release-cut' && git rebase origin/main
# resolve conflicts if any, then
git push --force-with-lease

Auto-rebase was removed because the bot has no signing key; rebasing as the bot strips author signatures and the required_signatures rule on main then blocks the merge. See #341.

[github-actions]

merge-train: merged c6f0c0c → main via FF push.