Repository navigation
feat(reviewer-report): clearer scorecard — count review events (Total PRs · Reviews · Rate-limited · No response) - #1236
Conversation
…imited / No response Replaces the confusing "Participation %" and "Resolved %" columns with a clear per-reviewer partition of the eligible (non-draft) PRs into three buckets that always sum to the eligible count: - Reviewed — the bot delivered a real review/comment - Rate-limited — the bot's SOLE action was to decline (out of quota) - No response — the bot never appeared Correctness fix: previously a PR was flagged "rate-limited" if the bot posted ANY rate-limit text, even when it also gave a real review — massively overstating refusals (e.g. CodeRabbit showed 147 "rate-limited" while actually reviewing most of them). Now a PR is a refusal only when the bot's real_responses == 0. On live data this corrects CodeRabbit from 147 → 16 refusals and isolates Codex's genuine 63/75 refusal rate. Normalization now tags each submission as a refusal vs real response; latency is measured to the first REAL response so quota notices no longer pollute the percentiles. Thread-resolution and reactions are still computed into the snapshot but dropped from the headline table for legibility. Dropped the now-redundant "Reliability & no-shows" section (folded into the buckets). Tests: 24/24 (added bucket-sum invariant, refusal-vs-review classification, and "real review alongside a rate-limit comment still counts as reviewed"). Follow-up to #1196. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
Next review available in: 20 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (5)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request refactors the reviewer report generation to partition eligible pull requests into three distinct buckets per reviewer: Reviewed, Rate-limited, and No response. This improves the accuracy of the metrics by separating out-of-quota refusals from absent work. The review feedback highlights two valuable improvement opportunities: optimizing the JQ aggregation logic from $BATS_TEST_TMPDIR for automatic cleanup.
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-07-14T21:53:14Z. |
There was a problem hiding this comment.
Pull request overview
Refactors the deterministic third-party reviewer scorecard to report a clearer three-bucket partition of eligible PRs per reviewer (Reviewed / Rate-limited / No response), fixing the prior over-counting of “rate-limited” when a real review was also present.
Changes:
- Reworked normalization to split bot activity into
real_responsesvsrefusals, and measure latency only to the first real response. - Updated snapshot aggregation + rendering to output the three-bucket scorecard (and WoW deltas on Reviewed/Rate-limited).
- Updated fixtures, bats tests, and documentation to reflect the new metric model and invariants.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
scripts/reviewer_report.sh |
Implements the new refusal-vs-real-response normalization, three-bucket aggregation, and updated scorecard rendering. |
tests/reviewer_report.bats |
Adds/updates unit tests for refusal classification, bucket-sum invariant, and updated rendered headers/snapshot keys. |
tests/fixtures/reviewer_jsonl/repo1.jsonl |
Updates normalized fixture records to the new real_responses/refusals + latency semantics. |
tests/fixtures/reviewer_jsonl/repo2.jsonl |
Same as repo1 fixture update for the new normalized schema. |
docs/reviewer-report.md |
Updates the design doc to describe the three-bucket model and updated latency definition. |
…ommit reviews) A PR with multiple commits gets re-reviewed on each push, producing multiple review submissions by the same reviewer. The bucket model collapsed these to one "reviewed" PR, hiding real activity — on live data CodeRabbit shows 193 review events across 97 PRs (up to 11 on a single PR), and Codex shows 121 rate-limit refusals vs 11 real reviews. Scorecard columns are now: Total PRs · Reviews · ✅/🔄 · Rate-limited · No response. - Reviews = count of real review submissions, EACH occurrence - Rate-limited = count of refusal events, EACH occurrence - No response = eligible PRs the bot never engaged with at all The per-PR bucket fields (reviewed_prs / refused_prs / no_response_prs) remain in the snapshot for the sum-to-eligible invariant and history; the headline now surfaces event counts. SonarCloud posts status-check comments rather than reviews, so its Reviews reads 0 by design (noted in the report). WoW deltas now track Reviews and Rate-limited event counts. Tests: 26/26 (added event-count aggregation + a "PR reviewed twice = 2 review events" case).
don-petry
changed the title
Update: count review events, not distinct PRsPer follow-up feedback — a PR with multiple commits is re-reviewed on each push, so a reviewer submits many reviews on one PR. The scorecard now counts each event. Columns: Total PRs · Reviews · ✅/🔄 · Rate-limited · No response Live 30-day data (
|
Dev-Lead — review-changes (applied)Changes committed and pushed. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
|
Dev-Lead — waiting on PR blockers (intent: fix-reviews)PR: #1236 |
donpetry-bot
left a comment
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: MEDIUM
Reviewed commit: afac2c2a187b73ccc37a32e173b3fb52778031a7
Review mode: triage-approved (single reviewer)
Summary
Refactors the deterministic reviewer scorecard to partition eligible PRs into Reviewed / Rate-limited / No response buckets, fixing over-counting of rate-limited PRs (a real review alongside a quota notice now counts as Reviewed; only refusal-only touches count as rate-limited). Latency now measures time to first real response. Docs, fixtures, and bats tests updated in step; all advisory-reviewer findings were addressed and threads resolved.
Linked issue analysis
No linked issues. The PR is a follow-up to #1196 with a thorough description including live 30-day validation data showing the correctness fix (CodeRabbit refusals corrected from 147 to 16; Codex's genuine 63/75 refusal rate isolated).
Findings
No blocking findings.
- Correctness: refusal classification (
real_responses == 0) is the right fix for the prior over-counting; the three-bucket partition invariant (reviewed + refused + no_response == eligible) is enforced by construction and covered by a dedicated test. Latency correctly excludes refusal-only touches (latency_s: null). - Reviewer feedback resolved: the O(N×M)
$elig_urls | index()scan was replaced with areduce-built O(1) lookup map, and the baremktempwas moved into$BATS_TEST_TMPDIR— all 3 review threads are resolved. CodeRabbit approved. - Tests: 6 new/updated bats tests cover refusal-only classification, real-review-beside-quota-notice, bucket-sum invariant, event counts, multi-commit re-review counting, and render columns.
- Security: pure jq/awk/bash reporting logic; values passed via
--arg/--argjson; no new dependencies, no workflow changes. Secret-scanning MCP tool unavailable in this run; the gitleaks CI check passed.
CI status
All checks green: shellcheck, ShellCheck, Lint, bats, unit-tests, CodeQL, Secret scan (gitleaks), SonarCloud (quality gate passed), AgentShield, guard, holdout-guard, template-drift, and all validators passed. Two cancelled dev-lead runs (dispatch, ci-relay) were superseded seconds later by successful/skipped re-runs on the same SHA — concurrency cancellation, not failures. Dependency-audit jobs skipped (no matching ecosystems).
Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.
Dev-Lead — fix-reviews (no-changes)Agent reasoning |
What & why
Follow-up to #1196. The Participation % and Resolved % columns were confusing, so the scorecard now reports the metric you actually want: a per-reviewer partition of the eligible (non-draft) PRs into three buckets that always sum to the eligible count.
Correctness fix (the important part)
Previously a PR was flagged "rate-limited" if the bot posted any rate-limit text — even when it also gave a real review. That massively overstated refusals. New logic classifies a PR as a refusal only when the bot's
real_responses == 0.On live 30-day data for
.github-private:147→ 16CodeRabbit's refusals drop from a misleading 147 to a true 16 (it reviews most PRs despite occasional limits), while Codex's genuine 63/75 refusal rate is now isolated cleanly.
Changes
Validation
shellcheck --severity=warning -xcleantests/reviewer_report.bats— 24/24 (added: three-bucket sum invariant, refusal-vs-review classification, and "real review + rate-limit comment ⇒ reviewed")🤖 Generated with Claude Code