Repository navigation
feat(reviewer-report): count a comment-only reviewer's response as a Review (SonarCloud) - #1237
Conversation
…s no formal review SonarCloud (and any bot that delivers its verdict as a top-level comment rather than a GitHub review) previously showed 0 Reviews. Now, when a bot submits no formal review on a PR, its non-refusal top-level comment is counted as its review — so its work is visible. Bots that DO submit formal reviews are unaffected (their extra summary comments are never double-counted), and rate-limit refusals are still excluded. On live data SonarCloud now reads 97 reviews (its quality-gate comments) instead of 0; CodeRabbit's 193 formal reviews are unchanged. Tests: 28/28 (added comment-as-review for a comment-only responder, and a no-double-count guard for a formal-review bot that also posts a summary comment).
|
Warning Review limit reached
Next review available in: 16 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the reviewer report script and documentation to count top-level comments as reviews for bots that do not submit formal reviews (such as SonarCloud), while ensuring that bots with formal reviews do not have their summary comments double-counted. It also adds corresponding BATS tests to verify this behavior. The review feedback suggests improving the BATS tests by creating temporary files inside $BATS_TEST_TMPDIR instead of using bare mktemp to ensure automatic cleanup and avoid parallel-execution conflicts.
There was a problem hiding this comment.
Pull request overview
Adjusts the reviewer-report normalization so “comment-only” bot responses (notably SonarCloud’s quality-gate comment) are credited as Reviews when the bot did not submit any formal GitHub reviews, making those bots’ contributions visible in the scorecard.
Changes:
- Update normalization logic to count a bot’s top-level (issue) comments as review events only when it has zero formal review submissions on the PR.
- Add Bats coverage for the comment-only reviewer case and for preventing double-counting when a bot has both a formal review and a summary comment.
- Update report text and documentation to reflect the revised “Reviews” definition.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
scripts/reviewer_report.sh |
Implements “comment-only counts as a review when no formal reviews exist” in _NORMALIZE_JQ, and updates rendered metric definitions accordingly. |
tests/reviewer_report.bats |
Adds regression tests for comment-only review counting and for avoiding double-counting with formal reviews. |
docs/reviewer-report.md |
Updates the metric definition for Reviews to match the new behavior and removes the prior SonarCloud-specific note. |
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-07-14T22:47:26Z. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Dev-Lead — fix-bot-comment (applied)Changes committed and pushed. |
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
donpetry-bot
left a comment
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: LOW
Reviewed commit: b4195bc51919e88c65fca2a4a0fdec5e31ffbf9d
Review mode: triage-approved (single reviewer)
Summary
Small, well-tested reporting-logic change: when a bot submits zero formal GitHub reviews on a PR, its non-refusal top-level comment is now counted as its review (making SonarCloud's quality-gate comments visible in the scorecard). Bots that do submit formal reviews are unaffected — the $rev_events > 0 guard prevents double-counting their summary comments, and rate-limit refusals remain excluded (the fallback filters from $real, which already excludes refusals). Docs and rendered metric definitions updated consistently; two regression tests added (comment-only responder counted, formal-review bot not double-counted). Confirms the triage assessment: low-risk, no security surface.
Linked issue analysis
No linked issues. The PR is a stated follow-up to #1236 and its body documents the requested behavior ("count SonarCloud's response as a Review unless it says Rate Limited"), which the diff implements as a general comment-only-bot rule rather than a SonarCloud special case.
Findings
No blocking findings.
- All 4 inline review threads (Gemini ×2, Copilot ×2 — bare
mktemp→$BATS_TEST_TMPDIR) were addressed in commitb4195bcand are resolved. - Logic verified: the issue-comment fallback only activates when a bot has zero formal reviews on the PR, so formal reviewers (CodeRabbit, Copilot, Gemini) are unchanged; refusal comments can never be counted.
- Secret-scanning MCP tool unavailable in this run; the gitleaks CI check passed on the head SHA.
- Note: an earlier sweep withheld auto-approval because CodeRabbit was rate-limited (reset 2026-07-14T22:47:26Z); this triage-approved re-review is the scheduled follow-up, and CodeRabbit's commit status reports SUCCESS.
CI status
All required checks green on b4195bc: Lint (shellcheck, bats 28/28, template-drift, personas, agent profiles), CI (ShellCheck, secret scan/gitleaks, agent security scan, workflow compile), CodeQL (actions + python), unit-tests, AgentShield, holdout-guard, test-deletion guard, SonarCloud quality gate (0 new issues, 0 hotspots), CodeRabbit status SUCCESS. One superseded "review / review" trigger run was cancelled and re-run successfully; dependency-audit sub-jobs skipped (no matching ecosystems). Branch is BEHIND main but MERGEABLE.
Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
donpetry-bot
left a comment
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: LOW
Reviewed commit: 262f27ec7aba51255d7f4a060237d52bd7e6b7ce
Review mode: triage-approved (single reviewer)
Summary
Small, well-tested change to the reviewer-report normalization: a bot that submits no formal review but posts a non-refusal top-level comment (e.g. SonarCloud's quality-gate comment) now has that comment counted as its review. The jq guard (rev_events > 0) ensures bots with formal reviews are unaffected (no double-counting), refusals remain excluded, and APPROVED/CHANGES_REQUESTED counts still derive only from formal reviews. Docs and rendered footnotes updated consistently; two new bats tests cover both the fallback and the no-double-count case. Since the prior approval at b4195bc, the only new commits are merges from main — the PR's own diff is unchanged.
Linked issue analysis
No linked issues. The PR is a follow-up to #1236, and the body documents the requested behavior (count SonarCloud's response as a review unless rate-limited), the general rule adopted, and validation against live 30-day data.
Findings
No blocking findings.
- Logic verified:
$rev_eventscounts formal reviews in the non-refusal set; the issue-comment fallback only engages when it is 0, so formal-review bots never double-count their summary comments. Verdict columns (✅/🔄) correctly remain formal-review-only, matching the updated docs. - Tests:
tests/reviewer_report.batsadds coverage for the comment-only responder and the no-double-count case (28/28 per PR body; bats CI check green). Temp files use$BATS_TEST_TMPDIRper reviewer suggestions — all 4 review threads resolved. - Secret scan: the
run_secret_scanningMCP tool is not available in this environment; the gitleaks CI check is green and the diff (docs, jq logic, test fixtures) contains no secret-like content. - Prior bot rate-limit hold (sha a669453) is superseded: it referenced an older commit, and advisory reviews from Gemini and Copilot have since been delivered at the current diff.
CI status
All required checks green: shellcheck, bats, unit-tests, Lint, SonarCloud (quality gate passed), CodeQL (neutral — no findings), gitleaks secret scan, agent-shield, holdout-guard, template-drift, validators. Cancelled dev-lead dispatch jobs and skipped dependency-audit ecosystems are non-blocking.
Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.
What
Follow-up to #1236. SonarCloud (and any bot that delivers its verdict as a top-level comment rather than a formal GitHub review) previously showed 0 Reviews. Now, when a bot submits no formal review on a PR, its non-refusal top-level comment is counted as its review — so its work is visible.
Requested: "count SonarCloud's response as a Review (unless it says Rate Limited)."
Rule (general, not a SonarCloud special-case)
Per PR, per bot:
Verified against live data: SonarCloud posts exactly one issue comment per PR (its quality-gate result), no formal review / no inline — so this counts the right thing.
Effect on live 30-day data (
.github-private)Validation
shellcheck --severity=warning -xclean · markdownlint cleantests/reviewer_report.bats— 28/28, added:🤖 Generated with Claude Code