Repository navigation
feat: weekly Third-Party Reviewer Scorecard (deterministic, no LLM) - #1196
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
Next review available in: 32 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (8)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-07-14T01:28:03Z. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a Third-Party Reviewer Scorecard weekly report system, featuring a documentation file, a Bash script (scripts/reviewer_report.sh) that queries and aggregates PR review data via the GitHub GraphQL API, and a BATS test suite. The reviewer feedback highlights several opportunities to improve script robustness and test hygiene. Specifically, it recommends using jq's optional chaining operator (?) to prevent crashes under set -e when parsing potentially null API responses, avoiding pipelines to head -1 to prevent SIGPIPE failures under set -o pipefail, removing the unused percentile function, and utilizing $BATS_TEST_TMPDIR for temporary files in tests to ensure proper cleanup.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Dev-Lead — fix-reviews (applied)Changes committed and pushed. |
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
donpetry-bot
left a comment
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: MEDIUM
Reviewed commit: 883976dab2f0ff10c4a8d7b1e87c7c669f9a7865
Review mode: triage-approved (single reviewer)
Summary
Adds a weekly, fully deterministic Third-Party Reviewer Scorecard (jq/awk over GitHub review data — no LLM): new scheduled workflow, org-wide GraphQL collection script with pure unit-tested helpers, bats tests, fixtures, and docs (+1,074/-0 across 8 files). Mirrors the established token_report.sh / token-report.yml pattern and reuses the advisory-review-gate registry as the single source of truth for bot identity and rate-limit detection.
Linked issue analysis
No linked issues (closingIssuesReferences is empty). The PR body itself fully specifies motivation, metrics, data-collection strategy, known gaps, and an explicitly separate human-approved LLM roadmap — self-contained scope, no issue gate to satisfy.
Findings
No blocking findings.
Verified during review:
- Workflow security: schedule + workflow_dispatch triggers only (no untrusted PR context); dispatch inputs flow through
env:— no inline ${{ }} interpolation in run steps, so no script-injection surface. Minimal permissions (contents: read,issues: write). All three actions are SHA-pinned to the exact SHAs already used across this repo's workflows (checkout v6.0.3, upload-artifact v7.0.1, github-script v9.0.0). - Secret usage:
GH_PAT_WORKFLOWSis an existing org secret used identically totoken-report.yml(cross-repo read for the org sweep); the tracking-issue post deliberately uses the lower-privilegegithub.token. No new secrets or privilege elevation. - Reuse claims verified:
RATE_LIMIT_NOTICE_BOTSand_advisory_rate_limit_patternexist inscripts/lib/advisory-review-gate.sh; the script sources them with a sensible fallback for stripped test envs. - Script robustness: per-call
ghtimeouts, bounded pagination (MAX_PR_PAGES), per-repo failure degrades to WARN + partial data, best-effort WoW snapshot fetch, repo names passed as positional args to worker subshells (no string interpolation). - Tests: 20 bats tests covering the pure normalize/aggregate/render path, wired into
lint.yml; CI bats + shellcheck green. - Review threads: all 11 prior gemini-code-assist threads (GraphQL error guards, optional chaining, BATS_TEST_TMPDIR) resolved; dev-lead fix cycle applied and re-confirmed no-changes at the current head SHA.
Non-blocking observations:
- Latency p50 uses
floorwhile p95 usesroundfor the nearest-rank index — a cosmetic inconsistency in percentile method. aggregate_snapshotswallows jq errors into a zeroed snapshot (2>/dev/null || echo …) — acceptable for a report, but a malformed JSONL bug would degrade silently.- Branch is BEHIND main (MERGEABLE); auto-rebase automation applies.
- Triage rated this LOW; single-reviewer assessment is MEDIUM (new scheduled workflow + non-trivial logic) — MEDIUM still satisfies the auto-approve gate.
- Secret-scan MCP tool unavailable in this environment; the gitleaks CI check (green) covers secret detection.
CI status
All checks green (31 total): shellcheck ✓, bats ✓, unit-tests ✓, Lint ✓, AgentShield ✓, Agent Security Scan ✓, CodeQL (actions + python) ✓, Secret scan (gitleaks) ✓, SonarCloud ✓, CodeRabbit ✓, holdout-guard ✓, template-drift ✓, gh-aw-compile ✓, validate-agent-profiles ✓. Dependency-audit jobs skipped (no matching ecosystems).
Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.
Dev-Lead — rate-limited (intent: review-changes)PR: #1196 |
|
Note @don-petry I received your request but all AI engines are currently rate-limited. I'll retry automatically once the rate limit clears. |
Measures the external agentic PR reviewers used org-wide — GitHub Copilot, Gemini Code Assist, Codex, CodeRabbit, and SonarCloud — and renders a weekly Markdown scorecard. Every metric is computed with jq/awk from GitHub's own review data; no LLM is in the pipeline (an LLM narrative is a deliberately separate, human-approved v2 add-on). Mirrors the Token Cost Observatory pattern: pure render/aggregate helpers (unit-tested, no network) split from a network main(); org-wide sweep via GH_PAT_WORKFLOWS; published to the Step Summary + a find-or-create pinned tracking issue comment, with a failure-alert step. Reuses the advisory-review-gate registry (scripts/lib/advisory-review-gate.sh) as the single source of truth for bot logins and rate-limit body detection, so this report can never drift from the approval gate. Metrics per reviewer: participation %, reviews, comments/PR, first-touch latency p50/p95, approve/changes-requested mix, rate-limited incidents, inline- thread resolution %, reactions, coverage overlap, and week-over-week deltas (via a snapshot artifact diffed against the prior run). Cost is explicitly a stated gap — external SaaS bots emit no token-usage artifacts. Collection uses one batched GraphQL query per repo (reviews + reviewThreads + comments + reactions), paginated newest-first and stopping at the lookback window; per-call timeouts and bounded xargs concurrency keep it inside the job budget. - scripts/reviewer_report.sh — collector + pure renderer - .github/workflows/reviewer-report.yml — weekly cron (Mon 09:00 UTC) - tests/reviewer_report.bats (+ fixtures) — 21 unit tests - docs/reviewer-report.md — design doc - lint.yml — register the new bats suite Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2fcb51f to
719509d
Compare
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
|
… PRs · Reviews · Rate-limited · No response) (#1236) * refactor(reviewer-report): three-bucket scorecard — Reviewed / Rate-limited / No response Replaces the confusing "Participation %" and "Resolved %" columns with a clear per-reviewer partition of the eligible (non-draft) PRs into three buckets that always sum to the eligible count: - Reviewed — the bot delivered a real review/comment - Rate-limited — the bot's SOLE action was to decline (out of quota) - No response — the bot never appeared Correctness fix: previously a PR was flagged "rate-limited" if the bot posted ANY rate-limit text, even when it also gave a real review — massively overstating refusals (e.g. CodeRabbit showed 147 "rate-limited" while actually reviewing most of them). Now a PR is a refusal only when the bot's real_responses == 0. On live data this corrects CodeRabbit from 147 → 16 refusals and isolates Codex's genuine 63/75 refusal rate. Normalization now tags each submission as a refusal vs real response; latency is measured to the first REAL response so quota notices no longer pollute the percentiles. Thread-resolution and reactions are still computed into the snapshot but dropped from the headline table for legibility. Dropped the now-redundant "Reliability & no-shows" section (folded into the buckets). Tests: 24/24 (added bucket-sum invariant, refusal-vs-review classification, and "real review alongside a rate-limit comment still counts as reviewed"). Follow-up to #1196. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reviewer-report): count review EVENTS, not distinct PRs (multi-commit reviews) A PR with multiple commits gets re-reviewed on each push, producing multiple review submissions by the same reviewer. The bucket model collapsed these to one "reviewed" PR, hiding real activity — on live data CodeRabbit shows 193 review events across 97 PRs (up to 11 on a single PR), and Codex shows 121 rate-limit refusals vs 11 real reviews. Scorecard columns are now: Total PRs · Reviews · ✅/🔄 · Rate-limited · No response. - Reviews = count of real review submissions, EACH occurrence - Rate-limited = count of refusal events, EACH occurrence - No response = eligible PRs the bot never engaged with at all The per-PR bucket fields (reviewed_prs / refused_prs / no_response_prs) remain in the snapshot for the sum-to-eligible invariant and history; the headline now surfaces event counts. SonarCloud posts status-check comments rather than reviews, so its Reviews reads 0 by design (noted in the report). WoW deltas now track Reviews and Rate-limited event counts. Tests: 26/26 (added event-count aggregation + a "PR reviewed twice = 2 review events" case). * chore: dev-lead update (review-changes) [skip ci-relay] --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: donpetry-bot <281750570+donpetry-bot@users.noreply.github.com>
What
A new weekly report analyzing the third-party agentic PR reviewers used across the org — GitHub Copilot, Gemini Code Assist, Codex, CodeRabbit, and SonarCloud. Built deterministically (
jq/awkover GitHub's own review data) — no LLM anywhere in the pipeline, per the requirement that any LLM use be a separate, human-approved step.Modeled on the Token Cost Observatory (
token_report.sh/token-report.yml): pure render/aggregate helpers (unit-tested, no network) split from a networkmain(), org-wide sweep viaGH_PAT_WORKFLOWS, published to the Step Summary and a find-or-create pinned tracking-issue comment, with a failure-alert step.Metrics (per reviewer, weekly, org-wide)
Cost is an explicit stated gap — external SaaS bots emit no token-usage artifacts, so
$cost isn't observable from our side (unlike our own Claude reviewer, tracked in the Token Cost Observatory).Reuse, not reinvention
Bot logins and rate-limit body detection come from the existing
scripts/lib/advisory-review-gate.shregistry (ADVISORY_BOTS/RATE_LIMIT_NOTICE_BOTS/_advisory_rate_limit_pattern), so this report can never drift from the approval gate's notion of who these bots are.Collection
One batched GraphQL query per repo (reviews +
reviewThreadswith resolution/reactions + issue comments), paginated newest-first and stopping at the lookback window. Per-calltimeoutwrapper + boundedxargsconcurrency keep it inside the job budget (mirrorstoken_report.sh).Validation
shellcheck --severity=warning -x— cleantests/reviewer_report.bats— 21/21 passing (normalize, aggregate, render, WoW deltas).github-private(30-day window, 50 PRs, 226 bot-review events) produced plausible, discriminating results: Gemini fast (81s p50) / 96% participation; Copilot 3m p50 (matches documented ~186s median); Codex 43/45 PRs rate-limited; SonarCloud p95 36.3h outlier. It also surfaced a real drift — CodeRabbit posting 20 approvals despite the gate doc stating it no longer approves.Files
scripts/reviewer_report.sh— collector + pure renderer.github/workflows/reviewer-report.yml— weekly cron (Mon 09:00 UTC, 1h after token-report)tests/reviewer_report.bats(+ fixtures)docs/reviewer-report.md— design doc.github/workflows/lint.yml— register the new bats suiteRoadmap (v2, LLM-optional, human-approved)
Comment usefulness / false-positive scoring, cross-bot semantic dedup, finding categorization, and a narrative synthesis — all gated behind an explicit
workflow_dispatchopt-in, never unattended.🤖 Generated with Claude Code