Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 21 additions & 15 deletions docs/reviewer-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,27 +43,33 @@ entire history. This is far fewer API calls than per-PR REST fetches.

Each PR node is normalized (pure `jq`) into two record kinds:

- `{kind:"pr", …}` — one per PR (the participation **denominator**).
- `{kind:"bot_pr", …}` — one per tracked bot that touched the PR, carrying its first-touch
time, latency, verdict counts, inline-comment count, thread resolution, reactions, and a
rate-limited flag.
- `{kind:"pr", …}` — one per PR (the eligible-PR **denominator**).
- `{kind:"bot_pr", …}` — one per tracked bot that touched the PR, carrying its
`real_responses` / `refusals` split (a rate-limit-only touch is a refusal), latency to
first real response, verdict counts, inline-comment count, thread resolution, and reactions.

Human authors and untracked bots are dropped at normalization time.

## Metrics

Everything below is deterministic. _Participation denominator_ = non-draft PRs active in
the window.
Everything is deterministic. The headline is a **three-bucket partition of the eligible
(non-draft) PRs, per reviewer**: every eligible PR is exactly one of Reviewed, Rate-limited,
or No response for a given bot, so the three always sum to the eligible-PR count. This makes
"did the reviewer actually do its job?" legible at a glance and separates *refused* work
(out-of-quota) from *absent* work (never showed up).

| Family | Metric | Definition |
|---|---|---|
| **Participation** | PRs reviewed, participation %, reviews, comments/PR | Distinct PRs a bot touched ÷ eligible PRs; review + inline-comment counts. |
| **Timeliness** | Latency p50 / p95 | Seconds from PR creation to the bot's first review/comment (nearest-rank percentiles). Targets the "review arrived after auto-approval" failure mode (PR #453). |
| **Verdict mix** | ✅ approved / 🔄 changes-requested | GitHub review `state` per bot. (CodeRabbit no longer approves — its mix reflects that.) |
| **Reliability** | Rate-limited PRs, coverage gap, no-shows | Out-of-quota notices detected by body text (shared gate pattern); eligible PRs a bot did not touch. |
| **Signal quality** | Thread resolution %, 👍/👎 | Share of a bot's inline threads later resolved; reactions on its comments. |
| **Overlap** | PRs reviewed by ≥2 bots | Redundancy vs specialization signal. |
| **Trend** | Week-over-week Δ (▲/▼) | Every headline metric vs last week's snapshot. Arrows are directional only. |
| Metric | Definition |
|---|---|
| **Reviewed** | PRs where the bot delivered a real review or comment. A PR still counts as Reviewed if the bot *also* posted a rate-limit notice — only the real response matters. |
| **Rate-limited** | PRs where the bot's **sole** action was to decline (out-of-quota / rate-limit notice, detected by body text via the shared gate pattern). It refused to review. |
| **No response** | Eligible PRs the bot never appeared on. `= eligible − Reviewed − Rate-limited`. |
| **✅ / 🔄** | APPROVED / CHANGES_REQUESTED review counts. A rate-limit notice is never counted as a review. |
| **Latency p50 / p95** | Seconds from PR creation to the bot's first **real** response (refusals excluded, so quota notices don't pollute the percentiles). Targets the "review arrived after auto-approval" failure mode (PR #453). |
| **Coverage overlap** | PRs reviewed by ≥2 bots — a redundancy vs specialization signal. |
| **Trend** | Week-over-week Δ (▲/▼) on Reviewed and Rate-limited vs last week's snapshot. Arrows are directional only. |

Thread-resolution and reaction counts are still computed and stored in the snapshot for
downstream use, but are kept out of the headline table to keep it legible.

### Week-over-week deltas

Expand Down
135 changes: 67 additions & 68 deletions scripts/reviewer_report.sh
Original file line number Diff line number Diff line change
Expand Up @@ -177,34 +177,44 @@ _NORMALIZE_JQ='
| ( { kind: "pr", repo: $repo, pr: ($pr.url),
created: $pr.createdAt, merged: ($pr.mergedAt // null),
draft: ($pr.isDraft // false), author: ($pr.author.login // "") } ),
( $subs | group_by(.bot)[]
| { bot: .[0].bot }
| . as $g
| ($subs | map(select(.bot == $g.bot))) as $mine
| ($mine | map(.at) | sort | .[0]) as $first_at
( $subs | group_by(.bot)[] as $grp
# Tag each submission as a refusal (an out-of-quota / rate-limit notice) vs a
# real response. A PR counts as "reviewed" if the bot gave ANY real response,
# even alongside a rate-limit notice; it counts as "refused" only when the
# bot'"'"'s SOLE action on the PR was to decline. This is the key correctness
# fix over the old "any rate-limit text present" flag.
| ($grp | map(. + {refusal: ((.body // "") | test($rl; "i"))})) as $mine
| ($mine | map(select(.refusal | not))) as $real
| ($mine | map(select(.refusal))) as $refd
| ($real | map(.at) | min) as $first_real
| { kind: "bot_pr", repo: $repo, pr: ($pr.url),
bot: $g.bot, created: $pr.createdAt, first_at: $first_at,
latency_s: (if $first_at == null then null
else (($first_at | fromdateiso8601) - ($pr.createdAt | fromdateiso8601)) end),
reviews: ($mine | map(select(.kind=="review")) | length),
approved: ($mine | map(select(.kind=="review" and .state=="APPROVED")) | length),
changes_req: ($mine | map(select(.kind=="review" and .state=="CHANGES_REQUESTED")) | length),
commented: ($mine | map(select(.kind=="review" and .state=="COMMENTED")) | length),
dismissed: ($mine | map(select(.kind=="review" and .state=="DISMISSED")) | length),
rate_limited: ($mine | map(select(.body | test($rl; "i"))) | length | if . > 0 then 1 else 0 end),
inline_comments: ($mine | map(select(.kind=="inline")) | length),
threads_total: ($mine | map(select(.kind=="inline")) | length),
threads_resolved: ($mine | map(select(.kind=="inline" and .resolved)) | length),
threads_outdated: ($mine | map(select(.kind=="inline" and .outdated)) | length),
thumbs_up: ($mine | map(.reactions_up // 0) | add // 0),
thumbs_down: ($mine | map(.reactions_down // 0) | add // 0) } )
bot: $grp[0].bot, created: $pr.createdAt,
real_responses: ($real | length),
refusals: ($refd | length),
# Latency = time to first REAL response (null when the bot only refused,
# so refusals never pollute the latency percentiles).
latency_s: (if ($real | length) == 0 then null
else (($first_real | fromdateiso8601) - ($pr.createdAt | fromdateiso8601)) end),
reviews: ($real | map(select(.kind=="review")) | length),
approved: ($real | map(select(.kind=="review" and .state=="APPROVED")) | length),
changes_req: ($real | map(select(.kind=="review" and .state=="CHANGES_REQUESTED")) | length),
inline_comments: ($real | map(select(.kind=="inline")) | length),
threads_total: ($real | map(select(.kind=="inline")) | length),
threads_resolved: ($real | map(select(.kind=="inline" and .resolved)) | length),
thumbs_up: ($real | map(.reactions_up // 0) | add // 0),
thumbs_down: ($real | map(.reactions_down // 0) | add // 0) } )
'

# aggregate_snapshot <jsonl_dir> — PURE. Reads normalized JSONL and emits the
# per-bot headline-metric snapshot as compact JSON (also the WoW artifact payload):
# per-bot headline-metric snapshot as compact JSON (also the WoW artifact payload).
# Every eligible (non-draft) PR falls into exactly one bucket PER BOT, and the three
# buckets sum to eligible_prs:
# reviewed_prs — the bot gave a real review/comment
# refused_prs — the bot responded ONLY to decline (rate-limited / out of quota)
# no_response_prs — the bot never appeared on the PR
# { "eligible_prs": N, "total_prs": N,
# "bots": { "<login>": { prs_reviewed, reviews, inline_comments, approved,
# changes_req, commented, rate_limited, threads_total,
# "bots": { "<login>": { reviewed_prs, refused_prs, no_response_prs, reviews,
# approved, changes_req, inline_comments, threads_total,
# threads_resolved, thumbs_up, thumbs_down,
# latency_p50, latency_p95 } , ... } }
aggregate_snapshot() {
Expand All @@ -215,18 +225,23 @@ aggregate_snapshot() {
(map(select(.kind=="pr"))) as $prs
| (map(select(.kind=="bot_pr"))) as $bp
| ($prs | length) as $total
| ($prs | map(select(.draft == false)) | length) as $eligible
| ($bp | group_by(.bot) | map(
# Bucket over ELIGIBLE (non-draft) PRs only, so the three buckets sum to eligible.
| ([ $prs[] | select(.draft == false) | .pr ] | unique) as $elig_urls
| ($elig_urls | length) as $eligible
| ($bp | map(select(.pr as $u | $elig_urls | index($u)))) as $bpe
Comment thread
don-petry marked this conversation as resolved.
Resolved
| ($bpe | group_by(.bot) | map(
Comment thread
don-petry marked this conversation as resolved.
Resolved
(.[0].bot) as $bot
| (map(select((.real_responses // 0) > 0))) as $reviewed
| (map(select((.real_responses // 0) == 0 and (.refusals // 0) > 0))) as $refused
| ([.[].latency_s] | map(select(. != null)) | sort) as $lat
| { key: $bot, value: {
prs_reviewed: ([.[].pr] | unique | length),
reviewed_prs: ($reviewed | length),
refused_prs: ($refused | length),
no_response_prs: ($eligible - ($reviewed | length) - ($refused | length)),
reviews: ([.[].reviews] | add),
inline_comments: ([.[].inline_comments] | add),
approved: ([.[].approved] | add),
changes_req: ([.[].changes_req] | add),
commented: ([.[].commented] | add),
rate_limited: ([.[].rate_limited] | add),
inline_comments: ([.[].inline_comments] | add),
threads_total: ([.[].threads_total] | add),
threads_resolved: ([.[].threads_resolved] | add),
thumbs_up: ([.[].thumbs_up] | add),
Expand Down Expand Up @@ -274,64 +289,48 @@ render_reviewer_report() {
printf 'Deterministic report — every figure is computed with `jq`/`awk` from GitHub review data; '
printf 'no LLM is involved. Bots are identified by their GraphQL App login '
printf '(`scripts/lib/advisory-review-gate.sh`); rate-limit/out-of-quota notices are detected by body text. '
printf '_Participation_ = distinct non-draft PRs a bot touched ÷ %s review-eligible PRs. ' "$(_fmt_int "$eligible")"
printf '_Latency_ = time from PR creation to the bot'"'"'s first review/comment. '
printf 'Each of the **%s review-eligible (non-draft) PRs** falls into exactly one bucket per reviewer, so `Reviewed + Rate-limited + No response` = %s for every row. ' "$(_fmt_int "$eligible")" "$(_fmt_int "$eligible")"
printf '_Latency_ = time from PR creation to the bot'"'"'s first **real** review (refusals excluded). '
printf 'Deltas (▲/▼) are vs the prior week and directional only.\n\n'

# ---- Scorecard table -----------------------------------------------------
printf '## Scorecard\n\n'
printf '| Reviewer | PRs reviewed | Participation | Reviews | Comments/PR | Latency p50 | Latency p95 | ✅ / 🔄 | Rate-limited | Resolved | 👍/👎 |\n'
printf '|---|---:|---:|---:|---:|---:|---:|:--:|---:|---:|:--:|\n'
printf '| Reviewer | Reviewed | Rate-limited | No response | ✅ / 🔄 | Latency p50 | Latency p95 |\n'
printf '|---|---:|---:|---:|:--:|---:|---:|\n'

local bot label prs_reviewed reviews inline approved changes_req rate_limited thr_tot thr_res p50 p95 tup tdn
local bot label reviewed_prs refused_prs no_resp approved changes_req p50 p95
for bot in "${REVIEWER_BOTS[@]}"; do
label="${REVIEWER_LABELS[$bot]:-$bot}"
# Pull this bot's aggregates out of the snapshot (0 when absent this week).
# IFS=tab preserves empty latency fields (a missing percentile must not shift
# the remaining columns left — that was a field-misalignment bug).
IFS=$'\t' read -r prs_reviewed reviews inline approved changes_req rate_limited thr_tot thr_res tup tdn p50 p95 <<<"$(
jq -r --arg b "$bot" '
IFS=$'\t' read -r reviewed_prs refused_prs no_resp approved changes_req p50 p95 <<<"$(
jq -r --arg b "$bot" --argjson elig "${eligible:-0}" '
(.bots?[$b] // {}) as $x
| [ ($x.prs_reviewed // 0), ($x.reviews // 0), ($x.inline_comments // 0),
($x.approved // 0), ($x.changes_req // 0), ($x.rate_limited // 0),
($x.threads_total // 0), ($x.threads_resolved // 0),
($x.thumbs_up // 0), ($x.thumbs_down // 0),
| [ ($x.reviewed_prs // 0), ($x.refused_prs // 0),
($x.no_response_prs // $elig),
($x.approved // 0), ($x.changes_req // 0),
(if $x.latency_p50 == null then "" else $x.latency_p50 end),
(if $x.latency_p95 == null then "" else $x.latency_p95 end) ]
| @tsv' <<<"$snap"
)"

local prev_prs; prev_prs="$(_prev "$prev" "$bot" prs_reviewed)"
local part cpp
part="$(_fmt_pct "$prs_reviewed" "$eligible")"
cpp="$(awk -v c="$inline" -v p="$prs_reviewed" 'BEGIN { if (p<=0) print "—"; else printf "%.1f", c/p }')"
local resolved_pct; resolved_pct="$(_fmt_pct "$thr_res" "$thr_tot")"
printf '| %s | %s %s | %s | %s | %s | %s | %s | %s / %s | %s | %s | %s/%s |\n' \
"$label" "$(_fmt_int "$prs_reviewed")" "$(_delta_arrow "$prs_reviewed" "$prev_prs")" \
"$part" "$(_fmt_int "$reviews")" "$cpp" \
"$(_fmt_dur "$p50")" "$(_fmt_dur "$p95")" \
local prev_reviewed prev_refused
prev_reviewed="$(_prev "$prev" "$bot" reviewed_prs)"
prev_refused="$(_prev "$prev" "$bot" refused_prs)"
printf '| %s | %s %s | %s %s | %s | %s / %s | %s | %s |\n' \
"$label" \
"$(_fmt_int "$reviewed_prs")" "$(_delta_arrow "$reviewed_prs" "$prev_reviewed")" \
"$(_fmt_int "$refused_prs")" "$(_delta_arrow "$refused_prs" "$prev_refused")" \
"$(_fmt_int "$no_resp")" \
"$(_fmt_int "$approved")" "$(_fmt_int "$changes_req")" \
"$(_fmt_int "$rate_limited")" "$resolved_pct" \
"$(_fmt_int "$tup")" "$(_fmt_int "$tdn")"
"$(_fmt_dur "$p50")" "$(_fmt_dur "$p95")"
done
printf '\n'
printf '_✅ = APPROVED reviews · 🔄 = CHANGES_REQUESTED · Resolved = share of a bot'"'"'s inline threads later resolved · Comments/PR = inline findings per reviewed PR._\n\n'

# ---- Reliability ---------------------------------------------------------
printf '## Reliability & no-shows\n\n'
printf '| Reviewer | Eligible PRs missed | Coverage gap | Rate-limited PRs |\n'
printf '|---|---:|---:|---:|\n'
for bot in "${REVIEWER_BOTS[@]}"; do
label="${REVIEWER_LABELS[$bot]:-$bot}"
prs_reviewed="$(jq -r --arg b "$bot" '.bots[$b].prs_reviewed // 0' <<<"$snap")"
rate_limited="$(jq -r --arg b "$bot" '.bots[$b].rate_limited // 0' <<<"$snap")"
local missed gap
missed=$(( eligible - prs_reviewed )); [ "$missed" -lt 0 ] && missed=0
gap="$(_fmt_pct "$missed" "$eligible")"
printf '| %s | %s | %s | %s |\n' \
"$label" "$(_fmt_int "$missed")" "$gap" "$(_fmt_int "$rate_limited")"
done
printf '\n_Coverage gap = eligible PRs a bot did **not** touch. External SaaS reviewers only sample a subset of PRs; this is a participation signal, not necessarily a fault._\n\n'
printf -- '- **Reviewed** — the bot delivered a real review/comment on the PR.\n'
printf -- '- **Rate-limited** — the bot appeared but its *only* action was to decline (out of quota / rate-limited). It refused to review.\n'
printf -- '- **No response** — the bot never appeared on the PR.\n'
printf -- '- **✅ / 🔄** = APPROVED / CHANGES_REQUESTED review counts (a rate-limit notice is never counted as a review).\n\n'

# ---- Coverage / overlap --------------------------------------------------
local co_review multi_bot
Expand Down
8 changes: 4 additions & 4 deletions tests/fixtures/reviewer_jsonl/repo1.jsonl
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{"kind":"pr","repo":"petry-projects/markets","pr":"https://github.com/petry-projects/markets/pull/42","created":"2026-07-10T10:00:00Z","merged":"2026-07-10T13:00:00Z","draft":false,"author":"alice"}
{"kind":"bot_pr","repo":"petry-projects/markets","pr":"https://github.com/petry-projects/markets/pull/42","bot":"chatgpt-codex-connector","created":"2026-07-10T10:00:00Z","first_at":"2026-07-10T10:20:00Z","latency_s":1200,"reviews":0,"approved":0,"changes_req":0,"commented":0,"dismissed":0,"rate_limited":1,"inline_comments":0,"threads_total":0,"threads_resolved":0,"threads_outdated":0,"thumbs_up":0,"thumbs_down":0}
{"kind":"bot_pr","repo":"petry-projects/markets","pr":"https://github.com/petry-projects/markets/pull/42","bot":"coderabbitai","created":"2026-07-10T10:00:00Z","first_at":"2026-07-10T10:30:00Z","latency_s":1800,"reviews":1,"approved":0,"changes_req":0,"commented":1,"dismissed":0,"rate_limited":1,"inline_comments":0,"threads_total":0,"threads_resolved":0,"threads_outdated":0,"thumbs_up":0,"thumbs_down":0}
{"kind":"bot_pr","repo":"petry-projects/markets","pr":"https://github.com/petry-projects/markets/pull/42","bot":"copilot-pull-request-reviewer","created":"2026-07-10T10:00:00Z","first_at":"2026-07-10T10:03:06Z","latency_s":186,"reviews":1,"approved":0,"changes_req":0,"commented":1,"dismissed":0,"rate_limited":0,"inline_comments":1,"threads_total":1,"threads_resolved":1,"threads_outdated":0,"thumbs_up":2,"thumbs_down":1}
{"kind":"bot_pr","repo":"petry-projects/markets","pr":"https://github.com/petry-projects/markets/pull/42","bot":"gemini-code-assist","created":"2026-07-10T10:00:00Z","first_at":"2026-07-10T10:00:50Z","latency_s":50,"reviews":1,"approved":1,"changes_req":0,"commented":0,"dismissed":0,"rate_limited":0,"inline_comments":1,"threads_total":1,"threads_resolved":0,"threads_outdated":1,"thumbs_up":0,"thumbs_down":0}
{"kind":"bot_pr","repo":"petry-projects/markets","pr":"https://github.com/petry-projects/markets/pull/42","bot":"chatgpt-codex-connector","created":"2026-07-10T10:00:00Z","real_responses":0,"refusals":1,"latency_s":null,"reviews":0,"approved":0,"changes_req":0,"inline_comments":0,"threads_total":0,"threads_resolved":0,"thumbs_up":0,"thumbs_down":0}
{"kind":"bot_pr","repo":"petry-projects/markets","pr":"https://github.com/petry-projects/markets/pull/42","bot":"coderabbitai","created":"2026-07-10T10:00:00Z","real_responses":0,"refusals":1,"latency_s":null,"reviews":0,"approved":0,"changes_req":0,"inline_comments":0,"threads_total":0,"threads_resolved":0,"thumbs_up":0,"thumbs_down":0}
{"kind":"bot_pr","repo":"petry-projects/markets","pr":"https://github.com/petry-projects/markets/pull/42","bot":"copilot-pull-request-reviewer","created":"2026-07-10T10:00:00Z","real_responses":2,"refusals":0,"latency_s":186,"reviews":1,"approved":0,"changes_req":0,"inline_comments":1,"threads_total":1,"threads_resolved":1,"thumbs_up":2,"thumbs_down":1}
{"kind":"bot_pr","repo":"petry-projects/markets","pr":"https://github.com/petry-projects/markets/pull/42","bot":"gemini-code-assist","created":"2026-07-10T10:00:00Z","real_responses":2,"refusals":0,"latency_s":50,"reviews":1,"approved":1,"changes_req":0,"inline_comments":1,"threads_total":1,"threads_resolved":0,"thumbs_up":0,"thumbs_down":0}
Loading
Loading