ci: add a bounded 1-round reviewer↔fixer rebuttal + stronger verify by allxsmith · Pull Request #246 · allxsmith/bestax

allxsmith · 2026-07-07T06:15:55Z

What

Gives the deep reviewer a single rebuttal round when the fix agent refutes one of its findings, and makes the verify/re-review pass thorough + well-formatted + human (matching #245).

Why

Today the reviewer only re-engages to verify a Fixed in <sha> claim. When the fixer refutes a finding, the loop jumps straight to needs-human-review — the reviewer never gets to concede or push back. Our own research (14 Bun PRs) shows robobun's reviewers don't rebut at all, so 1 round is already beyond Bun — but it's a real quality gain: the reviewer can catch a wrong refutation before it reaches you, and it produces one watchable exchange.

State machine (rounds = Claude's comment count in the thread, cap < 2)

finding → fixer refutes → REVIEWER rebuts once (or concedes)
        → fixer answers (fix / hold)
        → still contested → needs-human-review

Route a refuted deep-review thread (fixer spoke last, no Fixed in, ccount < 2) to the reviewer (verify), not the fixer.
After the reviewer's one rebuttal, the thread returns to the fixer; if the fixer holds again (ccount ≥ 2), the existing terminal hands it to a human. No new job, no infinite loop.
Fix terminal uses the same rebuttal-aware ACTIONABLE, so a just-refuted thread re-dispatches to the reviewer instead of escalating.
Fixer prompt skips a claude thread whose last comment is its own refutation (reviewer's turn) and answers the reviewer's rebuttal when it lands.
Continue-loop guard also pauses if a rebuttable thread is left untouched — a no-op reviewer can't self-dispatch forever.

Stronger verify reviewer

Re-reads the code and runs the test before concluding (no rubber-stamping), CodeRabbit-grade Markdown + a human surfer sign-off (matching #245's deep review), --max-turns 15 → 25, broadened read-only allowlist (gh api + inspection utils).

Testing

Routing jq unit-tested against all 6 thread states → ACTIONABLE / PENDING_VERIFY / REBUTTABLE = 4 / 1 / 1 as intended.
Anti-loop guard jq unit-tested → pauses only on unverified-fix or untouched-rebuttable threads.
YAML validated.

Scope

claude-pr-loop.yml only (gate + fix terminal + fixer prompt skip + verify job/prompt/guard). No new jobs, permissions, or models.

Generated by Claude Code

Summary by CodeRabbit

Bug Fixes
- Improved review-loop handling so unresolved feedback is routed more accurately between automated fixing and verification.
- Reduced repeated re-processing of already-addressed review comments, helping avoid unnecessary back-and-forth.
- Made escalation behavior more consistent when a response is still under review or needs another rebuttal.

Today the deep reviewer only re-engages to verify a 'Fixed in <sha>' claim; when the fix agent REFUTES a finding, the loop escalates straight to a human without the reviewer getting to concede or push back. Bun's reviewers don't rebut at all, so 1 round is already beyond them — but it adds real value: the reviewer can catch a WRONG refutation before it reaches a human, and it produces one watchable exchange. Gate/state-machine (rounds tracked by claude comment count, capped at 1): - Route a refuted deep-review thread (fixer spoke last, no 'Fixed in', claude ccount < 2) to the REVIEWER (verify) instead of the fixer. - After the reviewer's single rebuttal the thread returns to the fixer; if the fixer holds again (ccount >= 2, still refuted) it stays actionable and the existing terminal hands it to a human. No new job, no infinite loop. - The fix terminal uses the same rebuttal-aware ACTIONABLE so a just-refuted thread re-dispatches to the reviewer rather than escalating. - Fixer prompt now skips a claude thread whose last comment is its own refutation (the reviewer's turn), and answers the reviewer's rebuttal when it comes. - Continue-loop guard also pauses if a rebuttable thread is left untouched, so a no-op reviewer can't self-dispatch forever. Stronger verify reviewer: - Actually re-reads the code and runs the test before concluding (no rubber-stamping), CodeRabbit-grade Markdown formatting + a human surfer sign-off (matching the deep-review pass), turns 15 -> 25, and a broadened read-only allowlist (gh api + inspection utils). Routing and the anti-loop guard are unit-tested against every thread state. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVR5yWevceZEFbTJmpviiM

coderabbitai · 2026-07-07T06:16:13Z

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 4d5270ce-6807-471a-ae22-4b5291f7b636

📥 Commits

Reviewing files that changed from the base of the PR and between c4e80c0 and fa98ff9.

📒 Files selected for processing (1)

.github/workflows/claude-pr-loop.yml

Walkthrough

This PR refines the Claude PR loop GitHub Actions workflow to introduce rebuttal-aware classification of unresolved review threads. A new REBUTTABLE thread bucket is added alongside ACTIONABLE and PENDING_VERIFY, affecting gate decisions, fix-agent behavior, terminal-state escalation, and post-verify loop continuation logic.

Changes

Claude PR Loop rebuttal handling

Layer / File(s)	Summary
Mode documentation and gate thread classification `.github/workflows/claude-pr-loop.yml`	Updates the top-level mode comment and refactors the gate's GraphQL thread classification to compute ACTIONABLE, PENDING_VERIFY, and a new REBUTTABLE set, with verify now triggered when REBUTTABLE > 0 in addition to PENDING_VERIFY > 0.
Fix-agent skip logic for self-refuted threads `.github/workflows/claude-pr-loop.yml`	Updates fix-agent instructions to skip Claude threads whose latest reply is the agent's own prior refutation until the reviewer rebuts it.
Terminal-state ACTIONABLE and continue-loop gating `.github/workflows/claude-pr-loop.yml`	Aligns terminal-state ACTIONABLE selection with gate logic and updates post-verify continue-loop condition to consider last comment author and Claude comment counts rather than just "Fixed in" prefix.
Verify-agent prompt and turn limits `.github/workflows/claude-pr-loop.yml`	Increases max-turns, updates allowed tools, and rewrites the verify prompt with detailed instructions for resolving "Verified fixed" cases and a single rebuttal-or-concede round.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

allxsmith/bestax#216: Both PRs modify the claude-pr-loop.yml state machine logic (gate/fix/verify loop behavior and terminal escalation conditions).
allxsmith/bestax#228: Both PRs modify the fix agent's behavior/prompt for handling disputed or rebuttal review threads.
allxsmith/bestax#244: Both PRs update fix job handling of unresolved threads and the "Fixed in " marker logic.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch claude/one-round-rebuttal

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-07-07T06:18:05Z

Preview Deployment

Preview URL: https://a1711110.bestax.pages.dev

github-actions · 2026-07-08T12:44:04Z

🎉 This PR is included in version 5.3.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

github-actions · 2026-07-11T12:10:39Z

🎉 This PR is included in version 3.2.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

github-actions · 2026-07-20T23:55:10Z

🎉 This PR is included in version 1.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

bestax-release-bot · 2026-08-12T03:46:07Z

🎉 This PR is included in version 1.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

allxsmith merged commit 2773f8f into main Jul 7, 2026
8 of 9 checks passed

github-actions Bot added the released label Jul 8, 2026

coderabbitai Bot mentioned this pull request Jul 22, 2026

ci: post story screenshots at ai-loop handoff (#284) #352

Merged

allxsmith mentioned this pull request Sep 2, 2026

test(bestax-migrate): keep the fixture manifests out of Dependabot's reach #630

Merged

Navigation Menu

Repository navigation

ci: add a bounded 1-round reviewer↔fixer rebuttal + stronger verify - #246

ci: add a bounded 1-round reviewer↔fixer rebuttal + stronger verify#246
allxsmith merged 1 commit into
mainfrom
claude/one-round-rebuttal

allxsmith commented Jul 7, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 7, 2026 •

edited

Loading

Review failed

Uh oh!

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

github-actions Bot commented Jul 20, 2026

Uh oh!

bestax-release-bot Bot commented Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

allxsmith commented Jul 7, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

State machine (rounds = Claude's comment count in the thread, cap < 2)

Stronger verify reviewer

Testing

Scope

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Walkthrough

Changes

Uh oh!

Uh oh!

github-actions Bot commented Jul 7, 2026

Preview Deployment

Uh oh!

github-actions Bot commented Jul 8, 2026

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

github-actions Bot commented Jul 20, 2026

Uh oh!

bestax-release-bot Bot commented Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

allxsmith commented Jul 7, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 7, 2026 •

edited

Loading