feat(review): multi-model debate orchestrator - #9429
Closed
joahg wants to merge 1 commit into
Closed
Conversation
Adds an opt-in debate path to 'goose review' that runs the main correctness pass through N (provider, model) debaters over R rounds. Round 1 each debater reviews independently; rounds 2..=R show each debater the anonymized findings from every other debater in the previous round and ask it to keep / drop / refine claims with cited evidence, or add new claims. After the last round, findings are clustered by (path, line_start, line_end) and only emitted when at least '--debate-min-agreement' debaters retained the cluster. Inspired by https://milvus.io/blog/ai-code-review-gets-better-when-models-debate-claude-vs-gemini-vs-codex-vs-qwen-vs-minimax.md which measured a 53% -> 80% bug-recall lift moving from a single model to a 5-model x 5-round debate. Design highlights: - New 'crates/goose-cli/src/commands/review/debate.rs'; 'Debater', 'DebateOptions', 'parse_debaters', 'run_debate_review', 'build_debate_round_prompt', 'aggregate_by_agreement'. - 'Finding' gains two optional fields ('round', 'agreement') serialized with skip_serializing_if so the JSONL emitted by single- model runs is byte-for-byte unchanged; downstream parsers that want the new fields can opt in. - Reuses 'run_subprocess_for_findings' (now pub(super)) and 'split_diff_by_file' so timeout, kill_on_drop, and per-file fan-out semantics stay identical to the existing single-model orchestrator. - Debater labels (A, B, C, ...) are opaque; real provider/model names never appear in peer prompts -- Milvus found small models defer to recognized large ones when identified. - Per-debater self-duplicates can't satisfy 'min_agreement' (label dedup happens before counting), so a verbose debater can't outvote the others by emitting duplicates. - Concurrency cap scales with debater count up to 16 (each debater hits a different provider so rate-limit budget is per-provider). - Rounds hard-capped to 5; '--debate-min-agreement' floored at 1. - '--debate-models' is rejected with '--no-orchestrate' (the debate inherently relies on subprocess fan-out). CLI: goose review \\ --debate-models google/gemini-3.1-pro-preview,anthropic/claude-sonnet-4.5,openai/gpt-5.5 \\ --debate-rounds 3 \\ --debate-min-agreement 2 15 new unit tests under 'commands::review::debate::tests' covering label generation, debater parsing (whitespace, error cases), concurrency scaling, agreement aggregation (including self-dedup), severity merging, prompt rendering for round 1 and round 2+, and peer-view schema (no leaked metadata fields). Amp-Thread-ID: https://ampcode.com/threads/T-019e6661-1089-71ab-b5af-defc029195b6 Co-authored-by: Amp <amp@ampcode.com>
joahg
force-pushed
the
feature/goose-review-debate
branch
from
May 26, 2026 22:51
bba0ef0 to
12850ce
Compare
joahg
marked this pull request as ready for review
May 26, 2026 22:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an opt-in debate path to
goose review:Round 1 each debater reviews independently. Rounds 2..=R show each
debater the anonymized findings from every other debater and ask
it to keep / drop / refine with cited evidence, or add new claims.
After the last round, findings cluster by
(path, line_start, line_end)and are emitted only ifmin_agreementdebaters retainedthe cluster.
Why
Inspired by the Milvus AI code-review debate experiment
which measured a 53% → 80% bug-recall lift moving from a single
model to a 5-model × 5-round debate.
goose reviewis one of thefew reviewer entrypoints where the full pipeline is open and
extensible end-to-end, which makes it a natural place to try the
debate pattern.
Design
crates/goose-cli/src/commands/review/debate.rs.run_subprocess_for_findingsandsplit_diff_by_filefromthe existing orchestrator — same timeout, kill_on_drop, per-file
fan-out semantics.
never appear in peer prompts. (Milvus found small models defer to
recognized large ones when identified.)
min_agreement(labeldedup happens before counting) so a verbose debater can't outvote
the others by emitting duplicates.
hits a different provider).
--debate-min-agreementfloored at 1.--debate-modelsrejected with--no-orchestrate(debateinherently relies on subprocess fan-out).
Output schema compatibility
Findinggains two optional fields:For single-model runs, the JSONL emitted on stdout is byte-for-byte
identical to today's output, so any existing parser of the JSONL
stream keeps working unchanged. Debate runs emit the same shape plus
the two optional fields, which downstream consumers can opt into.
Tests
15 new unit tests under
commands::review::debate::tests(17 totalin the module). All pass plus the pre-existing review tests.
cargo clippy -p goose-cli --lib --no-deps -- -D warningsis clean.Covered:
trailing commas, fewer-than-two
keep/drop/refine/add rubric, surfaces reviewer instructions
[]marker (no panic)render_findings_blockstripscheck,round,agreementsopeers never see the metadata
Out of scope (follow-ups)
only; checks still run with the single per-check model).
count) ships first; an LLM judge can be a follow-up once we have
eval signal on the cheap path.