Skip to content

feat(review): multi-model debate orchestrator - #9429

Closed
joahg wants to merge 1 commit into
aaif-goose:mainfrom
joahg:feature/goose-review-debate
Closed

feat(review): multi-model debate orchestrator#9429
joahg wants to merge 1 commit into
aaif-goose:mainfrom
joahg:feature/goose-review-debate

Conversation

@joahg

@joahg joahg commented May 26, 2026

Copy link
Copy Markdown
Collaborator

What

Adds an opt-in debate path to goose review:

goose review \\
  --debate-models google/gemini-3.1-pro-preview,anthropic/claude-sonnet-4.5,openai/gpt-5.5 \\
  --debate-rounds 3 \\
  --debate-min-agreement 2

Round 1 each debater reviews independently. Rounds 2..=R show each
debater the anonymized findings from every other debater and ask
it to keep / drop / refine with cited evidence, or add new claims.
After the last round, findings cluster by (path, line_start, line_end) and are emitted only if min_agreement debaters retained
the cluster.

Why

Inspired by the Milvus AI code-review debate experiment
which measured a 53% → 80% bug-recall lift moving from a single
model to a 5-model × 5-round debate. goose review is one of the
few reviewer entrypoints where the full pipeline is open and
extensible end-to-end, which makes it a natural place to try the
debate pattern.

Design

  • New module crates/goose-cli/src/commands/review/debate.rs.
  • Reuses run_subprocess_for_findings and split_diff_by_file from
    the existing orchestrator — same timeout, kill_on_drop, per-file
    fan-out semantics.
  • Debater labels (A, B, C, …) are opaque; real provider/model names
    never appear in peer prompts. (Milvus found small models defer to
    recognized large ones when identified.)
  • Per-debater self-duplicates can't satisfy min_agreement (label
    dedup happens before counting) so a verbose debater can't outvote
    the others by emitting duplicates.
  • Concurrency cap scales with debater count up to 16 (each debater
    hits a different provider).
  • Rounds hard-capped to 5; --debate-min-agreement floored at 1.
  • --debate-models rejected with --no-orchestrate (debate
    inherently relies on subprocess fan-out).

Output schema compatibility

Finding gains two optional fields:

#[serde(default, skip_serializing_if = "Option::is_none")]
pub round: Option<u32>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub agreement: Option<Vec<String>>,

For single-model runs, the JSONL emitted on stdout is byte-for-byte
identical
to today's output, so any existing parser of the JSONL
stream keeps working unchanged. Debate runs emit the same shape plus
the two optional fields, which downstream consumers can opt into.

Tests

15 new unit tests under commands::review::debate::tests (17 total
in the module). All pass plus the pre-existing review tests.
cargo clippy -p goose-cli --lib --no-deps -- -D warnings is clean.

Covered:

  • label generation (A..Z, then AA..AZ, BA..)
  • debater parsing — whitespace, missing slash, empty provider/model,
    trailing commas, fewer-than-two
  • concurrency scaling (4, 6, 10, 16, capped at 16)
  • agreement aggregation — union (k=1), majority, self-duplicate dedup
  • severity merging (max wins, longest summary tiebreak)
  • round-1 prompt has no debate header or peer block
  • round-2 prompt anonymizes peers, demands evidence, includes
    keep/drop/refine/add rubric, surfaces reviewer instructions
  • empty self-prior renders [] marker (no panic)
  • render_findings_block strips check, round, agreement so
    peers never see the metadata

Out of scope (follow-ups)

  • Per-check debate (this PR covers the main correctness pass
    only; checks still run with the single per-check model).
  • LLM judge / peer-evaluation round. The cheap path (agreement
    count) ships first; an LLM judge can be a follow-up once we have
    eval signal on the cheap path.
  • Benchmarking against a known-bug fixture set.

Adds an opt-in debate path to 'goose review' that runs the main
correctness pass through N (provider, model) debaters over R rounds.
Round 1 each debater reviews independently; rounds 2..=R show each
debater the anonymized findings from every other debater in the
previous round and ask it to keep / drop / refine claims with cited
evidence, or add new claims. After the last round, findings are
clustered by (path, line_start, line_end) and only emitted when at
least '--debate-min-agreement' debaters retained the cluster.

Inspired by https://milvus.io/blog/ai-code-review-gets-better-when-models-debate-claude-vs-gemini-vs-codex-vs-qwen-vs-minimax.md
which measured a 53% -> 80% bug-recall lift moving from a single
model to a 5-model x 5-round debate.

Design highlights:

- New 'crates/goose-cli/src/commands/review/debate.rs'; 'Debater',
  'DebateOptions', 'parse_debaters', 'run_debate_review',
  'build_debate_round_prompt', 'aggregate_by_agreement'.
- 'Finding' gains two optional fields ('round', 'agreement')
  serialized with skip_serializing_if so the JSONL emitted by single-
  model runs is byte-for-byte unchanged; downstream parsers that want
  the new fields can opt in.
- Reuses 'run_subprocess_for_findings' (now pub(super)) and
  'split_diff_by_file' so timeout, kill_on_drop, and per-file fan-out
  semantics stay identical to the existing single-model orchestrator.
- Debater labels (A, B, C, ...) are opaque; real provider/model names
  never appear in peer prompts -- Milvus found small models defer to
  recognized large ones when identified.
- Per-debater self-duplicates can't satisfy 'min_agreement' (label
  dedup happens before counting), so a verbose debater can't outvote
  the others by emitting duplicates.
- Concurrency cap scales with debater count up to 16 (each debater
  hits a different provider so rate-limit budget is per-provider).
- Rounds hard-capped to 5; '--debate-min-agreement' floored at 1.
- '--debate-models' is rejected with '--no-orchestrate' (the debate
  inherently relies on subprocess fan-out).

CLI:

  goose review \\
    --debate-models google/gemini-3.1-pro-preview,anthropic/claude-sonnet-4.5,openai/gpt-5.5 \\
    --debate-rounds 3 \\
    --debate-min-agreement 2

15 new unit tests under 'commands::review::debate::tests' covering
label generation, debater parsing (whitespace, error cases),
concurrency scaling, agreement aggregation (including self-dedup),
severity merging, prompt rendering for round 1 and round 2+, and
peer-view schema (no leaked metadata fields).

Amp-Thread-ID: https://ampcode.com/threads/T-019e6661-1089-71ab-b5af-defc029195b6
Co-authored-by: Amp <amp@ampcode.com>
@joahg
joahg force-pushed the feature/goose-review-debate branch from bba0ef0 to 12850ce Compare May 26, 2026 22:51
@joahg
joahg marked this pull request as ready for review May 26, 2026 22:52
@joahg
joahg requested a review from DOsinga May 26, 2026 22:55
@joahg joahg closed this Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant