docs(problems): add graduated approval policy problem doc

[Benkapner]

Review agents currently make a single binary decision: approve or request changes. There is no middle ground. This forces a trade-off between two failure modes.

When the agent leans toward approving, risky changes slip through. Submodule bumps get approved without the agent inspecting the actual changes (#1462). Medium-severity correctness findings get suppressed to avoid false-positive noise (#1453). Author uncertainty signals like "i think this is right but i'm not sure" get ignored in the approval decision (#1143).

When the agent leans toward blocking, legitimate work gets stuck. Humans spend time reviewing changes that are almost certainly fine, defeating the purpose of automation.

Human reviewers don't work this way. A human would say "LGTM, but get a second pair of eyes on the crypto changes" or "looks fine to me but the test coverage concerns me." That graduated response doesn't exist for agents today. It's yes or no.

This doc proposes replacing the binary verdict with risk-scored approval routing. The review agent (or a separate scoring layer) assigns a risk score based on multiple signals:

From the diff: which files changed (auth code is riskier than docs), change type (deletions score higher than additions), whether tests changed too (no tests = higher risk), and whether binary/opaque files are present.

From the review findings: number and severity of findings, confidence level, and whether sub-agents disagreed with each other (disagreement = uncertainty = higher risk).

From context: author history (first-time contributors score higher), branch target (default branch = higher risk), and whether related code was recently modified.

The score maps to a routing action:

Score	Action
Low (0-30)	Auto-approve
Medium-low (30-50)	Approve with advisory comment
Medium (50-70)	Request secondary human review
High (70-90)	Request changes with explanation
Critical (90-100)	Block and alert security team

Thresholds are configurable per repo and per org.

The doc explores three implementation approaches: scoring inside the review agent (simplest but model-dependent), a separate deterministic scoring layer in the harness (more robust against manipulation), and multi-reviewer consensus where disagreement triggers escalation (expensive but naturally handles uncertainty). The parallel sub-agent architecture from #1550 partially enables the third approach, since the Challenger role already contests findings, but disagreement between sub-agents does not currently influence the approval decision.

Addresses #1143, #1453, and fullsend-ai/agents#189.

[fullsend-ai-review]

Review

Findings

Medium

• [logic-error] docs/problems/graduated-approval-policy.md:89 — Approach 3 states "Challenger outcomes do not currently influence the approval decision; they only affect whether a finding survives." This is factually incorrect. In the current implementation (pr-review SKILL.md step 6e–6f), the Challenger pass removes or retains findings, and the verdict in step 6f is determined by the severity of surviving findings. By removing a high-severity finding, the Challenger directly changes the outcome from request-changes to approve. Challenger outcomes influence the approval decision precisely because finding survival determines the verdict.
  Remediation: Reword to: "Challenger outcomes indirectly influence the approval decision by determining which findings survive to the verdict step, but the Challenger is an intra-agent verification step within the orchestrator's own process, not disagreement between independent sub-agents."

• [architectural-coherence] docs/problems/graduated-approval-policy.md — This document overlaps significantly with existing discussions in code-review.md without explaining the relationship. Specifically: (1) code-review.md lines 158–163 describe an "Escalation-based" composition model where sub-agents produce findings and a separate decision layer evaluates them — essentially the pattern this doc proposes with risk scoring. (2) code-review.md lines 166–170 ("The confidence problem") asks the same core questions: when should agents escalate, how do they express uncertainty? (3) autonomy-spectrum.md lines 67–94 describe "per-decision escalation dimensions" (reversibility, blast radius, visibility) that also modulate per-change risk assessment at runtime. The new doc should position itself relative to these existing mechanisms — is it an elaboration of the escalation-based model, a replacement, or a complementary layer?
  Remediation: Add a paragraph near the beginning explaining how this doc relates to code-review.md's escalation-based composition model and confidence problem, and to autonomy-spectrum.md's per-decision escalation dimensions.

Low

• [edge-case] docs/problems/graduated-approval-policy.md:56 — The routing rules table has overlapping score boundaries. A score of exactly 30, 50, 70, or 90 falls into two adjacent ranges (e.g., 0–30 is "Low" and 30–50 is "Medium-low"). While the doc notes thresholds are configurable and this is a design exploration rather than a spec, clarifying boundary behavior (e.g., exclusive upper bounds) would prevent ambiguity if implemented directly.

• [naming-alignment] docs/problems/graduated-approval-policy.md — The filename uses a "-policy" suffix that doesn't match the problem-domain naming pattern of other docs (code-review, autonomy-spectrum, intent-representation, security-threat-model). Per CLAUDE.md, these are design explorations, not specs or policies. Consider a name that describes the problem space rather than a solution: e.g., "review-verdict-composition.md" or "approval-decision-model.md."

• [heading-structure] docs/problems/graduated-approval-policy.md:12 — "## The problem" is a generic heading. Other problem docs use descriptive titles that capture the specific issue: code-review.md uses "Why code review is the hardest problem", autonomy-spectrum.md uses "The model: binary with CODEOWNERS." A more descriptive heading would improve scannability.

• [cross-reference-consistency] docs/problems/graduated-approval-policy.md:92 — PR feat(review): parallel specialized sub-agents for PR review #1550 is referenced without sufficient context for readers who haven't seen it. The doc's own problem statement provides good context for Review agent should consider author uncertainty signals before approving #1143, Escalate medium-severity correctness findings instead of auto-approving #1453, Review agent cannot inspect submodule bump contents — blind approval risk agents#189 — apply the same pattern to feat(review): parallel specialized sub-agents for PR review #1550 (e.g., "PR feat(review): parallel specialized sub-agents for PR review #1550, which introduced the parallel sub-agent architecture").

Previous run

Review

Findings

Medium

• [internal-consistency] docs/problems/graduated-approval-policy.md:97 — The document states "The autonomy-spectrum.md defines per-repo autonomy tiers (Tier 0-3) based on change type." This is factually incorrect on two counts: (1) autonomy-spectrum.md defines a binary per-repo autonomy model (autonomous or not), not tiered levels; (2) Tier 0-3 classification is defined in intent-representation.md as intent tiers based on change type (standing rules, tactical, strategic, organizational). The document conflates two distinct concepts.
  Remediation: Correct to reference both documents accurately, e.g.: "The autonomy-spectrum.md defines a binary per-repo autonomy model. Separately, intent-representation.md defines per-change intent tiers (Tier 0-3). Graduated approval operates orthogonally to both: even among changes at the same intent tier within an autonomous repo, some are higher risk than others."

• [broken-reference] docs/problems/graduated-approval-policy.md:106 — The document links to [tool-call-risk-assessment.md](tool-call-risk-assessment.md) but this file does not exist in docs/problems/. While the text hedges with "If a tool call risk assessment layer is implemented," the markdown link syntax implies the document exists and creates a broken link.
  Remediation: Either remove the hyperlink and keep it as plain text (e.g., "If a tool call risk assessment layer is implemented"), or create a stub document, or note explicitly that this references a planned document.

• [missing-readme-link] docs/problems/graduated-approval-policy.md — CLAUDE.md states: "When adding new problem areas, create a new file in docs/problems/ and link it from README.md." The README.md maintains an explicit index of all problem documents, and this new document is not listed. This violates an explicit repository convention.
  Remediation: Add an entry for graduated-approval-policy.md to the problem document list in README.md, following the existing format with a one-line description.

Low

• [internal-consistency] docs/problems/graduated-approval-policy.md:88 — The document claims the Challenger role is part of "the parallel sub-agent architecture (PR feat(review): parallel specialized sub-agents for PR review #1550)" and frames it as inter-agent disagreement ("disagreement between sub-agents does not currently influence the approval decision"). In the current codebase, the Challenger is step 6e of the orchestrator's own process — a self-verification pass within a single agent run, not disagreement between independent sub-agents.
  Remediation: Verify how PR feat(review): parallel specialized sub-agents for PR review #1550 implements the Challenger concept and correct the characterization if it is an intra-agent verification step rather than inter-agent disagreement.

• [edge-case] docs/problems/graduated-approval-policy.md:61 — The routing rules and the CODEOWNERS relationship section create an ambiguity for mixed-path PRs: when a PR touches both CODEOWNERS-guarded and non-guarded paths, it is unclear whether the graduated policy applies to the non-guarded portion only or is bypassed entirely.
  Remediation: Add a note clarifying interaction semantics for mixed-path PRs.

• [algorithm-logic] docs/problems/graduated-approval-policy.md:38 — The input signals state "deletions and modifications score higher than additions" as a general rule. In security contexts, additions can be equally or more risky — adding a new API endpoint, a new dependency, or a new permission grant can have greater blast radius than modifying existing code.
  Remediation: Qualify the statement to acknowledge that additions of new attack surface should be scored independently.

[fullsend-ai-review]

See the review comment for full details.

[Benkapner]

Addressed all findings in b099173.

[internal-consistency] (medium, line 97) Corrected the autonomy-spectrum.md reference. The autonomy spectrum defines binary per-repo autonomy (autonomous or not), while Tier 0-3 classification comes from intent-representation.md. Updated the text to reference both documents accurately and clarify that graduated approval operates orthogonally to both.

[broken-reference] (medium, line 106) Replaced the broken markdown link to tool-call-risk-assessment.md with a reference to PR #2009 instead. The document exists on a separate branch and is not yet merged, so a relative link would 404.

[missing-readme-link] (medium) Added README.md entry for graduated-approval-policy.md, positioned after the Code Review entry.

[internal-consistency] (low, line 88) Corrected the Challenger characterization. The Challenger is an intra-agent verification step within the orchestrator's own process (step 6e), not disagreement between independent sub-agents. Updated the text to reflect this distinction.

[edge-case] (low, line 61) Added a note clarifying mixed-path PR semantics: when a PR touches both CODEOWNERS-guarded and non-guarded paths, CODEOWNERS takes precedence for the entire PR, but the risk score still provides context to the human reviewer.

[algorithm-logic] (low, line 38) Qualified the change-type scoring statement. Additions of new attack surface (API endpoints, dependencies, permission grants) can have equal or greater blast radius than modifications to existing code, and should be scored independently.

[github-actions]

Site preview

Preview: https://75d40f56-site.fullsend-ai.workers.dev

Commit: b099173defcd0cf023271862149fc02bccc82c5a

[ralphbean]

I like this idea — a rubric for review sub-agents to score against, with the coordinator making a routing decision based on that. The problem statement is sharp and the referenced issues make the case well.

But I think this is a solution proposal, not a problem doc. The first 25 lines frame the problem (binary verdicts are insufficient), and then the remaining 90 lines propose a specific architecture: numeric scoring, a 5-tier routing table, three implementation approaches. That's an ADR.

We've been making changes to the agents without eval coverage, and it's been bugging me. A change like this — restructuring how the review agent renders verdicts — is exactly the kind of thing we should have evals in place for before we move on it. #1682 and @ascerra's eval work are getting us there.

If you're up for it: write eval cases where the current review agent gets it wrong (approves something it shouldn't, blocks something it shouldn't). Show that graduated scoring produces better outcomes on those cases. TDD for agent architecture. Then propose the ADR with those cases as evidence, implement it, and let the evals tell us if it's actually better.

Closing this, but the idea has legs. Thanks for the writeup.

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 6:39 PM UTC · Completed 6:44 PM UTC
Commit: b099173 · View workflow run →

[fullsend-ai-retro]

Retro: PR #2012 — graduated approval policy problem doc

What happened

PR #2012 added a problem doc proposing risk-scored approval routing. The review agent requested changes, finding legitimate issues: incorrect cross-references between autonomy-spectrum.md and intent-representation.md, a broken link to a non-existent file, and a mischaracterization of the Challenger role. The author addressed all findings in one iteration.

However, a human reviewer (ralphbean) then closed the PR without merging for a completely different reason: the document was structured as a solution proposal / ADR, not a problem doc. The first ~25 lines framed the problem, but the remaining ~90 lines proposed a specific architecture with scoring tables and implementation approaches.

The review agent caught low-level correctness issues but missed the highest-impact structural concern — the one that actually determined the PR's outcome.

Proposals

1 proposal filed. 1 candidate skipped due to overlap with existing issues (#2233, #1906 cover doc convention enforcement and structure-aware review broadly).

Proposals filed

• Review agent should detect document-type misclassification in docs PRs (in fullsend-ai/fullsend)