docs(problems): add graduated approval policy problem doc

[Benkapner]

Summary

• Reframes the graduated approval concept from docs(problems): add graduated approval policy problem doc #2012 as a pure problem statement, without prescribing a specific architecture
• Documents the binary verdict's two failure modes (too aggressive, too conservative) with concrete examples from Review agent cannot inspect submodule bump contents — blind approval risk agents#189, Escalate medium-severity correctness findings instead of auto-approving #1453, and Review agent should consider author uncertainty signals before approving #1143
• Describes what graduated approval would need (confidence level, risk signals, routing recommendations) without proposing scoring algorithms or routing tables
• Includes a "path forward" section aligned with @ralphbean's feedback on docs(problems): add graduated approval policy problem doc #2012: eval cases first, then ADR with evidence

This is a follow-up to #2012, which was closed with the feedback: "the idea has legs" but "the first 25 lines frame the problem well, and then the remaining 90 lines propose a specific architecture: numeric scoring, a 5-tier routing table, three implementation approaches. That's an ADR." This PR extracts the problem statement only. The solution architecture will come as an ADR backed by evaluation evidence, following the TDD approach ralph suggested.

Also addresses the bot review findings from #2012: corrected the autonomy-spectrum reference (binary per-repo autonomy, not Tier 0-3), fixed the broken link to tool-call-risk-assessment.md (which now exists since #2009 was merged), and clarified the CODEOWNERS interaction for mixed-path PRs.

[github-actions]

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

[qodo-code-review]

PR Summary by Qodo

Add graduated approval policy problem statement doc

📝 Documentation 🕐 10-20 Minutes

AI Description

• Add a problem-focused doc for graduated approval (no solution architecture).
• Document binary approve/block failure modes with concrete incident references.
• Outline required inputs and an eval-first path toward an evidence-backed ADR.

Diagram

graph TD
  A["Review agent"] --> B["Verdict output"] --> C{"Binary today?"}
  C -->|"Yes"| D["Approve / Block"] --> E["Outcomes"]
  C -->|"No (future)"| F["Graduated verdict"] --> G["Routing action"] --> E
  H["Risk signals"] --> F
  I["Confidence level"] --> F
  J["CODEOWNERS rules"] --> G

Loading

High-Level Assessment

The PR’s approach—capturing graduated approval strictly as a problem statement and deferring solution details to an ADR backed by eval evidence—is the right separation of concerns for this stage.

Files changed (1) +87 / -0

Documentation (1) +87 / -0

graduated-approval-policy.mdAdd graduated approval policy problem statement +87/-0

Add graduated approval policy problem statement

• Introduces a new problem doc describing why binary approve/block creates aggressive vs. conservative failure modes. Defines the minimum information a graduated verdict would need (confidence, risk signals, routing recommendation) and lays out an eval-first path toward an evidence-backed ADR.

docs/problems/graduated-approval-policy.md

[github-actions]

Site preview

Preview: https://7164d0ff-site.fullsend-ai.workers.dev

Commit: cc713720c088200e871181fde02cdb79eba3118c

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (1) 📎 Requirement gaps (0) 📜 Skill insights (0)

Context used

✅ Tickets: 🎫 Review agent cannot inspect submodule bump contents — blind approval risk 🎫 Review agent should consider author uncertainty signals before approving

✅ Compliance rules (platform): 54 rules

✅ Skills: writing-user-docs, writing-adrs

1. Binary verdict claim wrong ✓ Resolved 🐞 Bug ≡ Correctness

Description

The new doc says review agents make a binary decision (approve vs request changes), but the current
scaffold contract explicitly defines additional outcomes (comment-only/comment, reject, failure).
This misstates the current system and weakens the doc’s premise about “no middle ground.”

Code

docs/problems/graduated-approval-policy.md[3]

+Review agents currently make a single binary decision: approve or request changes. There is no middle ground. This forces a trade-off between two failure modes that human reviewers do not face, because humans naturally express graduated confidence.

Relevance

⭐⭐⭐ High

Team fixes inaccurate agent-review docs; PR #1782 explicitly corrected an “inaccurate claim” in
pr-review skill docs.

PR-#1782

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new doc asserts only approve/request-changes exist, but the scaffold’s review agent definition
and PR review skill explicitly define and map additional outcomes/actions (including comment-only →
comment).

docs/problems/graduated-approval-policy.md[1-4]
internal/scaffold/fullsend-repo/agents/review.md[233-246]
internal/scaffold/fullsend-repo/skills/pr-review/SKILL.md[859-865]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`docs/problems/graduated-approval-policy.md` states the review agent makes a binary decision (approve vs request changes), but the current review agent contract supports more than two outcomes (e.g., `comment-only`/`comment`, `reject`, `failure`). Update the problem statement to accurately reflect the current verdict/action model (or clarify that the *merge gating* is effectively binary even if outcomes aren’t).

### Issue Context
The scaffold defines both a human-facing “Outcome” list and a machine-facing `action` enum; both include non-binary outcomes.

### Fix Focus Areas
- docs/problems/graduated-approval-policy.md[3-4]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Autonomy spectrum misdescribed ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

The doc says the autonomy spectrum defines per-repo autonomy “based on change type,” but the
referenced autonomy-spectrum doc defines autonomy as binary per-repo with CODEOWNERS path-guarding
as the escape hatch. This contradiction makes it unclear whether autonomy is repo-readiness or
change-type classification.

Code

docs/problems/graduated-approval-policy.md[64]

+**Autonomy spectrum.** The autonomy spectrum defines per-repo autonomy based on change type. Graduated approval operates within a given autonomy level: even among changes classified at the same tier, some are higher risk than others. The autonomy spectrum determines *whether* an agent can act; graduated approval determines *how confidently* it should act.

Relevance

⭐⭐⭐ High

Repo history shows reviewers accept doc-accuracy corrections to referenced mechanisms/specs (e.g.,
GitLab event mapping fixes in PR #601).

PR-#601

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new doc’s sentence claims change-type-based per-repo autonomy, while the autonomy-spectrum doc
explicitly states a binary per-repo model with CODEOWNERS path gating, contradicting the new
wording.

docs/problems/graduated-approval-policy.md[64-65]
docs/problems/autonomy-spectrum.md[5-13]
docs/problems/autonomy-spectrum.md[15-21]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
In the “Relationship to existing mechanisms” section, the autonomy spectrum is described as per-repo autonomy based on change type, which conflicts with the autonomy-spectrum doc’s stated model (binary per-repo; CODEOWNERS for guarded paths). Adjust the sentence to match the referenced doc (repo-level autonomy/readiness + path-level gating).

### Issue Context
The autonomy-spectrum problem doc is the link target in the “Related” list and is the intended source of truth for that model.

### Fix Focus Areas
- docs/problems/graduated-approval-policy.md[64-65]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. No options in problem doc 📘 Rule violation ⌂ Architecture

Description

docs/problems/graduated-approval-policy.md is written as a pure problem statement and explicitly
defers solution architecture to a future ADR, but the compliance rule for problem docs requires at
least two solution options with clearly labeled trade-offs. Without options/trade-offs, the document
does not meet the required structure for docs/problems/ problem documents.

Code

docs/problems/graduated-approval-policy.md[R72-81]

+## Path forward
+
+Ralph Bean noted on [PR #2012](https://github.com/fullsend-ai/fullsend/pull/2012) that this area needs eval coverage before architectural proposals. The right sequence:
+
+1. **Write eval cases** where the current binary verdict gets it wrong (approves something it shouldn't, blocks something it shouldn't). These cases demonstrate the problem concretely.
+2. **Show that graduated scoring produces better outcomes** on those cases. This is TDD for agent architecture.
+3. **Propose the ADR** with those cases as evidence, and let the evals tell us if it's actually better.
+
+This document intentionally stays at the problem level. The solution architecture, threshold design, and implementation approach belong in an ADR backed by evaluation evidence.
+

Relevance

⭐ Low

Similar “problem docs must present multiple options/trade-offs” enforcement was rejected in PR #2549
(debugging problem doc).

PR-#2549

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The checklist requires problem docs to include at least two options with trade-offs. The added
document explicitly states it "intentionally stays at the problem level" and defers solution
architecture to an ADR, which results in no options/trade-offs being documented in this problem doc.

Rule 1062035: Problem docs must present multiple options with trade-offs, not a single prescribed solution
docs/problems/graduated-approval-policy.md[72-87]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Problem documents under `docs/problems/` must describe at least two distinct solution options/approaches and include clearly labeled trade-offs for each. The new problem doc currently defers solution architecture to a future ADR and does not present options.

## Issue Context
The document can remain problem-focused, but it still needs an "Options" (or equivalent) section with at least two approaches plus pros/cons (or risks/benefits) for each, and it must not mandate a single option without comparison.

## Fix Focus Areas
- docs/problems/graduated-approval-policy.md[72-87]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Benkapner]

about the no optional in in problem doc: thats intentional. this doc stays at the problem level per maintainer feedback on #2012. the solution architecture (with options and trade-offs) will come as a future ADR backed by eval evidence.

[rh-hemartin]

@Benkapner do you want me to merge it?

[ben-alkov]

@Benkapner; A (hopefully interesting) footnote for the eventual implementation: I read a paper recently, one of whose findings was that a "large" scale for evals by LLMs showed a strong tendency to improve the objective quality of classifications, e.g. a 1-5 point scale is "weaker" than a 1-10, or even 1-20 point scale. Food for thought.

[Benkapner]

@rh-hemartin yes please, go ahead and merge it. thanks!

@ben-alkov thanks for the reference, thats directly relevant to the calibration question in the doc. when we get to writing eval cases for this the scale choice will be one of the first design decisions. do you have the paper link? would be good to reference it.

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 7:40 AM UTC · Completed 8:07 AM UTC
Commit: cc71372 · View workflow run →

[fullsend-ai-retro]

Retro: PR #3063 — docs(problems): add graduated approval policy problem doc

Timeline

1. 2026-07-06 10:44 UTC — PR opened by Benkapner from a fork (Benkapner/fullsend → fullsend-ai/fullsend). Single file addition: docs/problems/graduated-approval-policy.md (+87/−0).
2. 2026-07-06 10:44 — fullsend.yaml dispatch ran (run 28785886625). Stage determination: "No stage matched — skipping dispatch." Benkapner failed is_event_actor_authorized because the PR is from a fork by an external contributor without triage+ permission on the target repo.
3. 2026-07-06 10:48 — qodo-code-review[bot] posted automated review with 3 findings: (1) binary-verdict claim was inaccurate — the scaffold contract defines additional outcomes; (2) autonomy-spectrum description was wrong — the referenced doc defines binary per-repo autonomy, not change-type-based; (3) no options section in the problem doc (intentional per maintainer feedback).
4. 2026-07-06 11:00 — Benkapner addressed findings 1 and 2, explained finding 3 was intentional.
5. 2026-07-16 08:15 — rh-hemartin approved with no inline comments.
6. 2026-07-21 07:37 — PR merged. Retro dispatched (run 29811167028).

Assessment

The workflow was clean. The fork-PR authorization gate worked as designed. Qodo filled the automated-review gap and caught real inaccuracies. Human review was lightweight but appropriate for a docs-only change. The 15-day time-to-merge reflects human pacing, not a pipeline issue.

The retro itself is of limited value: zero fullsend agents were involved in this PR's lifecycle (no review, code, or fix agent ran). All analysis effort went toward confirming that the system behaved correctly.

Evidence for existing open issues

• Skip retro dispatch for merged PRs with zero meaningful agent involvement #5399 (Skip retro dispatch for merged PRs with zero meaningful agent involvement): This PR is a direct example. No fullsend agents posted comments, reviews, or labels. The retro consumed significant compute to conclude "nothing to retrospect." A pre-dispatch or pre-script check for agent-authored activity on the PR would have saved this entire run.
• Enable review dispatch for fork PRs from trusted contributors #2967 (Enable review dispatch for fork PRs from trusted contributors): Benkapner is an active contributor whose docs changes were substantive enough to warrant fullsend review. The fork authorization gate prevented it. Qodo caught issues the fullsend review agent likely would have also flagged (cross-referencing docs against codebase, per Review agent should verify docs PRs reference features that exist in the codebase #2835 and Review agent: cross-reference external identifiers in docs against codebase #2172).
• Review agent should perform post-merge informational review when no pre-merge review occurred #4768 (Post-merge informational review when no pre-merge review occurred): This PR merged with zero fullsend review. A post-merge informational review would have provided a retrospective quality signal.
• Post user-visible feedback when /fs-review dispatch is skipped #2724 (Post user-visible feedback when dispatch is skipped): When the review dispatch was silently skipped on PR open, neither the contributor nor maintainers received any signal that automated review would not occur. Note: Post user-visible feedback when /fs-review dispatch is skipped #2724's scope is slash-command skips; the automatic dispatch-skip-on-open path is a related but distinct trigger.
• Dispatch should trigger review agent for fork PRs when ok-to-test label is applied #4374 (Trigger review agent for fork PRs when ok-to-test label is applied): An alternative path that would have enabled review for this PR if a maintainer had applied the label.

No novel proposals warranted — all findings reinforce existing improvement tracks.