Skip to content

feat(review): add confidence annotation to review verdicts (#5514) #5515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Benkapner wants to merge 1 commit into from
Closed

feat(review): add confidence annotation to review verdicts (#5514) #5515

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions internal/scaffold/fullsend-repo/schemas/review-result.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@
},
"label_actions": {
"$ref": "#/$defs/label_actions"
},
"confidence": {
"type": "string",
"enum": ["high", "medium", "low"],
"description": "How confident the review agent is in its verdict. Derived from sub-agent agreement, challenger removals, and verdict threshold proximity."
}
Comment on lines +30 to 34

@qodo-code-review qodo-code-review Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Informational

2. Confidence schema untested 🐞 Bug ☼ Reliability

No schema-validation tests currently assert that confidence is accepted for valid enum values or
rejected for invalid ones, so regressions to the new contract may go uncaught. Adding targeted
fixtures would harden the rollout of this new field.
Agent Prompt 
### Issue description
`confidence` was added to `review-result.schema.json`, but CI fixtures don’t currently validate acceptance/rejection behavior for this new enum field.

### Issue Context
Existing tests validate several review-result shapes but do not include any cases with `confidence`, so they don’t exercise the new property.

### Fix Focus Areas
- internal/scaffold/fullsend-repo/scripts/validate-output-schema-test.sh[345-371]
- internal/scaffold/fullsend-repo/schemas/review-result-label-actions-test.sh[40-100]
- internal/scaffold/fullsend-repo/schemas/review-result.schema.json[18-35]

### Suggested changes
- Add a passing fixture that includes `"confidence":"high"` (or medium/low) on an approve/reject payload.
- Add a failing fixture with an invalid value like `"confidence":"unknown"` to ensure enum enforcement is covered.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

@Benkapner Benkapner Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the field is optional in the schema, so omitting it is valid and won't break existing tests. adding test fixtures for this is reasonable as a follow-up once the field is actually in use.

@Benkapner Benkapner Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the pr-review skill is what drives agent behavior, not agents/review.md. the skill now has the confidence derivation instructions. agents/review.md is a high-level doc for humans, not an instruction set for the agent.

},
"allOf": [
Expand Down
26 changes: 26 additions & 0 deletions internal/scaffold/fullsend-repo/skills/pr-review/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -777,6 +777,32 @@ challenger-adjudicated finding set and evaluate:
Use `reject` only when no amount of code-level iteration will make
the PR mergeable.

#### 6g. Determine confidence level

After the verdict is set, derive a confidence level based on the signals
available from the sub-agent and challenger passes. Confidence does not
change the verdict or any routing behavior. It is an informational
annotation for humans reviewing the agent's output.

**Confidence levels:**

- **high:** all sub-agents agreed on severity for every finding, the
challenger removed zero or one findings, and the verdict was not close
to a threshold boundary (e.g., no medium findings when the verdict is
approve)
- **medium:** the challenger removed more than one finding (initial
disagreement that was resolved), OR sub-agents flagged the same code
with different severities, OR the verdict is one finding away from
flipping (e.g., one medium finding on an approve verdict)
- **low:** significant sub-agent disagreement (same code flagged at
severities two or more levels apart), OR the challenger failed and
the pre-challenger set was used, OR the change scope is ambiguous
(e.g., large refactoring PR where correctness is hard to verify)

Include the confidence level in the review result JSON as a top-level
`confidence` field. This field is optional in the schema; omitting it
is acceptable if you cannot determine it.

### 7. Produce the review result

Compose the review comment using this structure:
Expand Down
Loading