Skip to content

feat(review): add confidence annotation to review verdicts #396

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Benkapner wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat(review): add confidence annotation to review verdicts #396

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1555d73
feat(review): annotate review verdicts with a confidence level
Benkapner Aug 26, 2026
392d2a1
fix(review): scope confidence after post-script verdict overrides
Benkapner Sep 9, 2026
5c0acf2
fix(review): extract confidence rubric and close 6g/6b/6d holes
Benkapner Sep 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions agents/review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,7 @@ fields such as `outcome`, `summary`, `prior_review_sha`, or
| `reason` | string | conditional | One of: `tool-failure`, `missing-context`, `ambiguous-findings`, `token-limit`, `time-budget` |
| `label_actions` | object | no | Contextual label recommendations (see `issue-labels` skill) |
| `risk_assessment` | object | no | Risk assessment from the risk-assessment sub-agent (see `pr-risk-assessment` skill) |
| `confidence` | string | no | One of: `high`, `medium`, `low`. Verdict confidence from the `pr-review` skill step 6g. Set on any action except `failure` (the schema rejects it on `failure`). |

**Required fields per action:**

Expand All @@ -280,7 +281,10 @@ fields such as `outcome`, `summary`, `prior_review_sha`, or
| `actionable` | boolean | no | When true with a non-empty `remediation`, routes the verdict to `request-changes` so the fix agent can address the finding automatically (follow-up issue creation is temporarily disabled; see #1137) |

Schema validation failures trigger a harness retry iteration. The jq
examples below show the exact JSON shape for each action.
examples below show a valid JSON shape for each action, including the
optional `confidence` field. They are not an exhaustive field list —
omit `confidence` (or pass an empty string) when you have not determined
a band, and never set it on `failure`.

For `approve` with no actionable findings, or for `comment`:

Expand All @@ -291,8 +295,10 @@ jq -n \
--arg repo "<owner/repo>" \
--arg head_sha "<sha>" \
--arg body "<markdown review comment>" \
--arg confidence "<high|medium|low>" \
'{action: $action, pr_number: $pr_number, repo: $repo,
head_sha: $head_sha, body: $body}' \
head_sha: $head_sha, body: $body}
+ (if $confidence != "" then {confidence: $confidence} else {} end)' \
> "$FULLSEND_OUTPUT_DIR/agent-result.json"
```

Expand All @@ -306,8 +312,10 @@ jq -n \
--arg head_sha "<sha>" \
--arg body "<markdown review comment>" \
--argjson findings '<findings array>' \
--arg confidence "<high|medium|low>" \
'{action: $action, pr_number: $pr_number, repo: $repo,
head_sha: $head_sha, body: $body, findings: $findings}' \
head_sha: $head_sha, body: $body, findings: $findings}
+ (if $confidence != "" then {confidence: $confidence} else {} end)' \
> "$FULLSEND_OUTPUT_DIR/agent-result.json"
```

Expand All @@ -334,8 +342,10 @@ jq -n \
--arg head_sha "<sha>" \
--arg body "<markdown review comment>" \
--argjson label_actions '{"reason":"PR modifies API surface","actions":[{"action":"add","label":"area/api"}]}' \
--arg confidence "<high|medium|low>" \
'{action: $action, pr_number: $pr_number, repo: $repo,
head_sha: $head_sha, body: $body, label_actions: $label_actions}' \
head_sha: $head_sha, body: $body, label_actions: $label_actions}
+ (if $confidence != "" then {confidence: $confidence} else {} end)' \
> "$FULLSEND_OUTPUT_DIR/agent-result.json"
```

Expand Down
9 changes: 9 additions & 0 deletions schemas/review-result.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@
"repo": { "type": "string", "pattern": "^[^/]+/[^/]+$" },
"head_sha": { "type": "string", "pattern": "^[0-9a-fA-F]{40}$|^[0-9a-fA-F]{64}$" },
"body": { "type": "string", "minLength": 1 },
"confidence": {
"description": "Reviewer confidence in the verdict: how strongly the evidence and sub-agent agreement support this action. See skills/pr-review §6g. Never set when action is failure.",
"type": "string",
"enum": ["high", "medium", "low"]
},
"findings": {
"type": "array",
"items": { "$ref": "#/$defs/finding" },
Expand Down Expand Up @@ -82,6 +87,10 @@
"if": { "properties": { "action": { "const": "failure" } }, "required": ["action"] },
"then": { "required": ["reason"] }
},
{
"if": { "properties": { "action": { "const": "failure" } }, "required": ["action"] },
"then": { "not": { "required": ["confidence"] } }
},
{
"if": {
"properties": { "action": { "const": "approve" } },
Expand Down
96 changes: 96 additions & 0 deletions scripts/post-review-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1293,6 +1293,102 @@ run_body_test "label-actions-plus-action-hints-has-labels-section" \
run_body_test "label-actions-plus-action-hints-has-next-steps" \
"${LABEL_PLUS_HINTS_JSON}" "**Next steps:**"

# confidence present → body gets a "**Confidence:** <value>" annotation
CONFIDENCE_JSON='{"action":"comment","pr_number":99,"repo":"test-org/test-repo","head_sha":"abcdef0123456789abcdef0123456789abcdef01","body":"Some notes","confidence":"high"}'

run_body_test "confidence-present-appends-annotation" \
"${CONFIDENCE_JSON}" "**Confidence:** high"

# confidence absent → no annotation appended
NO_CONFIDENCE_JSON='{"action":"comment","pr_number":99,"repo":"test-org/test-repo","head_sha":"abcdef0123456789abcdef0123456789abcdef01","body":"Some notes"}'

run_body_count_test "confidence-absent-no-annotation" \
"${NO_CONFIDENCE_JSON}" "**Confidence:**" "0"

# Confidence after a post-script verdict override: the posted value must
# name the agent's original action, not look like confidence in `comment`.
run_body_test_with_env() {
local test_name="$1"
local json_content="$2"
local expected_body_pattern="$3"
local extra_env="$4"

local run_dir="${TMPDIR}/run-${test_name}"
mkdir -p "${run_dir}/iteration-1/output"
echo "${json_content}" > "${run_dir}/iteration-1/output/agent-result.json"
: > "${GH_LOG}"
rm -f "${TMPDIR}/last-result.json"

local exit_code=0
# shellcheck disable=SC2030,SC2031
(
cd "${run_dir}"
export PATH="${MOCK_BIN}:${PATH}"
export REVIEW_TOKEN="fake-token"
export PR_NUMBER="99"
export REPO_FULL_NAME="test-org/test-repo"
export PR_URL="https://github.com/test-org/test-repo/pull/99"
export FULLSEND_FORGE="github"
export REVIEW_FINDING_SEVERITY_THRESHOLD="low"
eval "${extra_env}"
bash "${POST_SCRIPT}"
) > "${TMPDIR}/stdout-${test_name}.log" 2>&1 || exit_code=$?

if [[ ${exit_code} -ne 0 ]]; then
echo "FAIL: ${test_name} — exit code ${exit_code}"
cat "${TMPDIR}/stdout-${test_name}.log"
FAILURES=$((FAILURES + 1))
return
fi

if [[ ! -f "${TMPDIR}/last-result.json" ]]; then
echo "FAIL: ${test_name} — no result file captured"
FAILURES=$((FAILURES + 1))
return
fi

local body
body="$(jq -r '.body' "${TMPDIR}/last-result.json")"
if ! echo "${body}" | grep -qF "${expected_body_pattern}"; then
echo "FAIL: ${test_name} — expected body pattern '${expected_body_pattern}' not found"
echo "Actual body:"
echo "${body}"
FAILURES=$((FAILURES + 1))
return
fi

echo "PASS: ${test_name}"
}

APPROVE_CONFIDENCE_JSON='{"action":"approve","pr_number":99,"repo":"test-org/test-repo","head_sha":"abcdef0123456789abcdef0123456789abcdef01","body":"Looks good to me","confidence":"high"}'

run_body_test_with_env "confidence-protected-path-scopes-original-verdict" \
"${APPROVE_CONFIDENCE_JSON}" \
"**Confidence:** high (agent verdict: approve — downgraded by protected-path check)" \
'export MOCK_PR_FILES="skills/pr-review/SKILL.md"; export REVIEW_PROTECTED_PATHS="skills/"'

FILTERED_CONFIDENCE_JSON='{"action":"request-changes","pr_number":99,"repo":"test-org/test-repo","head_sha":"abcdef0123456789abcdef0123456789abcdef01","body":"Please fix nits","confidence":"medium","findings":[{"severity":"low","category":"style","file":"a.go","description":"nit"}]}'

run_body_test_with_env "confidence-severity-filter-scopes-original-verdict" \
"${FILTERED_CONFIDENCE_JSON}" \
"**Confidence:** medium (agent verdict: request-changes — downgraded by severity filter)" \
'export REVIEW_FINDING_SEVERITY_THRESHOLD="high"; export MOCK_PR_FILES="src/main.go"'

# A leaked env var must not invent a downgrade the script did not perform.
run_body_test_with_env "confidence-stale-env-downgrade-ignored" \
"${CONFIDENCE_JSON}" \
"**Confidence:** high" \
'export CONFIDENCE_DOWNGRADE_REASON=stale-env CONFIDENCE_AGENT_ACTION=reject'

if [[ -f "${TMPDIR}/last-result.json" ]]; then
stale_body="$(jq -r '.body' "${TMPDIR}/last-result.json")"
if echo "${stale_body}" | grep -qF "downgraded by stale-env"; then
echo "FAIL: confidence-stale-env-downgrade-ignored — leaked env produced a fake downgrade"
echo "${stale_body}"
FAILURES=$((FAILURES + 1))
fi
fi

# ---------------------------------------------------------------------------
# REVIEW_PROTECTED_PATHS override tests
# Verify that setting REVIEW_PROTECTED_PATHS overrides the default list.
Expand Down
33 changes: 33 additions & 0 deletions scripts/post-review.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -545,6 +545,13 @@ severity_rank() {

threshold_rank=$(severity_rank "$REVIEW_FINDING_SEVERITY_THRESHOLD")

# Unconditional — do not use ${var:-}. A pre-set environment variable
# would otherwise survive into the confidence annotation and claim a
# downgrade that this run never performed. The severity-filter and
# protected-path blocks below are the only writers.
CONFIDENCE_AGENT_ACTION=""
CONFIDENCE_DOWNGRADE_REASON=""

if jq -e '.findings' "${RESULT_FILE}" >/dev/null 2>&1; then
original_count=$(jq '.findings | length' "${RESULT_FILE}")
FILTERED_RESULT=$(mktemp)
Expand Down Expand Up @@ -578,6 +585,8 @@ if jq -e '.findings' "${RESULT_FILE}" >/dev/null 2>&1; then
if [ "${original_action}" = "request-changes" ] || [ "${original_action}" = "reject" ]; then
echo "All findings removed by severity filter — downgrading '${original_action}' to 'comment'"
jq 'del(.findings) | .action = "comment"' "${FILTERED_RESULT}" > "${DOWNGRADE_RESULT}"
CONFIDENCE_AGENT_ACTION="${original_action}"
CONFIDENCE_DOWNGRADE_REASON="severity filter"
else
jq 'del(.findings)' "${FILTERED_RESULT}" > "${DOWNGRADE_RESULT}"
fi
Expand All @@ -590,6 +599,9 @@ fi

ACTION=$(jq -r '.action' "${RESULT_FILE}")
# ACTION retains the original value for the entire script — not re-read after protected-path downgrade.
# CONFIDENCE_AGENT_ACTION / CONFIDENCE_DOWNGRADE_REASON were cleared above
# the severity-filter block; that block and the protected-path check are
# the only writers.

# ---------------------------------------------------------------------------
# Protected-path check: the review agent must not approve PRs that touch
Expand Down Expand Up @@ -683,6 +695,8 @@ if [ "${ACTION}" = "approve" ]; then
"${RESULT_FILE}" > "${MODIFIED_RESULT}"
RESULT_FILE="${MODIFIED_RESULT}"
DOWNGRADED=true
CONFIDENCE_AGENT_ACTION="${ACTION}"
CONFIDENCE_DOWNGRADE_REASON="protected-path check"
fi
fi
fi
Expand Down Expand Up @@ -792,6 +806,25 @@ if [[ "${HAS_LABEL_ACTIONS}" == "true" ]]; then
fi
fi

# ---------------------------------------------------------------------------
# Append confidence annotation to body (skips failure, which has no body)
# ---------------------------------------------------------------------------

CONFIDENCE=$(jq -r '.confidence // empty' "${RESULT_FILE}")
if [ -n "${CONFIDENCE}" ] && [ "${ACTION}" != "failure" ]; then
if [ -n "${CONFIDENCE_DOWNGRADE_REASON}" ]; then
CONFIDENCE_NOTICE=$'\n\n---\n'"**Confidence:** ${CONFIDENCE} (agent verdict: ${CONFIDENCE_AGENT_ACTION} — downgraded by ${CONFIDENCE_DOWNGRADE_REASON})"
else
CONFIDENCE_NOTICE=$'\n\n---\n'"**Confidence:** ${CONFIDENCE}"
fi
CONFIDENCE_RESULT=$(mktemp)
CLEANUP_FILES+=("${CONFIDENCE_RESULT}")
jq --arg notice "${CONFIDENCE_NOTICE}" \
'.body = (.body + $notice)' \
"${RESULT_FILE}" > "${CONFIDENCE_RESULT}"
RESULT_FILE="${CONFIDENCE_RESULT}"
fi

# ---------------------------------------------------------------------------
# Append action-hints footer (request-changes only)
# ---------------------------------------------------------------------------
Expand Down
33 changes: 33 additions & 0 deletions scripts/post-review.src.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,13 @@ severity_rank() {

threshold_rank=$(severity_rank "$REVIEW_FINDING_SEVERITY_THRESHOLD")

# Unconditional — do not use ${var:-}. A pre-set environment variable
# would otherwise survive into the confidence annotation and claim a
# downgrade that this run never performed. The severity-filter and
# protected-path blocks below are the only writers.
CONFIDENCE_AGENT_ACTION=""
CONFIDENCE_DOWNGRADE_REASON=""

if jq -e '.findings' "${RESULT_FILE}" >/dev/null 2>&1; then
original_count=$(jq '.findings | length' "${RESULT_FILE}")
FILTERED_RESULT=$(mktemp)
Expand Down Expand Up @@ -168,6 +175,8 @@ if jq -e '.findings' "${RESULT_FILE}" >/dev/null 2>&1; then
if [ "${original_action}" = "request-changes" ] || [ "${original_action}" = "reject" ]; then
echo "All findings removed by severity filter — downgrading '${original_action}' to 'comment'"
jq 'del(.findings) | .action = "comment"' "${FILTERED_RESULT}" > "${DOWNGRADE_RESULT}"
CONFIDENCE_AGENT_ACTION="${original_action}"
CONFIDENCE_DOWNGRADE_REASON="severity filter"
else
jq 'del(.findings)' "${FILTERED_RESULT}" > "${DOWNGRADE_RESULT}"
fi
Expand All @@ -180,6 +189,9 @@ fi

ACTION=$(jq -r '.action' "${RESULT_FILE}")
# ACTION retains the original value for the entire script — not re-read after protected-path downgrade.
# CONFIDENCE_AGENT_ACTION / CONFIDENCE_DOWNGRADE_REASON were cleared above
# the severity-filter block; that block and the protected-path check are
# the only writers.

# ---------------------------------------------------------------------------
# Protected-path check: the review agent must not approve PRs that touch
Expand Down Expand Up @@ -273,6 +285,8 @@ if [ "${ACTION}" = "approve" ]; then
"${RESULT_FILE}" > "${MODIFIED_RESULT}"
RESULT_FILE="${MODIFIED_RESULT}"
DOWNGRADED=true
CONFIDENCE_AGENT_ACTION="${ACTION}"
CONFIDENCE_DOWNGRADE_REASON="protected-path check"
fi
fi
fi
Expand Down Expand Up @@ -382,6 +396,25 @@ if [[ "${HAS_LABEL_ACTIONS}" == "true" ]]; then
fi
fi

# ---------------------------------------------------------------------------
# Append confidence annotation to body (skips failure, which has no body)
# ---------------------------------------------------------------------------

CONFIDENCE=$(jq -r '.confidence // empty' "${RESULT_FILE}")

@waynesun09 waynesun09 Aug 26, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Confidence annotation is posted against a verdict the post-script already overrode

Verified at head 261f903. The new block reads .confidence and appends **Confidence:** <value> to the body, but by that point post-review.src.sh may already have replaced the agent's verdict, and the confidence value is never revisited.

Two confirmed override paths run BEFORE the confidence block:

  1. Severity filter: when filtering removes every finding, jq 'del(.findings) | .action = "comment"' rewrites request-changes/reject to comment. This runs before ACTION is read, so ACTION is already the rewritten value.
  2. Protected-path check: jq '.action = "comment" | .body = (.body + $notice)' rewrites approve to comment and appends a "human reviewer must approve" notice, setting DOWNGRADED=true. The script explicitly documents that the ACTION shell variable retains the original value here ("ACTION retains the original value for the entire script — not re-read after protected-path downgrade").

In both paths the confidence the agent computed for the original verdict is rendered verbatim under the new one. A protected-path downgrade will routinely post comment + **Confidence:** high — a combination the new rubric says is essentially unreachable (SKILL.md §6g caps comment-only at medium unless a narrow corroboration test passes). Per §6g confidence is a property of the action ("how strongly the evidence and sub-agent agreement support this action"), so after a downgrade the posted value describes an action that no longer exists.

This is not cosmetic: the PR's stated purpose is to emit this datum for downstream graduated-approval work, and the value is wrong precisely on the protected-path and all-findings-filtered paths — the paths where a human (and any future automation) most needs an accurate signal. Note that skills/pr-review/ is itself a protected path, so this scenario fires on this repo's own reviews of PRs like this one. Neither of the two added tests in post-review-test.sh covers a downgrade combined with confidence.

Suggestion: Re-read the action after the downgrade paths, or gate the annotation on a downgrade flag, and either drop confidence from the body when the post-script changed the verdict or scope it to the agent's original verdict, e.g. **Confidence:** high (agent verdict: approve — downgraded by protected-path check). Note that a flag-based fix must cover BOTH paths: DOWNGRADED exists only for the protected-path branch, while the severity-filter branch keeps original_action local to its own block. Add post-review-test.sh cases for approve+confidence+protected path and for request-changes+confidence fully filtered, asserting the resulting annotation.

if [ -n "${CONFIDENCE}" ] && [ "${ACTION}" != "failure" ]; then
if [ -n "${CONFIDENCE_DOWNGRADE_REASON}" ]; then
CONFIDENCE_NOTICE=$'\n\n---\n'"**Confidence:** ${CONFIDENCE} (agent verdict: ${CONFIDENCE_AGENT_ACTION} — downgraded by ${CONFIDENCE_DOWNGRADE_REASON})"
else
CONFIDENCE_NOTICE=$'\n\n---\n'"**Confidence:** ${CONFIDENCE}"
fi
CONFIDENCE_RESULT=$(mktemp)
CLEANUP_FILES+=("${CONFIDENCE_RESULT}")
jq --arg notice "${CONFIDENCE_NOTICE}" \
'.body = (.body + $notice)' \
"${RESULT_FILE}" > "${CONFIDENCE_RESULT}"
RESULT_FILE="${CONFIDENCE_RESULT}"
fi

# ---------------------------------------------------------------------------
# Append action-hints footer (request-changes only)
# ---------------------------------------------------------------------------
Expand Down
11 changes: 7 additions & 4 deletions skills/pr-review/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -897,6 +897,7 @@ When merging
- Combine descriptions if they add complementary detail
- Keep the more specific remediation
- Preserve `actionable: true` if either finding had it
- Attach merged_from on every 6b merge (confidence.md).

#### 6c. Preserve distinct-category findings

Expand Down Expand Up @@ -933,7 +934,7 @@ budget section), skip the challenger: keep the merged finding set from

**Part 3 — Context package:** the merged finding set from steps
6a–6c (as a JSON array), plus the full PR diff and changed files
list. Format as:
list. Strip/restore merged_from via confidence.md. Format as:

```markdown
## Context
Expand Down Expand Up @@ -984,9 +985,7 @@ budget section), skip the challenger: keep the merged finding set from
part of the standard finding schema.

@waynesun09 waynesun09 Sep 17, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] challenger_action is stripped in 6d before the 6g rubric reads challenger kept/downgraded/unchanged state

SKILL.md step 6d item 3 (lines 982-985) strips challenger_action and challenger_reason from every adjudicated finding before merging into the review set, saying they are "logged for transparency but are not part of the standard finding schema". The new rubric in confidence.md then depends on exactly that state at 6g: Low fires when "the verdict rests on a finding the challenger downgraded" (line 69), High requires findings "confirmed by the challenger" (91-92), the comment-only ceiling requires the driving finding "survived the challenger unchanged" (112-113), and the reject ceiling requires the objection "explicitly confirmed by the challenger" (118).

The PR gives merged_from an explicit internal-carrier treatment (confidence.md:13-15: keep it through 6c-6f, strip only before writing agent-result.json) but gives the challenger disposition no equivalent, so by 6g the orchestrator has no field to read and must rely on recall of the challenger output. The 6d copy-back rules in confidence.md:31-40 also key on kept/downgraded/merged values with no stated ordering relative to the strip in step 3 bullet 1.

No existing thread on this PR covers this: the synthesis notes thread at SKILL.md:885 was the undefined carrier for severity gaps, and the thread at SKILL.md:1002 was the merged_from copy-back.

Suggestion: Retain challenger_action (or an internal equivalent such as challenger_disposition: kept|downgraded|merged) on each finding through 6e-6g, mirroring the merged_from treatment, and strip it together with merged_from immediately before agent-result.json is written. State in confidence.md that the 6d copy-back runs before the strip.

- If `adjudicated_findings` is empty but the set sent to the
challenger was non-empty, treat this as a challenger failure (fall back
per the immediate next step below). A legitimate challenger pass
that removes all findings is unlikely — an empty result more likely
indicates a parsing error or context truncation.
per the immediate next step below).
- Otherwise, replace the challenged subset with the challenger's
`adjudicated_findings` (then re-append anything withheld).
- Log any `removed_findings` for transparency but do not include
Expand Down Expand Up @@ -1205,6 +1204,10 @@ require action, because `comment` (COMMENTED review state) does not
block the PR. When the summary language and the verdict action
contradict each other, escalate the verdict to match the language.

#### 6g. Confidence

Follow confidence.md.

### 7. Produce the review result

Compose the review comment using this structure:
Expand Down
Loading
Loading