fix(#1037): add re-review anchoring and coherence check to risk assessment

[fullsend-ai-coder]

Summary

The risk assessment sub-agent produced inconsistent scores across consecutive re-reviews on the same PR when the underlying risk signals hadn't changed. Both rationales described identical low-risk characteristics, but the score drifted (1 → 2) because: (1) the sub-agent evaluated each run independently with no anchoring to prior scores, and (2) no coherence check validated that the rationale actually justified the assigned score.

This PR adds three changes:

• Orchestrator (step 3c-2): Fetch the prior risk assessment sticky comment on re-reviews and pass the prior score/level/rationale to the sub-agent as context for anchoring.
• Sub-agent definition: Add re-review anchoring rules that preserve the prior score when Tier 1 signals are unchanged, and require the rationale to explain any score delta. Add a score-rationale coherence check before returning.
• Scoring model (procedure step 8): Add a coherence verification step that cross-checks the rationale against the scoring guidance tables and anchoring examples, flagging cases where described risk characteristics map to a different score than assigned.

Testing

• risk-tier1-test.sh — all 42 tests pass (Tier 1 script unchanged)
• validate-output-schema-test.sh — all schema validation tests pass (risk_assessment schema unchanged)
• post-review-test.sh — all post-processing tests pass (risk label/comment logic unchanged)
• Changes are prompt/instruction-only (markdown files). No script or schema modifications.

---

Closes #1037

Post-script verification

• Branch is not main/master (agent/1037-risk-score-anchoring)
• Secret scan passed (gitleaks — b7ef57f8a1a18326cfdcab9056187f1607b60c59..HEAD)
• PR body secret scan passed (gitleaks — no-git)

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 8:11 AM UTC · Completed 8:32 AM UTC

Commit: 5f14f73 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.27

[fullsend-ai-review]

Risk Assessment: moderate (2/5)

Details

Small prompt-only change (3 markdown files, 100 lines) by a bot author with clear issue alignment, offset by high protected-path count (skills/) and elevated churn/regression history on SKILL.md; overall risk is moderate.

Previous run

Risk Assessment: moderate (2/5)

Details

Small, well-scoped docs/config-only PR (3 files, 116 lines) by a known bot author with no security-sensitive, CI, or dependency changes; elevated protected path count (3) is the primary Tier 1 risk driver, and high churn on skills/pr-review/SKILL.md (20 commits in 30d, 10 distinct authors) elevates Tier 2, offset by low Tier 3 issue risk (well-scoped bug fix, no unresolved discussions). Re-review anchoring: signals unchanged from prior assessment — score preserved at 2.

Previous run (2)

Risk Assessment: moderate (2/5)

Details

Small, well-scoped docs/config-only PR (3 files, 108 lines) by a known bot author with no security-sensitive, CI, or dependency changes; elevated protected path count (3) is the primary Tier 1 risk driver, and high churn on skills/pr-review/SKILL.md elevates Tier 2, offset by low Tier 3 issue risk.

Previous run (3)

Risk Assessment: moderate (2/5)

Details

Small, well-scoped docs/config-only PR by a known bot author that adds anchoring and coherence check mechanisms to the risk assessment sub-agent; elevated git history signals (high churn and multi-author contention on these skill files) are offset by low metadata and issue risk, yielding moderate overall risk.

[fullsend-ai-review]

Review

Findings

Medium

• [protected-path] skills/ — This PR modifies 3 files under protected paths (skills/pr-review/SKILL.md, skills/pr-review/sub-agents/risk-assessment.md, skills/pr-risk-assessment/SKILL.md). The PR links to Risk assessment sub-agent produces inconsistent scores across re-reviews with identical rationale #1037 and explains the rationale, but human approval is always required for protected-path changes.

Low

• [edge-case] skills/pr-review/SKILL.md — The gh api --paginate command in step 3 applies --jq independently per page. With [.[] | select(...)] | last // empty, the last selector operates within each page rather than across all pages. In practice this works because the sticky comment marker guarantees at most one matching comment, but the intent is fragile for future readers or modifications. Consider adding --slurp for cross-page aggregation or a comment noting the single-comment assumption.

• [error-handling-gap] skills/pr-review/SKILL.md — Step 3 does not specify what happens if the gh api call itself fails (network error, rate limit, auth failure). While gh api writes errors to stderr (so the variable would typically be empty on failure), an explicit exit-code check would be more robust against future behavior changes.

• [internal-consistency] skills/pr-review/sub-agents/risk-assessment.md — Steps 4a and 4b run anchoring then coherence sequentially, but no priority ordering is defined for conflicts. If anchoring (rule 2) preserves a prior score because Tier 1 signals are unchanged, the coherence check could flag the preserved score as not matching current signals. A reasonable reading gives 4b final say, but this ambiguity is worth noting.

• [section-redundancy] skills/pr-review/sub-agents/risk-assessment.md — The standalone ## Score-rationale coherence section is a single delegation sentence identical to procedure step 4b. Consider removing it since step 4b already provides the same instruction.

Previous run

Review

Findings

Medium

• [protected-path] skills/ — This PR modifies 3 files under protected paths (skills/pr-review/SKILL.md, skills/pr-review/sub-agents/risk-assessment.md, skills/pr-risk-assessment/SKILL.md). The PR links to Risk assessment sub-agent produces inconsistent scores across re-reviews with identical rationale #1037 and explains the rationale, but human approval is always required for protected-path changes.

Low

• [cross-file-section-naming] skills/pr-review/sub-agents/risk-assessment.md — The "Score-rationale coherence" section is duplicated in substance across two files: the sub-agent definition and the linked skill (skills/pr-risk-assessment/SKILL.md, step 8). Both define the same verification procedure with overlapping but not identical wording. Since the spawned agent receives both Part 1 (sub-agent def) and Part 2 (linked skill), it will see two descriptions of the same check. Consider deferring to the linked skill's definition (e.g., "Apply the score-rationale coherence check defined in the linked skill's step 8") rather than restating it, matching the pattern where Procedure step 4 already defers to the linked skill for the scoring model.

Previous run (2)

Review

Findings

Medium

• [conflicting instructions] skills/pr-review/sub-agents/risk-assessment.md — The score-rationale coherence threshold in the sub-agent definition conflicts with skills/pr-risk-assessment/SKILL.md. The sub-agent doc says to reconcile when described characteristics map to "a different score" (any difference), while SKILL.md step 8 says to reconcile only when "off by ≥2." Both documents are composed into the sub-agent's prompt (Parts 1 and 2), creating ambiguous guidance. A 1-point drift (the exact scenario in Risk assessment sub-agent produces inconsistent scores across re-reviews with identical rationale #1037) would trigger reconciliation under the sub-agent's rule but not under SKILL.md's threshold.
  Remediation: Align the thresholds — either add "off by ≥2" to risk-assessment.md's coherence section, or remove the ≥2 qualifier from SKILL.md step 8.

• [protected-path] skills/ — This PR modifies 3 files under protected paths (skills/pr-review/SKILL.md, skills/pr-review/sub-agents/risk-assessment.md, skills/pr-risk-assessment/SKILL.md). The PR links to Risk assessment sub-agent produces inconsistent scores across re-reviews with identical rationale #1037 and explains the rationale, but human approval is always required for protected-path changes.

Low

• [missing procedure integration] skills/pr-review/sub-agents/risk-assessment.md — The sub-agent's Procedure (steps 1–5) does not reference the new "Re-review anchoring" or "Score-rationale coherence" sections. While "Score-rationale coherence" self-sequences ("Before returning the final JSON"), "Re-review anchoring" lacks an explicit sequencing cue relative to the numbered procedure.
  Remediation: Insert a reference between steps 4 and 5, e.g., "4a. If prior risk data is provided, apply the Re-review anchoring rules below. 4b. Verify score-rationale coherence before returning."

Previous run (3)

Review

Findings

High

• [cross-reference error] skills/pr-review/SKILL.md — In the risk assessment pre-pass procedure (step 3c-2), step 5 references “composed from parts 1–4” but only 3 parts are defined in step 4: Part 1 (Sub-agent definition), Part 2 (Linked skill), and Part 3 (Context). The prior risk assessment is added as a subsection within Part 3, not as a separate Part 4. This dangling reference could cause the orchestrating agent to fabricate a non-existent Part 4.
  Remediation: Change “composed from parts 1–4” to “composed from parts 1–3”.

Medium

• [format mismatch] skills/pr-review/SKILL.md — The comment format documented in step 3 for parsing the prior risk assessment does not match the format produced by post-review.sh. The documented format shows <details><summary>Details</summary><rationale></details> inline, while the actual format uses blank lines between elements. An agent following the documented format would fail to correctly extract the rationale.
  Remediation: Update the documented format to match the actual output with blank lines between elements.

• [protected-path] skills/ — This PR modifies 3 files under protected paths (skills/pr-review/SKILL.md, skills/pr-review/sub-agents/risk-assessment.md, skills/pr-risk-assessment/SKILL.md). The PR links to Risk assessment sub-agent produces inconsistent scores across re-reviews with identical rationale #1037 and explains the rationale, but human approval is always required for protected-path changes.

Low

• [edge-case] skills/pr-review/SKILL.md — The gh api command in step 3 uses per_page=100 without --paginate. For PRs with 100+ comments, the risk assessment sticky comment may be missed. The codebase consistently uses --paginate for the same endpoint elsewhere (e.g., pre-review.sh).
  Remediation: Use gh api --paginate consistent with existing patterns.

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

[fullsend-ai-coder]

🤖 Finished Fix · ✅ Success · Started 8:34 AM UTC · Completed 8:39 AM UTC

Commit: 5f14f73 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.30

[fullsend-ai-coder]

🔧 Fix agent — iteration 1 (bot-triggered)

Addressed 3 of 4 review findings with code fixes. The protected-path finding is informational (human approval required) and needs no code change. Fixed: dangling parts 1–4 cross-reference, comment format mismatch with post-review.sh output, and missing --paginate on gh api call.

Fixed (3):

1. cross-reference error: step 5 references parts 1–4 but only 3 parts exist (skills/pr-review/SKILL.md): Changed 'composed from parts 1–4' to 'composed from parts 1–3' since the prior risk assessment is a subsection of Part 3 (Context), not a separate Part 4.
2. format mismatch: documented comment format does not match post-review.sh output (skills/pr-review/SKILL.md): Updated the documented risk assessment comment format to match the actual output produced by post-review.sh, adding blank lines between elements and using multi-line details/summary tags.
3. edge-case: gh api uses per_page=100 without --paginate (skills/pr-review/SKILL.md): Replaced 'gh api' with 'gh api --paginate' and removed the per_page=100 query parameter, consistent with existing patterns in pre-review.sh and other scripts that query the same endpoint.

Disagreed (1):

1. protected-path: PR modifies files under skills/: This is an informational observation, not an actionable code finding. The reviewer correctly notes that human approval is required for protected-path changes — no code fix is needed.

Tests: passed

Next steps:

• /fs-review — request a re-review of the changes
• /fs-fix <your instruction> — run another fix pass with specific guidance
• Push commits directly — review re-runs automatically on push
  _{Updated by fullsend fix agent}

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 8:41 AM UTC · Completed 9:02 AM UTC

Commit: 1204795 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $4.80

Superseded by updated review

[maruiz93]

/fs-fix fix the review's from the reviewer, reply on the opened threads that are addressed and resolve them

[fullsend-ai-coder]

🤖 Finished Fix · ✅ Success · Started 9:07 AM UTC · Completed 9:12 AM UTC

Commit: 1204795 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.32

[fullsend-ai-coder]

🔧 Fix agent — iteration 2 (human-triggered)

Addressed both review findings: (1) aligned the coherence reconciliation threshold across SKILL.md and risk-assessment.md by removing the >=2 qualifier so any score-rationale mismatch triggers reconciliation, and (2) integrated the new Re-review anchoring and Score-rationale coherence sections into the sub-agent's numbered Procedure as steps 5 and 6.

Fixed (2):

1. conflicting coherence thresholds between sub-agent and SKILL.md (skills/pr-risk-assessment/SKILL.md): Removed 'off by >=2' qualifier from SKILL.md step 8 coherence check to align with sub-agent's 'any difference' threshold. Both documents are composed into the sub-agent prompt; the stricter threshold prevents the 1-point drift scenario from Risk assessment sub-agent produces inconsistent scores across re-reviews with identical rationale #1037.
2. missing procedure integration for Re-review anchoring and Score-rationale coherence (skills/pr-review/sub-agents/risk-assessment.md): Added explicit steps 5 and 6 to the sub-agent Procedure referencing the Re-review anchoring and Score-rationale coherence sections, with step 7 now being the return step. This gives the new sections clear sequencing within the numbered procedure.

Tests: passed

Next steps:

• /fs-review — request a re-review of the changes
• /fs-fix <your instruction> — run another fix pass with specific guidance
• Push commits directly — review re-runs automatically on push
  _{Updated by fullsend fix agent}

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 9:14 AM UTC · Completed 9:34 AM UTC

Commit: ee73b30 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.40

[maruiz93]

/fs-fix

In skills/pr-review/sub-agents/risk-assessment.md, the "Score-rationale coherence" section restates the verification procedure that is already defined in skills/pr-risk-assessment/SKILL.md step 8. Since the spawned agent receives both Part 1 (sub-agent def) and Part 2 (linked skill), it sees two overlapping descriptions of the same check.

Replace the restated coherence procedure in the sub-agent definition with a deferral to the linked skill's step 8, matching the pattern where Procedure step 4 already defers to the linked skill for the scoring model. For example: "Apply the score-rationale coherence check defined in the linked skill's step 8."

Also integrate references to "Re-review anchoring" and "Score-rationale coherence" into the numbered Procedure steps (between steps 4 and 5), so the sequencing is explicit. For example: "4a. If prior risk data is provided, apply the Re-review anchoring rules below. 4b. Verify score-rationale coherence before returning."

[fullsend-ai-coder]

🤖 Finished Fix · ✅ Success · Started 9:51 AM UTC · Completed 9:56 AM UTC

Commit: ee73b30 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.41

[fullsend-ai-coder]

🔧 Fix agent — iteration 3 (human-triggered)

Addressed both parts of the human instruction: (1) replaced the restated coherence procedure in the sub-agent definition with a deferral to the linked skill's step 8, and (2) integrated Re-review anchoring and Score-rationale coherence as sub-steps 4a/4b in the Procedure, making sequencing explicit.

Fixed (2):

1. Score-rationale coherence section restates linked skill step 8 (skills/pr-review/sub-agents/risk-assessment.md): Replaced the restated coherence verification procedure (14 lines) with a two-line deferral to the linked skill's step 8, eliminating the redundancy the spawned agent sees when receiving both Part 1 and Part 2.
2. Re-review anchoring and coherence check not integrated into numbered Procedure steps (skills/pr-review/sub-agents/risk-assessment.md): Restructured Procedure steps 5 (anchoring), 6 (coherence), and 7 (return) into sub-steps 4a and 4b under step 4, with step 5 as the return step. This makes the sequencing between score computation and verification explicit, matching the pattern where step 4 already defers to the linked skill for the scoring model.

Tests: passed

Next steps:

• /fs-review — request a re-review of the changes
• /fs-fix <your instruction> — run another fix pass with specific guidance
• Push commits directly — review re-runs automatically on push
  _{Updated by fullsend fix agent}

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 9:58 AM UTC · Completed 10:19 AM UTC

Commit: aa5487a · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.85

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 12:11 PM UTC · Completed 12:23 PM UTC

Commit: aa5487a · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $4.64

[fullsend-ai-retro]

PR #1038 added re-review anchoring and coherence checking to the risk assessment sub-agent, fixing the score drift reported in issue #1037. The code agent produced the initial PR in ~10 minutes ($3.56), but the PR required 4 review iterations and 3 fix iterations before merging, with two human interventions providing architectural guidance. Total agent cost: ~$25.

What went well:

• The triage agent correctly identified the root cause (missing anchoring mechanism, no coherence check) and recommended a clear two-part fix.
• The code agent's research phase was thorough — it traced the full risk assessment pipeline across 10+ files before implementing.
• The review agent caught real issues at each iteration: cross-reference error (high), conflicting thresholds (medium), section redundancy (low). No false positives in actionable findings.
• The auto-triggered fix agent resolved straightforward findings efficiently (3 fixes in 5 minutes on iteration 1).

Primary rework driver: The code agent added the same coherence-check procedure to both the sub-agent definition (risk-assessment.md, Part 1 of the composed prompt) and the linked skill (SKILL.md, Part 2), with different thresholds (any difference vs off by >=2). The review agent correctly diagnosed this but prescribed "align the thresholds" rather than "eliminate duplication via deferral" — the pattern already used in step 4 of the same file. The fix agent followed the review's guidance and aligned values but left the structural redundancy. The human had to provide the architectural fix (deferral pattern with step 4a/4b restructuring), costing 2 extra review/fix iterations (~$12).

Autonomy note: The human reviewer added clear value identifying the deferral-vs-restatement architectural pattern that the review agent missed. For prompt-engineering changes to agent definitions, human review remains essential for design-level decisions.

Existing issues with new supporting evidence:

• #865 (cross-check enumerated lists): The code agent wrote "parts 1–4" when only 3 parts existed, caught as a high finding in review 1.
• #961 (verify documented behavior claims against source): The code agent documented a comment format that didn't match the actual post-review.sh output despite having read the script during research.
• #694 (avoid duplicating canonical source content): Related to Proposal 1 below, though Code agent should avoid duplicating canonical source content in generated documentation #694 targets documentation generally while this is specifically about prompt-composition boundaries.

Proposals filed

• Code agent should detect prompt-composition boundaries and use deferral for shared procedures (in fullsend-ai/agents)
• Review agent should prescribe consolidation over alignment for redundant content in composed-prompt files (in fullsend-ai/agents)