feat(review): let a risk score of 1 narrow the dispatch behind an opt-in flag - #1246
Draft
guyoron1 wants to merge 2 commits into
Draft
feat(review): let a risk score of 1 narrow the dispatch behind an opt-in flag#1246guyoron1 wants to merge 2 commits into
guyoron1 wants to merge 2 commits into
Conversation
Functional tests did not runFunctional tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the |
…grade instead of drop, keep history ADR 0089 leaves tier 1 to a bash script but has the sub-agent re-emit it, which is how the same PR scored 1, 2, 1 across three re-reviews (fullsend-ai#1037). risk-tier1.sh now ends with TIER1_SCORE and RISK_FLOOR — the SKILL.md table computed once, deterministically — and the sub-agent copies them into tier1_score / risk_floor instead of re-deriving. RISK_FLOOR is 2 whenever a security-sensitive path is touched. post-review.sh enforces it from the echoed field, so risk/low cannot be applied to a security PR whatever the LLM returned. Over the 246 production PRs measured on fullsend#4698, no score-1 PR touched such a path, so the floor changes nothing today and closes the gap for later. When the sub-agent fails, the orchestrator no longer drops the score: it runs the script itself and emits max(round(TIER1_SCORE), RISK_FLOOR) with degraded: "tier1-only". Four production reviews since 08-25 lost their score to "claude-sonnet-4-5@20250929 is not available" (the error behind fullsend#6922, visible from 08-25); consumers must treat degraded as no score. The sticky risk comment now carries the tier 1 value, the degraded marker, and a per-head-SHA history table carried forward from the prior comment (GitHub only, 20 rows, rows re-admitted only when they match the exact shape this script writes), so drift across re-reviews is visible on the PR instead of only in run artifacts. Tests: risk-tier1-test.sh covers score_tier1, _score_size, has_source_files, risk_floor and both e2e fixtures; post-review-test.sh covers floor raises/never lowers, degraded header, history row, garbage provenance dropped, and legacy results unchanged. Baseline ceilings for the two SKILL.md files bumped for the added prose. Refs fullsend-ai#1037, fullsend-ai/fullsend#4698 Signed-off-by: guy oron <goron@redhat.com>
…-in flag ADR 0089 / fullsend-ai#861 compute the composite risk score and leave acting on it for later. This is the smallest later: REVIEW_RISK_ROUTING_ENABLED (default "false"). When on, risk-assessment runs first instead of in the step-4 batch, and a composite score of exactly 1 keeps only correctness and security, dropping intent-coherence, docs-currency, style-conventions and cross-repo-contracts. No model changes anywhere: since fullsend#7116 models are owned by the consuming repo's .fullsend/config.yaml, and the orchestrator's own model is never touched. A missing or unparseable score, or one marked degraded ("tier1-only", the orchestrator's fallback when the sub-agent failed), fails open and leaves the full review in place. Routing keys on the composite rather than on tier 1 alone because, over the 246 production PRs that carry both a risk comment and a review, composite == 1 had 0/53 with a major or critical finding while a tier-1-only gate would have narrowed 6/104 that did. The pre-pass latency is the price of that. Bumps the pr-review context-budget ceiling for the added text. Refs fullsend-ai/fullsend#4698 Signed-off-by: guy oron <goron@redhat.com>
guyoron1
force-pushed
the
feat/review-risk-routing
branch
from
September 10, 2026 13:44
c45af41 to
e385ea3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1245 (its commit is the first on this branch; review only the top commit).
ADR 0089 / #861 compute the composite risk score and leave acting on it for later. This is the smallest later:
REVIEW_RISK_ROUTING_ENABLED(default"false"). When on,risk-assessmentruns first instead of in the step-4 batch, and a score of exactly 1 with nodegradedmarker narrows the dispatch tocorrectnessandsecurity(intent-coherence, docs-currency, style-conventions, cross-repo-contracts are skipped; security-triage and challenger untouched). No model is changed anywhere — models are owned by the consuming repo's.fullsend/config.yamlsince fullsend-ai/fullsend#7116 — and the orchestrator's own model is never touched. Any other score, a missing or unparseable score, ordegradedpresent → the full selection (fail open).Routing keys on the composite rather than tier 1 because, over the 246 production PRs that carry both a risk comment and a review (fullsend-ai/fullsend#4698), composite == 1 had 0/53 with a major or critical finding while a tier-1-only gate would have narrowed 6/104 that did. Draft until Marta has looked at the numbers; the paired cheaper-model comparison is still to run.
Refs fullsend-ai/fullsend#4698.