review: surface why a sub-agent produced nothing, and fall back when it refused

[jwbron]

What

Two things, split out of #305 so they are not blocked by the unproven Pi harness: make a silent sub-agent failure visible, and fall back to another model when the pinned one refuses.

No model pin changes. No new dependencies. The default dispatch path is untouched.

The bug this found

correctness-reviewer is refusing outright on security-adjacent code, in production, today:

rawStopReason=refusal  tokens=2in/5207total
error=This request triggered restrictions on violative cyber content and was
      blocked under Anthropic's Usage Policy.

Reproduced on incident-auth-bypass and adversarial-injection-approve (run 30656579898, $1.57). Not a context limit — 5,207 tokens. Not a harness defect — both the Claude Agent SDK and the Pi runner hit it identically, because it is the model.

The hazard was documented and the mitigation went on the wrong roles. The README keeps the twelve specialist lenses on Opus "because Fable's cyber safety classifiers can refuse benign security-focused analysis, and a refused security lens would be a silent coverage hole". Meanwhile the 2026-07-20 A/B moved correctness-reviewer — the default roster's load-bearing recall agent — onto Fable 5 for its recall gain. The shield went on the opt-in roles; the default finder went the other way and has been there since.

A refused reviewer produces no error. It produces nothing, and the review proceeds without it.

Visibility

• An empty final is named as such (empty output: the agent returned no final text), distinct from unparseable prose, on both the eval and production contract parsers. The old message blamed the model's output shape for a request that was blocked before it produced any.
• Both runners keep the detail they were discarding: error message, provider raw stop reason, and token counts at failure. The token counts are what ruled out context overflow.
• A failed agent's raw final text is captured (truncated to 4k) and rendered in the A/B report under a collapsed heading.
• Per-agent tool-call counts on both arms — the investigation-depth signal the README already tells readers to check.

The cost of not having this: three eval runs, ~$30, chasing a contract bug that never existed. The instrumentation identified the real cause in one $1.57 scoped run.

Fallback

A refusal is deterministic in the model — the same prompt on the same pin refuses again — so the ordinary retry cannot help and the fallback must change the model. lib/refusal-fallback.ts:

Pinned model	Falls back to	Basis
claude-fable-5	claude-opus-4-8	measured (run 30656579898)
claude-opus-5	claude-opus-4-8	pre-emptive; #294 notes Opus 5 also ships elevated cyber safeguards

Three deliberate rules:

• One hop, never back to a model that already refused.
• No fallback for an unlisted pin. An unmapped model's refusal stands and is reported, so a new family's refusal profile stays visible instead of being papered over.
• Never silent. The swap is recorded per agent (fellBackTo) and lands in the report and run artifact. Trading an invisible skip for an invisible model swap would defeat the point, and the drift corpus needs to see how often this fires.

Anthropic's own refusal message points integrators at a fallback model. #294 states "gh-aw exposes no fallbacks parameter, so there's no server-side retry-on-another-model to lean on" — true of gh-aw, but scripted dispatch owns its runner, so the policy can live in our code regardless.

Known gaps, deliberately not in this PR

• The fallback is wired into the eval producer, not the production dispatcher. lib/dispatch.ts has its own retry path and still loses a refused reviewer silently. That is the more urgent half and wants its own review; this PR makes the failure visible on both paths, which is the prerequisite.
• Opus 4.8's own refusal behavior on this content is unverified. A probe is running (30658862532). If it refuses too, the fallback target is a one-line change — and a much more serious finding, since Opus 4.8 is the incumbent for all twelve lenses and claim-validator on the assumption that it is safe here.
• An empty final with a non-normal stop reason arguably belongs in the dispatch-failure path rather than the contract-parse path. Deferred: it changes production retry semantics.

Relationship to #294

#294 moves correctness-reviewer from Fable 5 to Opus 5. That may resolve this refusal or reproduce it — #294's own body flags Opus 5's "elevated cybersecurity safeguards" and stop_reason: \"refusal\". With this PR merged that becomes a $1.57 measurement on two named cases instead of a judgment call, and either way the fallback covers it.

Checks

1591 tests pass (15 new); tsc --noEmit clean. pnpm lint still cannot run in this worktree (pre-existing duplicate-eslint-plugin clash with the parent checkout, same as #294).

[changeset-bot]

🦋 Changeset detected

Latest commit: c22d9fd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
review	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

Review live A/B

No reviewable delta: review.md is byte-identical in both arms (baseline origin/jwies/aic-anthropic-discount-overlay, sha 3aa25ebe9719), so the extracted prompts and the orchestrator body match and no arms were run. Pass --force-arms for a deliberate wobble control.

[jwbron]

Production path wired, and one claim corrected

The fallback now runs on the production dispatcher too (lib/dispatch.ts), not just the eval producer. Shipping it eval-only had it backwards: the eval does not have the coverage hole, production does. A refused agent is re-dispatched on the fallback model before the contract-parse retry, and the swap is recorded as fellBackTo on the per-agent report, so it reaches the run artifact and the drift corpus rather than happening quietly.

Probe result: Opus 4.8 clean, but refusals are intermittent

Run 30658862532, same two cases, $2.83 total:

arm	correctness-reviewer	result
candidate	claude-opus-4-8	2/2 recall, no failures
baseline	claude-fable-5	2/2 recall, no failures

The candidate result is weak positive evidence for Opus 4.8 as the fallback target. The baseline result is the interesting one: the same Fable pin cleared both cases it had blocked in run 30656579898.

So refusals are intermittent, not a deterministic property of (model, case). This branch was asserting the opposite in code comments, the README, and the changeset, and that is now corrected throughout.

Note the probe is also confounded on a second axis: 30656579898 ran the Pi runner on both arms, this one ran the SDK runner on both. Two runs cannot separate model, harness, and chance. What can be said honestly is that Fable refused 3 of 4 case-arms once and 0 of 2 another time, and that Opus 4.8 has not refused in the one run where it was tried.

The fallback still stands, for a different reason than I first gave

Not "retrying the same pin always refuses again" — that is now known to be false. The reason is that the contract-parse retry cannot recover a refusal at all: it appends a corrective note about output shape, and a blocked request never produced an output to correct. That retry will keep firing uselessly no matter the refusal rate. Switching to a model with a different refusal profile is the reliable recovery; re-rolling the refusing pin is a coin flip.

Intermittency does raise the value of the fellBackTo counter: rate is now the thing to watch, and it is only measurable because the swap is recorded.

1592 tests pass (16 new); tsc --noEmit clean.

[github-actions]

Changes requested — see inline comments.

[github-actions]

issue (blocking): Production refusal fallback is dead code — the SDK runner never sets refused.

This branch gates on result.refused === true, but the production runner createSdkRunner (dispatch-runner.ts, unchanged by this PR) returns only {output, usd, turns, wallMs} (or structured: true) on every path — it never inspects stop_reason or sets refused. The only code that sets refused is the eval runner (live-runner.ts:146) and the test stubs. So in production a refusal still presents as an empty final with refused undefined: the fallback is skipped, the empty output fails the contract parse, parseWithRetry re-runs the same refusing pin, and the dimension is shed silently — the exact coverage hole this PR exists to close. dispatch.test.ts passes only because its stub fabricates refused: true. Teach createSdkRunner to detect stop_reason === "refusal" and return refused/stopReason so this path can fire.

1 additional low-confidence note

• workflows/review/lib/refusal-fallback.ts:51 — consider writing down the exit condition this fallback instruments (e.g. a per-agent fallback-rate threshold that means "repin"), so the counters section drives the pin decision it was built to inform rather than standing in for it.

[github-actions]

issue (blocking): Production refusal-fallback rate is never populated — no disk loader passes dispatchResult.

normalizeRunArtifacts derives refusalFallbacks from raw.dispatchResult here, and the README states counters reads fellBackTo from each run's out/dispatch-result.json. But neither disk loader feeds it: readRunArtifactsFromDir (counters.ts:687) builds raw from claims/validator/summary/rereviewPlan only, and loadRunDir (counters-report.ts:270) — the weekly report's actual loader — passes only {claims, validator, summary}. So raw.dispatchResult is always undefined, run.refusalFallbacks never populates, and the "Refusal fallbacks" section always renders "None" even after a fallback fires. The added RunArtifactLayout.dispatchResult field and default are therefore dead. This is also untested: the counters tests inject RunArtifacts directly, bypassing both disk readers. Read dispatch-result.json in both loaders and add a normalizeRunArtifacts test over the perAgent[].fellBackTo path.

[github-actions]

suggestion (non-blocking): Two latent issues on the fallback return path — they surface once the refusal detector from the blocking comment above lands.

1. Refused attempt's cost/turns are dropped. return dispatchAgent(name, malformedNote, fallback) discards the refused result entirely — no perAgent entry is pushed for it and its usd/turns/wallMs are not folded into the fallback entry, so totalUsd (a sum over perAgent) undercounts by the refused attempt. A refusal can land after a full investigation loop, so it is not free. The eval producer accumulates across attempts; this path does not.
2. Fallback is forgotten across the malformed-retry seam. The fallback is a recursion inside dispatchAgent, but parseWithRetry (~line 543) re-dispatches with no modelOverride, so a fallback whose output then fails the contract parse retries on definition.model — the refusing pin refusal-fallback.ts documents as never re-rolled — which refuses again and pushes a second fellBackTo entry. counters.ts flatMaps every fellBackTo, so one logical refusal is counted twice. Thread the resolved model into parseWithRetry, or move the malformed retry inside the model-aware loop.

[github-actions]

suggestion (non-blocking): Eval records fellBackTo (and swaps model) on the final attempt, when the fallback model is never dispatched.

When the refusal lands on attempt 1 (attempt 0 failed malformed, then the corrective retry refused), this branch sets report.fellBackTo = fallback and report.failed, then continues — exiting the two-iteration loop without ever invoking the runner on the fallback model. The A/B report then asserts a fallback to claude-opus-4-8 alongside a failure note, so a reader concludes the fallback model itself refused and chases a model that never ran. At attempt === 1, record the failure without swapping the model or setting fellBackTo.

[github-actions]

suggestion (non-blocking): Non-zero "Refusal fallbacks" render branch is untested.

renderCountersMarkdown's new section has a zero and a non-zero branch; the only test drives runs with no fallbacks, so just the "None" path is exercised. The per-agent listing, the count summary, and the singular/plural and runsAffected phrasing go unverified. Add a test that drives a run with a recorded fallback and asserts the rendered per-agent line.

[github-actions]

question (non-blocking): AgentResult gains stopReason/errorMessage/tokensAtFailure/toolCalls, but the production dispatcher records none of them.

The perAgent.push blocks persist only refused-derived state, never the failure detail or tool-call counts, though AgentResult now declares them (the eval producer does record them). The changeset scopes those bullets to the eval arms, but the PR summary lists "error message, provider raw stop reason, token counts at failure" under a "both the eval and production paths" heading. Was production visibility beyond the empty-final naming meant to be deferred here?

[github-actions]

suggestion (non-blocking): Failed-agent raw output is embedded in a bare fence without escaping.

The captured text is a failed final, and a common contract-parse failure mode is precisely a code-fenced reply, so the payload frequently contains a triple-backtick fence — which closes this fence early and renders the remainder as live markdown inside the job summary, breaking the <details> pairing for every section after it. Wrap the payload in a longer fence (four backticks) or indent it.

[github-actions]

nitpick (non-blocking): Composite Map key uses `` where the repo's idiom is a colon.

The package builds composite Map keys with a colon elsewhere (`${grain}:${id}` in thumbs-sweep-github.ts:583, `${corpusCase.id}:${detail.specKey}` in live-ab-report.ts:269). Agent and model names contain no colons, so : is collision-safe here and matches the prevailing pattern.