review: dispatch-conformance gate at the submit chokepoint (deterministic orchestrator, slice 0)

[jwbron]

Plan-of-record: The deterministic orchestrator (slice 0; it absorbs round three's step 5, promoted 07-21 from cost bet to integrity fix). This PR is that first slice: a deterministic dispatch-conformance gate at the submit chokepoint, so a review verdict cannot be submitted unless the sub-agent outputs it is supposed to summarize actually exist.

The motivating evidence (two production transcripts)

The violation: Khan/webapp#40992, run 29865480728 (2026-07-21, the v1.7.0 acceptance trial). Verified from the run artifacts, not from memory:

• The agent transcript contains zero Task (sub-agent) calls; its only tool use is Bash, GitHub reads, file Reads, and the safe-output calls themselves.
• out/ contains a single self-authored orchestrator-findings.json whose process field reads "streamlined direct review (3-file, 151-line self-contained Go change)". No router ran, no routing.json, no rereview-plan.json, no correctness-reviewer.json, no claim validation.
• The queued safe outputs: three inline comments (one carrying issue (blocking)) and a REQUEST_CHANGES whose entire body is Changes requested — see inline comments. No skipped-dimension notes, no fingerprint stamp, no disclosure of any kind.
• Every job in the run is green. Nothing on the PR surface distinguishes this run from a conforming one.

The contrast: #272, run 29763410312 (2026-07-20). Same workflow version family, small PR, Low-tier budget: the transcript dispatches pattern-triage, correctness-reviewer, skill-auditor, and the five enabled whole-change reviewers; out/ carries one <agent>.json per dispatch plus rereview-plan.json; and the one gap (no validator output) is disclosed in the submitted body, verbatim: Note: claim validation not assessed this run (claim-validator output unavailable).

So the failure is stochastic non-conformance of unknown frequency. Its consequences: model changes to sub-agents (the v1.7.0 Fable correctness-reviewer swap) silently do not run on such reviews, the claim-validator precision gate is bypassed, and a blocking verdict reaches the PR with none of the machinery the A/Bs price. Prompt rules cannot fix it: an orchestrator that skips dispatch also skips prompt-mandated self-checks. Code is the only lever (same family as v1.6.1's non-empty-body rule, #262).

The enforcement point, settled empirically

The design fork was whether submit_pull_request_review executes inside the agent job (detect-after-the-fact fallback) or in a separate code job (blockable). From the lock file and the real runs' artifacts:

• gh-aw queues safe outputs during the agent run; the "Ingest agent output" step finalizes them as /tmp/gh-aw/agent_output.json, which uploads inside the agent artifact; the separate safe_outputs job downloads that artifact and safe_output_handler_manager.cjs makes the actual GitHub API calls.
• gh-aw v0.81.6 has no step-injection point inside the safe_outputs job, but it does support post-steps: in the agent job, and (verified by compiling this exact frontmatter) they land after "Ingest agent output" and before "Upload agent artifacts", i.e. after the queue is final and before it ships. Failing the agent job alone would NOT block anything: safe_outputs runs whenever the agent job is not skipped, which is why the gate rewrites the queue rather than merely failing.

So the gate (lib/dispatch-gate.ts, invoked from the new post-steps: in review.md) runs on the agent runner with both the final queue and the real /tmp/gh-aw/review/ staging in view. On violation it:

1. strips every posting/mutating item from the queue (default-deny; only upload_artifact and the non-posting diagnostics survive, so the evidence artifact still lands),
2. preserves the original queue at agent_output.pre-gate.json and writes dispatch-gate.json beside it (both ride the agent artifact; the report doubles as the violation-rate instrument the plan doc's Q3 asks for), and
3. exits non-zero: the agent job goes red and gh-aw's conclusion job files its failure issue.

The submission is blocked, not detected: the safe_outputs job executes the rewritten queue and has nothing to post. if: always() on the step because safe_outputs also executes queues from partially-failed agent jobs.

The rules (per re-review depth, from rereview-plan.json; missing plan defaults to full)

1. Verdict queued at full/scoped/flip-gated: out/correctness-reviewer.json must exist (Step 3 requires even a failed dispatch to stage an error note, and the shed ranking never sheds a default). One waiver, proven by staged output: pattern-triage returned an empty reviewFiles. An existing-but-unparseable file needs the output unavailable disclosure note. fast dispatches no finding producers, so no correctness requirement there.
2. Inline comments queued (any depth): a parseable out/claim-validator.json, or the disclosed skipped-dimension note (both review: budget and shed tuning from the 07-10 production sheds #258 wordings match: planned shed under hard-ceiling pressure, or output unavailable).
3. Planned vs dispatched (full/scoped): every name in routing.json's enabledReviewers/lensesToSpawn with no out/<name>.json needs its not assessed this run note in the submitted body. Dispatched-below-planned with no disclosure is a violation.

Deliberately not enforced, to keep the false-positive rate at zero: thread-reconciler and skill-auditor existence (the conforming #272 run dispatched no reconciler; it had no prior threads), pattern-triage itself, and "the router ran" as an independent rule (a routerless run is already caught by rule 1). Fail-open only for the gate's own bugs (loud ::warning, review unblocked); a detected violation never passes silently.

Verification

• 24 deterministic vitest cases: both production shapes reproduced from the downloaded artifacts, each depth mode, missing-output, unparseable-output, shed-disclosed, shed-undisclosed, the triage waiver, and the CLI's strip/preserve/report behavior. pnpm test --run workflows/review (878 tests), pnpm typecheck, and eslint on the changed files are green.
• Synthetic end-to-end repro (definition of done): a fabricated /tmp/gh-aw tree with a queued REQUEST_CHANGES + blocking comment and an out/ missing correctness-reviewer.json exits 1 with correctness-missing and validator-missing-with-findings, rewrites the queue down to the upload_artifact item, and writes both forensics files; the same tree with conforming staging exits 0 with the queue byte-identical.
• The edited review.md frontmatter compiles clean under gh aw compile (gh-aw v0.81.6), with the gate step landing exactly between the staging upload and the agent-artifacts upload.
• The eval suite cannot cover this class and this PR does not pretend otherwise: the harness dispatches sub-agents from a script, so protocol fidelity is precisely what it never exercises. The vitest suite is the whole coverage story; a smoke A/B would measure nothing here.

What remains prompt-trusted after this gate

• Authenticity. The gate proves the reviewer outputs were staged, not that dispatched models produced them; an orchestrator could in principle fabricate out/ files (the prompt body deliberately does not mention the gate, to avoid teaching that move). Script-driven dispatch (slice 2) closes this structurally.
• Content fidelity. That posted comments actually come from validated claims, that plausible downgrades were applied, that the verdict matches the labels: still prompt-executed until slice 3's decision chokepoints.
• Roster completeness beyond the rules above: skill-auditor/thread-reconciler dispatch, and honest rereview-plan.json/routing.json staging (the plan/routing files the gate reads are themselves CLI-written, but the orchestrator invoking those CLIs is prompt-trusted until slice 1 moves staging into pre-workflow steps).

Nothing is enabled consumer-side; the gate ships with the next release like any other workflows/review change (the installed copy and lock bump at release time, per the #276 flow). Rollout note for consumers when it lands: a violated run flips from a green run with a normal-looking review to a red run that posts nothing, which is the intended loud failure mode; the gate report in the agent artifact says exactly why, and gives the violation rate for free (plan doc Q3).

[changeset-bot]

🦋 Changeset detected

Latest commit: db17815

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
review	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[jwbron]

Scoping note for the next slice (staging becomes pre-workflow steps; round-three step 5 / orchestrator doc slice 1), recorded here instead of expanding this PR:

What moves. The staging work the orchestrator currently performs by invoking CLIs mid-conversation becomes deterministic steps that run before the agent starts, so it wakes with files on disk: the prior-reviews fetch, the rereview-mode CLI (plan + scoped.diff), the scoped-diff swap and annotated-sibling refresh, and the provenance and new-scope staging that share the pattern. None of it needs model output. #246's prompt wiring merged as acknowledged interim; this slice deletes it wholesale. The eval producer runs the same pre-steps so the A/B keeps measuring production (behavior-neutral by construction, smoke A/B only).

Mechanism, informed by this PR's investigation. pre-agent-steps: already exists and is where these land (this PR's gate proved the sibling post-steps: injection point end to end). Two things the pre-steps need that the mid-run CLI invocations get implicitly: GitHub data (the prior-reviews fetch needs pull-requests: read on the agent job, available) and the Step 1 staging inputs (full.diff, files.json, pr-context.json) which today the orchestrator itself writes from get_files; the slice therefore has to move that Step 1 staging into code first, which is the real work item and the reason this was not bolted onto the gate PR.

What this slice buys the gate. Today the gate trusts the orchestrator to have invoked the router and rereview-mode CLIs honestly (routing.json and rereview-plan.json are its rule inputs, and a missing plan just defaults to full). Once staging is pre-agent, those inputs exist before the model runs and the gate's remaining trust in them drops to zero; the gate itself needs no change.

Inherited ledger this migration owns (deferred to it from the 07-13 bot-feedback pass): #244's post-submit accountability-splice check and executed-vs-decided resolve accounting, #245's code-side shape enforcement for out-of-lane handoffs and cross-source same-line dedup, #246's flip-gate code chokepoint, #247's extraction-verify strengthening (disciplines extraction becomes a pre-step, its verify becomes code). Rider on the same track, eval-gated separately with a powered run because it is recall-affecting: the callers.json staging step (enumerate call sites of diff-touched exported symbols for the removed-behavior audit; the webapp #40793 miss).

[github-actions]

Changes requested — see inline comments.

[github-actions]

Review live A/B

Baseline: origin/main (review.md ea89dc34d0eb); candidate: working tree (review.md 4a4b3f4f93ad).

Ruler: matcher deterministic+arbiter; corpus cc11988c0918 (9 cases).

Metric	Baseline	Candidate	Delta
Must-catch recall	100%	100%	+0%
Verdict agreement	89%	89%	+0%
Noise (unmatched posted)	53%	59%	+6%
Clean false flags	0	0
Judge mean quality	0.88	0.82	-0.06
Cost	$9.24	$9.76
Wall clock	1146s	1217s
Cases run / skipped	9 / 0	9 / 0
Misses found-but-dropped	0	0
Findings anchor-snapped	0	0

Adversarial hard gate: PASSED on the candidate arm.

Single-run-stable rows: recall, verdict agreement, regressions, adversarial gate. Judge quality and noise are not: they jitter run-to-run at this corpus size, and a regressed reviewer can score HIGHER on judge quality (fewer, surer comments each read better). Recall against the labeled specs is the load-bearing metric.

Measured noise floor (identical arms, run 29069228968, 2026-07-10, 6 arm-samples, full corpus x3, pre-arbiter; budget skips left the samples on unequal case sets, so these v1 bands also carry case-mix variance): must-catch recall 54%-86% (sd 10%), verdict agreement 75%-100% (sd 9%), noise (unmatched posted) 50%-60% (sd 3%), judge mean quality 82%-86% (sd 2%). A single-run delta whose arms both sit inside a band is indistinguishable from run-to-run wobble; use --repeats to resolve smaller effects.

[jwbron]

Stack guide (bottom-up, one slice per PR, each stacked on the previous): #279 slice 0 (this PR, the dispatch-conformance gate) → #280 slice 1 (staging becomes a pre-agent step) → #282 slice 2 (script-driven dispatch behind the ROUTING dispatch dial) → #283 slice 3 (verdict/flip/resolution chokepoints in the gate; code-owned disciplines extraction) → #284 slice 4 (the submission plan; Steps 4-6 as code in scripted mode). Slices 0/1/3 change every run deterministically and ride vitest; slices 2/4 are opt-in per repo (nothing flips by default) and are live-trial-gated per the review-trial protocol before any consumer sets dispatch scripted. Plan-of-record: https://claude.ai/code/artifact/f988965d-1697-4e68-a9f2-d46a0915d225

[github-actions]

Changes requested — see inline comments.

All 6 prior review threads are resolved.

[github-actions]

Approved — no blocking issues found.

All 6 prior review threads are resolved.

[github-actions]

nitpick (non-blocking): This gate could not run (infra failure; review not blocked) warning also fires when the gate did block. If the sentinel write throws but the queue rewrite succeeds (the degrade-to-detect path), runDispatchGateCli still exits 1 with no sentinel file, so both clauses are false — the gate ran and the queue was stripped, so nothing posts, yet the operator's only log line says the review was not blocked. Consider distinguishing the case (e.g. also test for dispatch-gate.json) or rewording to gate exited non-zero without a violation sentinel.

Lower-confidence observations (4)

• workflows/review/lib/agent-json.ts:45 — the 200-span cap in balancedSpans counts balanced-but-unparseable spans, so a final message quoting a large brace-rich code block before an unfenced trailing JSON payload could exhaust the cap before reaching the payload; consider capping failed parse attempts, or scanning from the end since the payload is terminal.
• workflows/review/lib/agent-json.ts:116 — fenced blocks are tried (last-first) before balanced spans, so an earlier fenced example ahead of the real bare payload wins; via triageEmptiedReview a quoted empty-reviewFiles example could waive rule 1.
• workflows/review/lib/rereview-mode.ts:385 — the cache carrier hard-rejects any verdict except APPROVE/REQUEST_CHANGES, while the body-stamp regex accepts any token; a comment-only arm recording COMMENT could never anchor via the only working carrier and would re-plan full forever.
• workflows/review/lib/rereview-mode.ts:42 — since no HTML-comment stamp has ever survived gh-aw sanitization, a sanitizer-surviving encoding could restore the durable body carrier rather than a stripped format plus an evictable cache fallback (eviction degrades safely to full, so this is cost, not correctness). If visible body noise rules that out, worth noting in the header.

[github-actions]

suggestion (non-blocking): Step 9 lists stampHunks, wasDraft, and reviewedHunks as pinned (backticked) cache keys, but the verdict is only described in prose (The verdict and whether a risks/patterns comment was posted this run) with no key name or allowed values. stampFromCacheMemory requires exactly verdict: "APPROVE" | "REQUEST_CHANGES", so a run that records the verdict under a different key — or as a COMMENT event — makes the cache fallback return null and every re-review re-plans full, the silent inertness this changeset exists to prevent. Consider pinning it like its neighbors:

Suggested change

	plan CLI wrote it in Step 3). This, with `verdict` and `wasDraft`, is the divergence
	- `verdict`: the submitted review event, recorded as exactly `APPROVE` or `REQUEST_CHANGES`

[github-actions]

suggestion (non-blocking): On a block, KEEP_ITEM_TYPES strips the verdict, every inline comment, and the risks comment, so the author gets a red run with zero review signal even though the findings were only unvalidated, not shown to be wrong. Did you weigh rewriting the submission to a COMMENT-event review (with a conspicuous non-conformance banner and blocking labels demoted) — preserving the findings while removing verdict authority and still going red? The module doc states the suppression trade but not this alternative; if total suppression is deliberate, a line naming that choice would help the next reader.

[github-actions]

Review Guidance

github-actions (4 files)

File	Reason
dispatch-gate.ts	The deterministic enforcement layer that rewrites the safe-output queue at the submit chokepoint; a bug here changes what is blocked or posted on every review run.
agent-json.ts	Defines what the gate counts as an existing, parseable sub-agent output, so its leniency rules set the gate's false-negative and false-positive rates.
rereview-mode.ts	The cache-memory fallback decides re-review depth on every push; every rejection path degrades toward a full review, so residual risk is cost, not coverage.
review.md	Adds a new post-steps execution surface to the shared reviewer's frontmatter that invokes the gate on the agent runner; fail-open by design.

Excluded from review (3 files)

Not individually reviewed — generated, formatting-only, or fully explained by a common pattern above:

• .changeset/review-dispatch-gate.md — formatting-only
• .changeset/stamp-carrier-cache-memory.md — formatting-only
• workflows/review/README.md — formatting-only

[github-actions]

3 of 3 prior review threads are still unaddressed as of 1d9d558:

3 non-blocking threads still open

• suggestion (non-blocking) workflows/review/lib/dispatch-gate.ts:409: On a block, KEEP_ITEM_TYPES strips the verdict, every inline comment, and the risks comment, so the author gets a red...
• suggestion (non-blocking) workflows/review/review.md:1659: Step 9 lists stampHunks, wasDraft, and reviewedHunks as pinned (backticked) cache keys, but the verdict is only de...
• nitpick (non-blocking) workflows/review/review.md:271: This gate could not run (infra failure; review not blocked) warning also fires when the gate did block. If the senti...

[github-actions]

note (non-blocking): Step 9's wasDraft (the run's draft status) is mapped onto anchorDraft in stampFromCacheMemory, but the body-stamp regime it replaces carried plan.stampAnchorDraft — the prior anchor's draft flag at fast/flip-gated depth, not the current run's. The mismatch only ever makes the reconstructed anchor stricter (an extra full review via the ready-for-review-anchor guard), so it's cost, not lost coverage — but it contradicts the comment's claim that the cache fields carry the same information as the stamp. Consider copying stampAnchorDraft verbatim in Step 9 and preferring it over wasDraft, falling back to wasDraft for older records.

Lower-confidence observation (1)

• workflows/review/lib/dispatch-gate.ts:194 — disclosesSkippedDimension requires the phrase and dimension alias to co-occur on one physical line; a model-authored disclosure note hard-wrapped mid-phrase would false-block a conforming, disclosed run. Matching within Note:-delimited segments (or a ~200-char window) would tolerate wrapping.

[github-actions]

nitpick (non-blocking): This changeset body opens with a review: prefix, unlike every other changeset in the repo — its sibling .changeset/review-dispatch-gate.md opens "The dispatch-conformance gate: ...". The prefix duplicates the "review" scope already declared in the frontmatter and renders straight into the published CHANGELOG. Consider dropping it and capitalizing: The re-review fingerprint anchors on cache memory; the body stamp never survives gh-aw ingest.

[github-actions]

Approved — no blocking issues found.

5 of 5 prior review threads are still unaddressed as of db17815:

5 non-blocking threads still open

• nitpick (non-blocking) .changeset/stamp-carrier-cache-memory.md:5: This changeset body opens with a review: prefix, unlike every other changeset in the repo — its sibling `.changeset/re...
• suggestion (non-blocking) workflows/review/lib/dispatch-gate.ts:409: On a block, KEEP_ITEM_TYPES strips the verdict, every inline comment, and the risks comment, so the author gets a red...
• note (non-blocking) workflows/review/lib/rereview-mode.ts:426: Step 9's wasDraft (the run's draft status) is mapped onto anchorDraft in stampFromCacheMemory, but the body-stamp...
• suggestion (non-blocking) workflows/review/review.md:1659: Step 9 lists stampHunks, wasDraft, and reviewedHunks as pinned (backticked) cache keys, but the verdict is only de...
• nitpick (non-blocking) workflows/review/review.md:271: This gate could not run (infra failure; review not blocked) warning also fires when the gate did block. If the senti...