Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/prose-judge-enforce.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"review": minor
---

A prose judge now runs inside every sub-agent's `submit_result` path, the way plain-prose runs it: a pinned haiku judges each finding's prose against the vendored plain-prose rubric (verbatim from the reference implementation, plus a per-label loose length tier), and a failing finding is rejected back to its own AUTHOR, who rewrites it in-session with its repo context intact and resubmits. No fresh-context rewrite exists (an earlier shape's rewriter confabulated a flag name on the live fixtures, which is why the author does it). Enforcement is structural rather than eval-gated: bounces are capped at 2 per agent and then the submission posts as-is; a judge error or unparseable verdict accepts immediately; a Stop hook redirects an agent ending without calling the tool back to it (capped at 2), so the free-text fallback path shrinks instead of bypassing the gate; and the run records four states per finding (skipped/pass/fail/error) in `judge-prose-verdicts.json` plus the dispatch result's `proseJudge` block, so a broken judge reads as errors, not clean prose. The judge call goes through the Agent SDK (the sandbox strips `ANTHROPIC_API_KEY` and proxy-injects auth, so a raw fetch would fail-open the judge into a silent no-op). The sub-agent `discussion` field contracts carry the same shape line (at most one claim, one line of evidence, at most one question), so findings are born short and the judge stays a backstop. Calibrated on the three Khan/webapp#41609 comments ("still as poetic as before"); `eval/judge-prose-live.ts` pins that the named complaint fails the judge.
142 changes: 142 additions & 0 deletions workflows/review/eval/judge-prose-live.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
/**
* Live calibration for the prose judge (lib/judge-prose.ts) over the
* Khan/webapp#41609 fixtures: real model, real prompts, the verdicts the
* done-when requires. The unit tests pin the bounce mechanics with a
* stubbed runner; THIS is where the model's judgment is asserted:
*
* - the SPEC.md comment (3823429680, "Graduation removes the last runtime
* lever ... cheap insurance") MUST fail — it is the named complaint the
* judge exists for, and a rubric change that stops flagging it should
* break this script, not ship;
* - the other two fixtures print their verdicts unpinned (both are dense
* 100+-word non-blocking comments, so a rule 3 fail is plausible and
* fine; a pass is not a defect);
* - every failing fixture also prints its bounce message, because in
* production that text is the entire style spec the authoring agent
* sees at rewrite time (there is no rewrite call to eyeball anymore;
* the author rewrites in-session).
*
* Eval-side, so a raw fetch is correct here (the sandbox-proxy constraint in
* judge-prose-runner.ts is a production concern; this runs on a dev machine
* with `ANTHROPIC_API_KEY`). Same API shape as judge-live-model.ts.
*
* Run: ANTHROPIC_API_KEY=... npx tsx workflows/review/eval/judge-prose-live.ts
*/

import {
buildBounceMessage,
buildJudgePrompt,
parseJudgeVerdict,
PINNED_PROSE_JUDGE_MODEL,
} from "../lib/judge-prose";
import {CLEAN_CONTROLS, FIXTURES_41609} from "../lib/judge-prose-fixtures";

const API_URL = "https://api.anthropic.com/v1/messages";

const callModel = async (prompt: string): Promise<string> => {
const response = await fetch(API_URL, {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought (non-blocking): The calibration script pins the judge on a call path production never takes. judge-prose-live.ts calls the Messages API directly while judge-prose-runner.ts deliberately routes production through the Agent SDK for the proxy-auth reason its header explains; the two paths share only buildJudgePrompt. That is the established eval pattern here (judge-live-model.ts does the same), but this is the first judge whose verdict is enforced, and the PR's case for skipping a log-only trial rests on exactly this script's evidence.

review details found by holistic

method: "POST",
headers: {
"x-api-key": process.env["ANTHROPIC_API_KEY"] ?? "",
"anthropic-version": "2023-06-01",
"content-type": "application/json",
},
body: JSON.stringify({
model: PINNED_PROSE_JUDGE_MODEL,
max_tokens: 1024,
messages: [{role: "user", content: prompt}],
}),
});
if (!response.ok) {
throw new Error(`API ${response.status}: ${await response.text()}`);
}
const payload = (await response.json()) as {
content?: {type?: string; text?: string}[];
};
return (payload.content ?? [])
.filter((block) => block.type === "text")
.map((block) => block.text ?? "")
.join("");
};

const main = async (): Promise<void> => {
if ((process.env["ANTHROPIC_API_KEY"] ?? "") === "") {
throw new Error("ANTHROPIC_API_KEY is required");
}
let pinnedFailMissed = false;
for (const fixture of FIXTURES_41609) {
const reply = await callModel(
buildJudgePrompt(fixture.discussion, fixture.label),
);
const verdict = parseJudgeVerdict(reply);
// eslint-disable-next-line no-console
Comment thread
khan-actions-bot marked this conversation as resolved.
Resolved
console.log(
`\n=== ${fixture.commentId} (${fixture.label}, expected ${fixture.expected})`,
);
if (verdict === null) {
// eslint-disable-next-line no-console
console.log(`unparseable reply: ${reply}`);
if (fixture.expected === "fail") {
pinnedFailMissed = true;
}
continue;
}
// eslint-disable-next-line no-console
console.log(
`verdict: ${verdict.pass ? "pass" : "FAIL"}${verdict.problems
.map((problem) => `\n - ${problem}`)
.join("")}`,
);
if (fixture.expected === "fail" && verdict.pass) {
pinnedFailMissed = true;
}
if (!verdict.pass) {
// eslint-disable-next-line no-console
console.log(
`bounce message the author would see:\n${buildBounceMessage([
{
key: `comment-${fixture.commentId}`,
problems: verdict.problems,
},
])}`,
);
}
}
// The clean controls: prose that should pass, printed as a canary for
// over-flagging (a warning, never an exit code; a style judgment call
// should not block a merge).
for (const control of CLEAN_CONTROLS) {
const verdict = parseJudgeVerdict(
await callModel(
buildJudgePrompt(control.discussion, control.label),
),
);
// eslint-disable-next-line no-console
console.log(`\n=== clean control (${control.label})`);
if (verdict === null) {
// eslint-disable-next-line no-console
console.log("unparseable reply");
} else if (verdict.pass) {
// eslint-disable-next-line no-console
console.log("verdict: pass (as expected)");
} else {
// eslint-disable-next-line no-console
console.log(
`WARNING, over-flagging canary tripped: FAIL${verdict.problems
.map((problem) => `\n - ${problem}`)
.join("")}`,
);
}
}
if (pinnedFailMissed) {
Comment thread
khan-actions-bot marked this conversation as resolved.
Resolved
throw new Error(
"the pinned fixture (3823429680, the named 41609 complaint) did not fail the judge",
);
}
};

main().catch((error) => {
// eslint-disable-next-line no-console
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});
76 changes: 76 additions & 0 deletions workflows/review/lib/dispatch-agents.ts
Original file line number Diff line number Diff line change
Expand Up @@ -64,3 +64,79 @@ export const loadAgents = (
}
return agents;
};

/** One sub-agent dispatch request (mirrors the eval's LiveAgentRequest). */
export type AgentRequest = {
name: string;
model: string;
prompt: string;
cwd: string;
maxTurns: number;
timeoutMs: number;
/**
* The structured-final contract check (trial suggestion h). When set,
* the runner exposes a `submit_result` tool whose input is validated by
* this function BEFORE it is accepted: null accepts the payload as the
* agent's result; a string rejects it back to the model, which corrects
* and re-calls in the same session (a few turns, not the $2-3 full
* re-dispatch the malformed-output retry costs). Free-text finals stay as
* the fallback for a model that never calls the tool.
*/
validate?: (payload: Record<string, unknown>) => string | null;
/**
* The prose style gate (judge-prose.ts), awaited by the runner AFTER
* `validate` accepts: `null` accepts the payload; a string rejects it
* back to the authoring model exactly like a contract rejection, so the
* AUTHOR rewrites its own prose in-session (the plain-prose loop). The
* gate caps its own bounces and fails open; it can slow a submission,
* never lose one.
*/
judgeProse?: (payload: Record<string, unknown>) => Promise<string | null>;
};

export type AgentResult = {
/** The agent's final text (expected to be its JSON contract). */
output: string;
usd: number;
turns: number;
wallMs: number;
/**
* Tool calls the agent made. The harness-parity signal: a loop that
* investigates with fewer tool calls and scores lower has a toolbox
* problem, not a model problem. Optional because a runner that cannot
* count them reports nothing rather than a misleading zero.
*/
toolCalls?: number;
/**
* The provider's stop reason for the agent's last assistant message, when
* the runner can see one. Load-bearing for one specific diagnosis: an
* EMPTY final on cyber-adjacent input is the signature of a refusal, which
* #294 documents as surfacing "as a missing agent result, not an error".
* Without this the empty result is indistinguishable from a dropped one.
*/
stopReason?: string;
/**
* Why the call failed, when the runner can see it. `stopReason=error`
* alone does not distinguish an overloaded provider from a prompt that
* outgrew the context window, and those have opposite fixes (retries vs
* compaction). `tokensAtFailure` is the discriminator: near the model's
* context window means overflow.
*/
errorMessage?: string;
/** The provider's own stop reason, before the runner normalizes it. */
rawStopReason?: string;
/** Input and total tokens on the last assistant message. */
tokensAtFailure?: {input: number; total: number};
/**
* The provider blocked the request under its usage policy. Distinct from
* every other failure because it is deterministic in the model, not
* transient: retrying the same pin returns the same refusal, so the only
* useful response is a different model.
*/
refused?: boolean;
/** The output came through the structured-final tool, pre-validated. */
structured?: boolean;
};

/** The model seam; the SDK-backed production runner lives in the CLI entry. */
export type AgentRunner = (request: AgentRequest) => Promise<AgentResult>;
116 changes: 116 additions & 0 deletions workflows/review/lib/dispatch-runner.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -192,3 +192,119 @@ describe("createSdkRunner sub-agent environment", () => {
expect(env["PATH"]).toBe(process.env["PATH"]);
});
});

type HookCallback = (
input?: unknown,
toolUseID?: unknown,
extra?: unknown,
) => Promise<Record<string, unknown>>;

/** The Stop hook the runner registered on the last query, if any. */
const stopHook = (): HookCallback | undefined => {
const hooks = lastOptions["hooks"] as
| Record<string, {hooks: HookCallback[]}[]>
| undefined;
return hooks?.["Stop"]?.[0]?.hooks[0];
};

describe("createSdkRunner prose gate (PRA-45)", () => {
it("bounces a style rejection to the author AFTER the contract accepts", async () => {
const judged: string[] = [];
session = async function* (tools) {
const bounced = await tools[0].handler(
{result: {findings: [{id: "f1"}]}},
undefined,
);
expect(bounced.isError).toBe(true);
expect(bounced.content[0].text).toBe("Result rejected: style");
// The author rewrites and re-calls; second submission passes.
const accepted = await tools[0].handler(
{result: {findings: [{id: "f1-rewritten"}]}},
undefined,
);
expect(accepted.isError).toBeUndefined();
yield success("trailing prose");
};
const result = await (
await createSdkRunner()
)(
request({
judgeProse: (payload) => {
judged.push(JSON.stringify(payload));
return Promise.resolve(
judged.length === 1 ? "Result rejected: style" : null,
);
},
}),
);
expect(result.structured).toBe(true);
expect(JSON.parse(result.output)).toEqual({
findings: [{id: "f1-rewritten"}],
});
expect(judged).toHaveLength(2);
});

it("never judges a payload the contract already rejected", async () => {
let judgeCalls = 0;
session = async function* (tools) {
const rejected = await tools[0].handler(
{result: {finding: "drifted"}},
undefined,
);
expect(rejected.isError).toBe(true);
yield success('{"findings": []}');
};
await (
await createSdkRunner()
)(
request({
judgeProse: () => {
judgeCalls += 1;
return Promise.resolve(null);
},
}),
);
expect(judgeCalls).toBe(0);
});
});

describe("createSdkRunner Stop hook (the free-text fallback funnel)", () => {
it("blocks a stop without a submission, twice, then lets the fallback proceed", async () => {
session = async function* () {
yield success("free text");
};
await (
await createSdkRunner()
)(request());
const hook = stopHook();
expect(hook).toBeDefined();
const first = await hook!();
expect(first).toMatchObject({decision: "block"});
expect(String(first["reason"])).toContain("submit_result");
expect(await hook!()).toMatchObject({decision: "block"});
// Cap spent: the third stop goes through (the genuine fallback).
expect(await hook!()).toEqual({});
});

it("lets a stop through once a payload was accepted", async () => {
session = async function* (tools) {
await tools[0].handler({result: {findings: []}}, undefined);
yield success("done");
};
await (
await createSdkRunner()
)(request());
const hook = stopHook();
expect(await hook!()).toEqual({});
});

it("registers no Stop hook without a validate contract (no tool to point at)", async () => {
session = async function* () {
yield success("free text");
};
await (
await createSdkRunner()
)(request({validate: undefined}));
expect(stopHook()).toBeUndefined();
});
});
Loading
Loading