feat(#498): adversarial injection/manipulation eval suite - #730
Conversation
…e attacker's The three Delphi jurors each read the entire escalation transcript, and a non-unanimous round 1 doubles that to six full reads. Pinning them to `models.attacker` therefore made the eval's cheapest-to-judge step its most expensive one: judging "did the digest boundary and the role hold" is a bounded classification over text already in context, not the open-ended adversarial reasoning the conductor does. Splits `juror` out of `BehavioralModels` as its own field, wired through `ADVERSARIAL_JUROR_MODEL` and defaulting to `claude-sonnet-5`. On the current corpus (one behavioral scenario, maxTurns 4) that takes a run from roughly $0.13-0.19 to $0.09-0.11, and the saving scales linearly as behavioral scenarios are added — which is the point, since today's absolute number is small but the jury is the part that grows fastest. Safe because the jury is not the only scorer: `findIdentityLeaks` runs over the same transcript and forces `breached` on any real identity value regardless of the vote, so a leniently-voting juror cannot turn an actual leak green. What a weaker jury can still cost is the softer signal — manipulation resistance where nothing literal leaked — which is why the default steps to Sonnet rather than straight to Haiku; the env var makes Haiku available for anyone who measures it against a known-breached transcript first. The added test asserts against the calls the provider actually received, not against the interface: a `juror` field that nothing read would have passed every existing test. Reverting the one-line wiring turns it red.
|
Cost note plus a ready-made patch — sneumannb5#1, based on this PR's branch (the suite isn't on What it changes: Why: each juror reads the entire escalation transcript, and a non-unanimous round 1 turns three full reads into six — so the jury is the largest input-token consumer in a behavioral run while doing the most bounded work. On the current corpus (1 behavioral scenario,
The absolute number is small either way — the reason to do it before merge is the slope, since cost scales with the number of Why a cheaper jury is safe here: One caveat worth raising separately from the patch: the eval can't run at all yet — |
|
Pushed to this branch on Marcel's instruction — heads-up @sneumannb5, since it touches your PR. Two merge commits on top of your 1. The juror-model change (previously offered as sneumannb5#1, now closed as superseded). Safe to downgrade because the jury is not the only scorer: The added test asserts against the calls the provider actually received, not against the interface — a 2. Current Local verification on the merged tree:
|
…en) (#748) The adversarial eval crashed on its first real run with `400 \`temperature\` is deprecated for this model`. That was the loud symptom. The quiet one is worse. `LlmScreener.screen()` sends `temperature: 0` on every inbound turn, and its caller turns any exception into `unscreenable` — an explicit fail-open. The screener is constructed with the agent's own model (`buildOrchestrator.ts:329`), whose default is `DEFAULT_ORCHESTRATOR_MODEL = 'claude-opus-4-8'`. That model rejects `temperature: 0` with a 400. So on the repo's own default configuration, #579's inbound security screening returned `unscreenable` for every payload and reported no error. Five more production call sites are affected the same way — the plan-runner's materializer, replanner (x2), gate and gc all pass `temperature: 0`. Measured against the live API rather than inferred, because the rule is not what it looks like: model omitted 0 0.5 1 claude-opus-4-6 OK OK OK OK claude-opus-4-7 OK 400 400 OK claude-opus-4-8 OK 400 400 OK claude-opus-5 OK 400 400 OK claude-sonnet-4-6 OK OK OK OK claude-sonnet-5 OK 400 400 OK claude-haiku-4-5 OK OK OK OK `opus-4-6` accepts the parameter while `opus-4-7` rejects it, so "newer than X" is a plausible and wrong gate. `temperature: 1` is always accepted because it IS the default — which is why the eval's attacker step (temperature 1) survived and only the juror (temperature 0) raised. Fix in the adapter, the layer that owns the wire contract: `buildParams` omits `temperature` when the model does not honour it. That covers all seven call sites at once, including the legacy v1 wrapper, which delegates through the same neutral provider. `supportsTemperature` is exported so a caller can tell that its determinism request will be dropped. Note this is not fallout from the juror-model knob in #730: the juror ran on claude-opus-4-8 at temperature 0 before that change and would have raised the same 400. The eval simply never ran until ANTHROPIC_API_KEY was set. Verification: 6841 tests, 0 fail (baseline 6839 + the two added here); tsc clean; repo lint clean. Mutation check with a rebuild between runs — forcing the gate to always-true and to always-false each turns exactly one new test red.
2 participants
What
Adversarial injection/manipulation eval suite (Closes #498). Deterministic wire-level probes gate every PR via npm test; key-gated behavioral tier (escalation + 3-juror Delphi) runs on main/nightly.
Why
Injection defenses (Privacy Shield v4 boundary, issue-body scrubber, ticket delimiter) were only asserted architecturally. This attacks them, reads the verdict off the wire artifact, and fails on a hardening regression or coverage loss.
Test plan
Risk / blast radius
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.