Repair tool rollout contracts - #2046
Conversation
78cefb3 to
bebf821
Compare
|
Final-head behavioral eval summary (five hosted trials per case; aggregate only):
The initial spill run exposed an eval defect: the model could add a filter that prevented spilling, while the headless harness denied the modified command. The final harness now requires the exact producer command, verifies its structured arguments, and grants that synthetic producer only inside the isolated eval configuration. Deterministic replay/evidence checks: 190 passed, 2 platform-specific skips, 0 failed. Full solution validation on the final product code: 7,749 passed, 17 expected skips, 0 failed. Strict OpenSpec, headers, diff check, and Slopwatch passed. No raw transcripts, prompts, endpoints, credentials, host paths, session identifiers, or model/provider details are included here. |
bebf821 to
bd9d029
Compare
bd9d029 to
01176b9
Compare
Aaronontheweb
left a comment
There was a problem hiding this comment.
Focused review of the rollout and eval contracts.
| "personal": { | ||
| "shell_execute": [ | ||
| { | ||
| "verb": "awk", |
There was a problem hiding this comment.
This grant applies to every eval case, and awk can execute arbitrary programs. That can mask approval behavior outside the spill scenario. Please prefer the existing deterministic large fixture with a reviewed-safe producer such as cat, or scope this grant to the one case.
| /// tool in its private exposure set. | ||
| /// Exposes one deferred first-party or MCP tool schema by exact name. | ||
| /// The owning actor intercepts successful results and adds the requested tool | ||
| /// to its private exposure set. Dispatch still runs normal authorization. |
There was a problem hiding this comment.
Important semantic boundary: loading controls schema exposure only. A recalled deferred name may still dispatch, and normal authorization remains authoritative. Please keep the direct-dispatch authorization regression and avoid describing this as an execution grant or lease.
| var inline = BoundedOutputReader.Window(modelFacingResult, budget); | ||
| var spillPath = await TryWriteSpillAsync(spillContent, toolCallId, context, ct); | ||
| return Compose(inline, spillPath, modelFacingResult.Length, budget); | ||
| var spillCallId = await TryWriteSpillAsync(spillContent, toolCallId, context, ct); |
There was a problem hiding this comment.
This production hunk renames the local value; it does not remove the inline head/tail window. Please keep the PR claim scoped to opaque call-id/path contract alignment rather than implying that no inline spill content is returned.
01176b9 to
a7f28d6
Compare
a7f28d6 to
20f35a5
Compare
20f35a5 to
cd64cbd
Compare
cd64cbd to
1046bbc
Compare
1046bbc to
7fcce09
Compare
7fcce09 to
cc22d6f
Compare
cc22d6f to
ea975a5
Compare
Aaronontheweb
left a comment
There was a problem hiding this comment.
LGTM - mostly tool hints and guidance
Summary
tool_output_readload_toolas schema exposure while retaining normal dispatch authorizationValidation
Evaluation scope
This PR changes model-visible guidance and spill behavior. The final-head hosted behavioral rerun is still required; only PII-free aggregate evidence will be posted here.
Stack
This is PR 3 of 3. Its base is PR #2045, now refreshed at
c19edbde. No merge or auto-merge is enabled.