Repository navigation
test(acp): record Haiku 5.5 qualification as FAIL - #98
Conversation
Endpoint auditContract: pinned release artifact
Actionable client-only calls (0)none Actionable server-only operations (18)
Documented non-divergences (13)Client calls intentionally absent from the filtered contract (11)
Reason: Operational Agent Server endpoints intentionally excluded from the filtered public release artifact.
Reason: Client-ahead API stacked on the pending Agent Server meta-profiles implementation.
Reason: Client-ahead API stacked on the pending Agent Server pre-flight LLM validation endpoint. Server operations covered by an exposed browser URL (2)
Reason: RemoteWorkspace.startWorkspaceSession exposes these authenticated URLs for browser iframe and file requests; they are not HttpClient method calls. |
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Record the production-shaped Haiku 5.5 High tuple as not qualified because the five bounded live runs lack same-session and same-turn acceptance evidence. Do not infer provider incompatibility or add Haiku-specific production logic. Add generic ACP regression cases and document the focused tests and repository validation results. The qualification is resolved with an explicit FAIL; Haiku routing is not authorized. Closes #69 Co-authored-by: openhands <openhands@all-hands.dev>
927e189 to
7a84c6f
Compare
Result
FAIL — Haiku 5.5 High is not qualified. The five bounded live attempts do not preserve same-session and same-turn evidence for the exact served model, effective
effort=high, read-only proof, subscription/OAuth without PAYG, and nativejson_schemaacceptance plus schema-valid output. This is missing evidence, not an observed provider incompatibility.This PR adds generic ACP regression coverage and records the live evidence gap, environment versions, SDK SHA, and validation results. No production source changed because the existing generic Claude path is sufficient and the live evidence does not justify a Haiku-specific branch.
Verification
make validate: passed (SDK 6,162; Agent Server 2,083; Workspace 191; Cross 438; Tools 926; Pyright 0 errors).git diff --check: passed.Closes #69
Haiku routing remains blocked for Scheduler #73 and Workflow OpenHands#362 because this qualification is FAIL.