Skip to content

test(acp): record Haiku 5.5 qualification as FAIL - #98

Merged
geeen-jp merged 1 commit into
mainfrom
aidw/exec/exec-196c093cdb867e7062661300039d3335
Oct 10, 2026
Merged

geeen-jp merged 1 commit into
mainfrom
aidw/exec/exec-196c093cdb867e7062661300039d3335

Conversation

@geeen-jp

Copy link
Copy Markdown
Owner

Result

FAIL — Haiku 5.5 High is not qualified. The five bounded live attempts do not preserve same-session and same-turn evidence for the exact served model, effective effort=high, read-only proof, subscription/OAuth without PAYG, and native json_schema acceptance plus schema-valid output. This is missing evidence, not an observed provider incompatibility.

This PR adds generic ACP regression coverage and records the live evidence gap, environment versions, SDK SHA, and validation results. No production source changed because the existing generic Claude path is sufficient and the live evidence does not justify a Haiku-specific branch.

Verification

  • Focused ACP tests: 526 passed.
  • make validate: passed (SDK 6,162; Agent Server 2,083; Workspace 191; Cross 438; Tools 926; Pyright 0 errors).
  • git diff --check: passed.

Closes #69

Haiku routing remains blocked for Scheduler #73 and Workflow OpenHands#362 because this qualification is FAIL.

@github-actions

Copy link
Copy Markdown

Endpoint audit

⚠️ 18 actionable Agent Server contract divergence(s) · report-only

Contract: pinned release artifact

Category Count
Actionable client-only calls 0
Actionable server-only operations 18
Documented non-divergences 13
Agent Server contract operations 129
Audited handwritten client endpoints 119

Actionable client-only calls (0)

none

Actionable server-only operations (18)

  • DELETE /api/canvas-extensions/installed/{}
  • DELETE /api/llm/provider-connections/{}
  • GET /api/canvas-extensions/installed
  • GET /api/canvas-extensions/installed/{}
  • GET /api/canvas-extensions/installed/{}/bundle
  • GET /api/conversations/{}/events
  • GET /api/file/archive
  • GET /api/git/commits
  • GET /api/git/commits/{}/changes
  • GET /api/init
  • GET /api/llm/provider-connections
  • PATCH /api/canvas-extensions/installed/{}
  • PATCH /api/llm/provider-connections/{}
  • POST /api/canvas-extensions/install
  • POST /api/conversations/{}/load_plugin
  • POST /api/file/create_directory
  • POST /api/init
  • POST /api/llm/provider-connections
Documented non-divergences (13)

Client calls intentionally absent from the filtered contract (11)

  • GET /
  • GET /alive
  • GET /health
  • GET /ready
  • GET /server_info

Reason: Operational Agent Server endpoints intentionally excluded from the filtered public release artifact.
Owner: OpenHands runtime maintainers

  • DELETE /api/meta-profiles/{}
  • GET /api/meta-profiles
  • GET /api/meta-profiles/{}
  • POST /api/meta-profiles/{}
  • POST /api/meta-profiles/{}/activate

Reason: Client-ahead API stacked on the pending Agent Server meta-profiles implementation.
Owner: OpenHands SDK maintainers
Tracking: OpenHands#3744

  • POST /api/profiles/{}/validate

Reason: Client-ahead API stacked on the pending Agent Server pre-flight LLM validation endpoint.
Owner: OpenHands TypeScript client maintainers
Tracking: OpenHands#4422

Server operations covered by an exposed browser URL (2)

  • GET /api/conversations/{}/workspace
  • GET /api/conversations/{}/workspace/{}

Reason: RemoteWorkspace.startWorkspaceSession exposes these authenticated URLs for browser iframe and file requests; they are not HttpClient method calls.
Owner: OpenHands TypeScript client maintainers

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Python API breakage checks — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Coverage

Coverage Report •
FileStmtsMissCoverMissing
TOTAL429131183372% 
report-only-changed-files is enabled. No files were changed during this commit :)

Record the production-shaped Haiku 5.5 High tuple as not qualified because the five bounded live runs lack same-session and same-turn acceptance evidence. Do not infer provider incompatibility or add Haiku-specific production logic.

Add generic ACP regression cases and document the focused tests and repository validation results. The qualification is resolved with an explicit FAIL; Haiku routing is not authorized.

Closes #69

Co-authored-by: openhands <openhands@all-hands.dev>
@geeen-jp
geeen-jp force-pushed the aidw/exec/exec-196c093cdb867e7062661300039d3335 branch from 927e189 to 7a84c6f Compare October 10, 2026 20:28
@geeen-jp
geeen-jp merged commit 10a3cc3 into main Oct 10, 2026
31 checks passed
@geeen-jp
geeen-jp deleted the aidw/exec/exec-196c093cdb867e7062661300039d3335 branch October 10, 2026 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Qualification: Claude Haiku 5.5 High を exact-model/read-only/native-schema tuple として検証する

1 participant