Skip to content

Qualify Claude Sonnet 5.5 native structured output - #61

Merged
geeen-jp merged 1 commit into
mainfrom
issue-60
Sep 30, 2026
Merged

geeen-jp merged 1 commit into
mainfrom
issue-60

Conversation

@geeen-jp

Copy link
Copy Markdown
Owner

AGENT:
Live qualification result: PASS. Minimal runs: 1 exploratory run plus 1 recorded run, one turn each. The recorded run at 2026-09-30 14:43 JST used fork main b1f94099269233b86dc0f93f79a03f1359eb1ff6. This PR changes only tests, so the runtime code is identical.

  • Setup:
    • real ACPAgent and LocalConversation, with the default Claude provider command;
    • adapter reported agent_name='@agentclientprotocol/claude-agent-acp', agent_version='0.81.0'; bundled Claude Agent SDK 0.3.280, Claude Code 2.1.280;
    • first-party subscription OAuth seeded into the isolated config, with ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN unset;
    • acp_model="claude-sonnet-5-5", acp_permission_policy="read_only", acp_config_options={"effort": "medium"}, and structured_output=StructuredOutputConfig(mode="json_schema", schema=...).
    • The schema: first_line string, line_count integer, verdict enum OK/EMPTY, all required, additionalProperties: false.
  • Same-turn evidence:
    • Served model: Claude ACP same-turn root model verification passed: requested=claude-sonnet-5-5 selector=default root_models=['claude-sonnet-5-5', 'claude-sonnet-5-5', 'claude-sonnet-5-5'] delegated_models=[]
    • Effort: ACP session config option verified: effort=medium
    • Read-only: session mode default, and the Write attempt was refused (ACP denying permission under read_only policy). The workspace was byte-identical afterwards and probe.txt was not created.
    • Native schema output: the provider emitted a StructuredOutput ACP tool call (pending → completed) with raw_input {"first_line":"sdk60 probe line one","line_count":3,"verdict":"OK"}.
    • get_agent_final_response(...), which backs the agent-server agent_final_response endpoint, returned exactly that JSON, and jsonschema.validate against the requested schema passed.
  • Cleanup: no adapter processes left, and no .credentials.json left in the isolated runtime dirs.
  • Tests: pytest tests/sdk/agent/test_acp_agent.py tests/sdk/conversation tests/sdk/settings gives 1247 passed; pre-commit (ruff, pycodestyle, pyright, import rules) passed.

Why

#58 qualified Claude Sonnet 5.5 but did not cover native schema structured output. Because of that, ai-dev-workflow-go#201 registered Sonnet 5.5 as prompt-only. This PR qualifies native json_schema output for the claude-sonnet-5-5 / medium / read-only / subscription OAuth tuple.

Summary

  • There are no runtime changes. The existing Claude ACP outputFormat session metadata and the StructuredOutput final-response path are not model-specific, and the live run confirms they work for Sonnet 5.5.
  • The fresh-session and resumed-session structured-output metadata tests now run for both exact models, claude-opus-5-5 and claude-sonnet-5-5, on the qualified read-only path. The alias + set_session_model path is still covered by test_claude_fresh_session_sends_meta_and_protocol_model.

Issue Number

Closes #60

How to Test

.venv/bin/python -m pytest -q tests/sdk/agent/test_acp_agent.py -k structured_output

Video/Screenshots

N/A

Design Doc

N/A (tests only)

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

Qualified tuple: claude-sonnet-5-5, effort medium, read-only, @agentclientprotocol/claude-agent-acp@0.81.0, subscription OAuth, native json_schema structured output PASS. A follow-up in geeen-jp/ai-dev-workflow-go will switch Sonnet 5.5 from prompt-only to native schema.

@github-actions

Copy link
Copy Markdown

Python API breakage checks — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

Copy link
Copy Markdown

📁 PR Artifacts Notice

This PR contains a .pr/ directory with temporary PR-specific documents. The directory will be automatically removed when the PR is approved.

@github-actions

Copy link
Copy Markdown

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

Copy link
Copy Markdown

Endpoint audit

⚠️ 18 actionable Agent Server contract divergence(s) · report-only

Contract: pinned release artifact

Category Count
Actionable client-only calls 0
Actionable server-only operations 18
Documented non-divergences 13
Agent Server contract operations 129
Audited handwritten client endpoints 119

Actionable client-only calls (0)

none

Actionable server-only operations (18)

  • DELETE /api/canvas-extensions/installed/{}
  • DELETE /api/llm/provider-connections/{}
  • GET /api/canvas-extensions/installed
  • GET /api/canvas-extensions/installed/{}
  • GET /api/canvas-extensions/installed/{}/bundle
  • GET /api/conversations/{}/events
  • GET /api/file/archive
  • GET /api/git/commits
  • GET /api/git/commits/{}/changes
  • GET /api/init
  • GET /api/llm/provider-connections
  • PATCH /api/canvas-extensions/installed/{}
  • PATCH /api/llm/provider-connections/{}
  • POST /api/canvas-extensions/install
  • POST /api/conversations/{}/load_plugin
  • POST /api/file/create_directory
  • POST /api/init
  • POST /api/llm/provider-connections
Documented non-divergences (13)

Client calls intentionally absent from the filtered contract (11)

  • GET /
  • GET /alive
  • GET /health
  • GET /ready
  • GET /server_info

Reason: Operational Agent Server endpoints intentionally excluded from the filtered public release artifact.
Owner: OpenHands runtime maintainers

  • DELETE /api/meta-profiles/{}
  • GET /api/meta-profiles
  • GET /api/meta-profiles/{}
  • POST /api/meta-profiles/{}
  • POST /api/meta-profiles/{}/activate

Reason: Client-ahead API stacked on the pending Agent Server meta-profiles implementation.
Owner: OpenHands SDK maintainers
Tracking: OpenHands#3744

  • POST /api/profiles/{}/validate

Reason: Client-ahead API stacked on the pending Agent Server pre-flight LLM validation endpoint.
Owner: OpenHands TypeScript client maintainers
Tracking: OpenHands#4422

Server operations covered by an exposed browser URL (2)

  • GET /api/conversations/{}/workspace
  • GET /api/conversations/{}/workspace/{}

Reason: RemoteWorkspace.startWorkspaceSession exposes these authenticated URLs for browser iframe and file requests; they are not HttpClient method calls.
Owner: OpenHands TypeScript client maintainers

@github-actions

Copy link
Copy Markdown

Coverage

Coverage Report •
FileStmtsMissCoverMissing
TOTAL427231181372% 
report-only-changed-files is enabled. No files were changed during this commit :)

@geeen-jp
geeen-jp merged commit d2d8b40 into main Sep 30, 2026
39 of 40 checks passed
@geeen-jp
geeen-jp deleted the issue-60 branch September 30, 2026 06:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Qualification: Claude Sonnet 5.5 のネイティブ structured output (schema) 対応を検証する

1 participant