Skip to content

Qualify Claude Sonnet 5.5 as an exact-model read-only ACP profile - #59

Merged
geeen-jp merged 2 commits into
mainfrom
issue-58
Sep 30, 2026
Merged

geeen-jp merged 2 commits into
mainfrom
issue-58

Conversation

@geeen-jp

Copy link
Copy Markdown
Owner

AGENT:
This PR qualifies Claude Sonnet 5.5 on the existing exact-model path. The evidence below is from a live run on this branch at 1a3865f9a3e57580ee665782980b66b6d9ea224d, whose tree is identical to the squash result.

  • Command: uv run python live_probe.py claude-sonnet-5-5. It uses the real ACPAgent and LocalConversation with:
    • the default Claude provider command (npx -y @agentclientprotocol/claude-agent-acp@0.81.0);
    • acp_model="claude-sonnet-5-5", acp_permission_policy="read_only", acp_config_options={"effort": "medium"} and acp_isolate_data_dir=True;
    • ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN unset.
  • Runtime:
    • The adapter reported agent_name='@agentclientprotocol/claude-agent-acp', agent_version='0.81.0'.
    • The bundled Claude Agent SDK is 0.3.280, with claudeCodeVersion 2.1.280.
    • Auth is first-party claude.ai subscription OAuth, seeded into the isolated config (Seeded Claude OAuth credentials into isolated config), with no API key or PAYG path.
  • Identity:
    • Requested: claude-sonnet-5-5.
    • Selector: default on fresh sessions, and claude-sonnet-5-5 after the in-process restart/load.
    • Served evidence: the provider-owned same-turn root assistant message.model from _claude/sdkMessage.
  • Log lines per turn:
    • fresh: Claude ACP same-turn root model verification passed: requested=claude-sonnet-5-5 selector=default root_models=['claude-sonnet-5-5', 'claude-sonnet-5-5', 'claude-sonnet-5-5'] delegated_models=[]
    • repeated: root_models=['claude-sonnet-5-5', 'claude-sonnet-5-5']
    • restart plus load_session of the same session (via restart_for_updated_credentials): selector=claude-sonnet-5-5 root_models=['claude-sonnet-5-5']
    • new agent instance: load_session returned "Resource not found" because the isolated config dir is per instance, so it fell back to a fresh session and root_models=['claude-sonnet-5-5'].
    • Every turn produced fresh evidence. None reused earlier evidence.
  • Effort: ACP session config option verified: effort=medium on every session, checked separately from model identity.
  • Read-only: session mode default, which is the Claude read-only mode from P0: Preserve and re-prove Claude read-only session mode after config updates #41.
    • The Write attempt was refused: ACP denying permission under read_only policy ... (option: reject).
    • The workspace was byte-identical before and after (README.txt sha256 0437f84c…, git status clean), and probe.txt was not created.
  • Cleanup: no adapter processes are left. The isolated openhands-claude-acp-* runtime dirs hold no .credentials.json afterwards.
  • Alias finding: the pinned runtime's model picker shows sonnet as "Sonnet 5", the previous generation, and claude-sonnet-5-5 only as "Custom model". The newer host Claude Code 2.1.284 maps sonnet to claude-sonnet-5-5 instead. The alias's meaning depends on the runtime version, so sonnet must never count as Sonnet 5.5 identity. The code already works this way: only concrete claude-* requests turn on the exact gate.
  • Tests: .venv/bin/python -m pytest tests/sdk tests/agent_server tests/cross gives 8569 passed and 4 failed.
    • test_activation_route_rejects_unavailable_binding and test_openai_chat_completions_gateway_over_real_server fail the same way on unchanged main in this environment.
    • The other two pass when run serially.
    • pyright, pre-commit and the ACP registry mirror check pass.

Why

ai-dev-workflow-go Planner, Reviewer and ScopeJudge are moving from Claude Opus 5.5 Medium to Claude Sonnet 5.5 Medium. Before that, the SDK has to qualify claude-sonnet-5-5 as an exact-model, read-only, subscription-only profile on the built-in Claude ACP path.

Summary

  • Adds claude-sonnet-5-5 ("Claude Sonnet 5.5") as an explicit full-model pin in the Claude pre-session suggestions and the TypeScript acp-providers.json mirror. Comments no longer describe the qualified 0.81.0 exact path as Opus-only.
  • The verification mechanics are unchanged. The existing checks already qualify Sonnet 5.5:
    • an exact claude- request on the pinned adapter;
    • same-turn root message.model equality;
    • a read-only re-proof;
    • an independent effort check.
  • Adds provider-shaped tests, with the Opus cases kept through parametrization:
    • exact match;
    • sonnet and default selectors with same-turn exact evidence;
    • an alias-only sonnet request, which does not publish an exact identity;
    • a different generation (claude-sonnet-5), which is rejected;
    • contradictory root models, which are rejected;
    • stale evidence across repeat, retry, resume, rebind and cancel, which is rejected;
    • effort kept independent of the model.

Issue Number

Closes #58

How to Test

.venv/bin/python -m pytest -q tests/sdk/agent/test_acp_agent.py tests/sdk/agent/test_acp_permission_policy.py tests/sdk/settings/test_acp_providers.py

The live probe needs Claude subscription OAuth and no API key.

Video/Screenshots

N/A

Design Doc

N/A (small change)

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

Qualified tuple for downstream geeen-jp/ai-dev-workflow-go#201:

Field Value
requested model claude-sonnet-5-5
effort medium
access read-only
adapter @agentclientprotocol/claude-agent-acp@0.81.0 (Claude Agent SDK 0.3.280, Claude Code 2.1.280)
auth subscription OAuth only
canonical SDK SHA the squash merge of this PR

There are no workflow or scheduler changes, and the Opus 5.5 profile is untouched.

@github-actions

Copy link
Copy Markdown

📁 PR Artifacts Notice

This PR contains a .pr/ directory with temporary PR-specific documents. The directory will be automatically removed when the PR is approved.

@github-actions

Copy link
Copy Markdown

Endpoint audit

⚠️ 18 actionable Agent Server contract divergence(s) · report-only

Contract: pinned release artifact

Category Count
Actionable client-only calls 0
Actionable server-only operations 18
Documented non-divergences 13
Agent Server contract operations 129
Audited handwritten client endpoints 119

Actionable client-only calls (0)

none

Actionable server-only operations (18)

  • DELETE /api/canvas-extensions/installed/{}
  • DELETE /api/llm/provider-connections/{}
  • GET /api/canvas-extensions/installed
  • GET /api/canvas-extensions/installed/{}
  • GET /api/canvas-extensions/installed/{}/bundle
  • GET /api/conversations/{}/events
  • GET /api/file/archive
  • GET /api/git/commits
  • GET /api/git/commits/{}/changes
  • GET /api/init
  • GET /api/llm/provider-connections
  • PATCH /api/canvas-extensions/installed/{}
  • PATCH /api/llm/provider-connections/{}
  • POST /api/canvas-extensions/install
  • POST /api/conversations/{}/load_plugin
  • POST /api/file/create_directory
  • POST /api/init
  • POST /api/llm/provider-connections
Documented non-divergences (13)

Client calls intentionally absent from the filtered contract (11)

  • GET /
  • GET /alive
  • GET /health
  • GET /ready
  • GET /server_info

Reason: Operational Agent Server endpoints intentionally excluded from the filtered public release artifact.
Owner: OpenHands runtime maintainers

  • DELETE /api/meta-profiles/{}
  • GET /api/meta-profiles
  • GET /api/meta-profiles/{}
  • POST /api/meta-profiles/{}
  • POST /api/meta-profiles/{}/activate

Reason: Client-ahead API stacked on the pending Agent Server meta-profiles implementation.
Owner: OpenHands SDK maintainers
Tracking: OpenHands#3744

  • POST /api/profiles/{}/validate

Reason: Client-ahead API stacked on the pending Agent Server pre-flight LLM validation endpoint.
Owner: OpenHands TypeScript client maintainers
Tracking: OpenHands#4422

Server operations covered by an exposed browser URL (2)

  • GET /api/conversations/{}/workspace
  • GET /api/conversations/{}/workspace/{}

Reason: RemoteWorkspace.startWorkspaceSession exposes these authenticated URLs for browser iframe and file requests; they are not HttpClient method calls.
Owner: OpenHands TypeScript client maintainers

@github-actions

Copy link
Copy Markdown

Python API breakage checks — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

Copy link
Copy Markdown

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

Copy link
Copy Markdown

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-sdk/openhands/sdk/settings
   acp_providers.py2371395%80, 617, 639, 652, 790, 830, 833, 847, 864–865, 869, 890, 893
TOTAL427231181372% 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

P1: Claude Sonnet 5.5 Mediumをexact-model/read-only Profileとしてqualifyする

1 participant