diff --git a/fern/versions/latest/pages/model-server/claude.mdx b/fern/versions/latest/pages/model-server/claude.mdx new file mode 100644 index 0000000000..d8fd0d212d --- /dev/null +++ b/fern/versions/latest/pages/model-server/claude.mdx @@ -0,0 +1,126 @@ +--- +title: "Claude" +description: "Use Anthropic Claude models through NeMo Gym's Responses API" +position: 4 +--- + +NeMo Gym offers two complementary ways to work with Claude, aimed at different audiences: + +- **Egress (`anthropic_model` — this page):** Your harness speaks NeMo Gym's native [Responses API](https://developers.openai.com/api/reference/resources/responses/methods/create) on `/v1/responses` and the backend is Anthropic Claude. Use this when you already run Gym agents or harnesses in Responses format and want Claude as the policy model. +- **Ingress (built into every model server):** Your harness speaks Anthropic's [Messages API](https://docs.anthropic.com/en/api/messages) on `/v1/messages` and the backend is any Gym model server (vLLM, OpenAI, an inference provider, etc.). Every Gym model server exposes `/v1/messages` by default, mapping Messages ↔ Responses around its own `responses()` implementation. Blackbox agents that already talk Anthropic Messages — notably the [Claude Code Agent](https://github.com/NVIDIA-NeMo/Gym/tree/main/responses_api_agents/claude_code_agent) — can target any Gym backend without rewriting to Responses. + +The `anthropic_model` server is the egress path: it accepts Responses API requests, translates them to Anthropic `/v1/messages`, and maps Anthropic responses back to Responses objects. + +Conversion logic lives in `nemo_gym.anthropic_converter` and is shared with the ingress `/v1/messages` route on other Gym model servers. + + +For **training** workloads that require token IDs and log probabilities, use [vLLM](/model-server/vllm) instead. Anthropic's hosted API does not expose the token-level information needed for RL training. + + + +## Supported APIs + +This server exposes one endpoint and converts to Anthropic Messages under the hood: + +- **OpenAI Responses** — `/v1/responses` + +Chat Completions (`/v1/chat/completions`) is not supported on this server. + +If your agent speaks Anthropic Messages instead of Responses, you do not need `anthropic_model`. Point the agent at any Gym model server's `/v1/messages` endpoint (see ingress above). + +## Set Your Credentials + +Store your values in `env.yaml` in the project root (gitignored): + +```yaml +policy_base_url: https://api.anthropic.com +policy_api_key: your-api-key +policy_model_name: claude-sonnet-4-6 +``` + +`policy_base_url` accepts either a host-only or `/v1` style URL. Both `https://api.anthropic.com` and `https://api.anthropic.com/v1` resolve to `/v1/messages`. + +## Configuration Reference + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `anthropic_base_url` | `str` | `https://api.anthropic.com/v1` | Base URL for the Anthropic API. | +| `anthropic_api_key` | `str` | — | **Required.** Anthropic API key (`x-api-key` header). | +| `anthropic_model` | `str` | — | **Required.** Model identifier (for example, `claude-sonnet-4-6`). | +| `max_tokens` | `int` | `32768` | Maximum tokens to generate per request when `max_output_tokens` is not set on the Responses request. | +| `anthropic_version` | `str` | `2023-06-01` | Value for the `anthropic-version` header. | +| `thinking` | `dict` | `null` | Typed thinking config for modern Claude models (for example, `{type: adaptive}`). | +| `thinking_budget_tokens` | `int` | `null` | Budget for older models that use `thinking: {type: enabled, budget_tokens: ...}`. | +| `max_concurrent_requests` | `int` | `null` | Cap on in-flight upstream requests (per-process). `null` = unlimited. | +| `extra_body` | `dict` | `{}` | Provider-specific fields merged into every Anthropic request body. | + + +**The model is fixed by configuration.** This server always sends the configured `anthropic_model` (from `policy_model_name`) to Anthropic. To run a different model, change the config and start a new server. + +Do not set both `thinking` and `thinking_budget_tokens` — the server rejects ambiguous thinking configuration. + + + +### Thinking configuration + +For modern Claude models, prefer adaptive thinking: + +```yaml +thinking: + type: adaptive +``` + +`thinking_budget_tokens` remains available for older models that require manual `thinking: {type: enabled, budget_tokens: ...}`. + +### Model-specific behavior + +Claude Opus 4.7 and 4.8 reject configurable sampling parameters (`temperature`, `top_p`, `top_k`). Omit them from requests and use prompting or adaptive thinking/effort controls instead. + +Responses `input_image` parts are supported when `image_url` is a base64 data URL. Supported media types are `image/jpeg`, `image/png`, `image/gif`, and `image/webp`; remote image URLs are rejected with HTTP 400. + +Provider-specific Anthropic fields that are not modeled as typed config can be passed through `extra_body`. + +## Usage Example + +### 1. Set model and environment config + +```bash +environment_config="resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml" +model_config="responses_api_models/anthropic_model/configs/anthropic_model.yaml" +``` + +### 2. Start servers + +```bash +ng_run "+config_paths=[${environment_config},${model_config}]" +``` + +### 3. Evaluate your agent + +```bash +mkdir -p results + +ng_collect_rollouts +agent_name=example_single_tool_call_simple_agent \ + +input_jsonl_fpath=resources_servers/example_single_tool_call/data/example.jsonl \ + +output_jsonl_fpath=results/claude_example_single_tool_call_rollouts.jsonl \ + +limit=1 \ + +num_repeats=1 +``` + +### Smoke test the model server + +Once the model server is running, send a direct Responses request: + +```bash +curl -s /v1/responses \ + -H 'Content-Type: application/json' \ + -d '{ + "input": "Say hello in one short sentence.", + "max_output_tokens": 64 + }' | python -m json.tool +``` + + +This example uses the simple agent harness because it exercises `anthropic_model` as the policy model server through NeMo Gym's `/v1/responses` interface. For Claude Code workflows that speak Anthropic Messages, use the ingress path: set the agent's `model_server` ref so the CLI targets `/v1/messages` on any Gym model server (see [Claude Code Agent](https://github.com/NVIDIA-NeMo/Gym/tree/main/responses_api_agents/claude_code_agent)). + + diff --git a/fern/versions/latest/pages/model-server/index.mdx b/fern/versions/latest/pages/model-server/index.mdx index 38d805bf45..bb52a4b19f 100644 --- a/fern/versions/latest/pages/model-server/index.mdx +++ b/fern/versions/latest/pages/model-server/index.mdx @@ -26,6 +26,12 @@ Use models hosted on Azure OpenAI deployments. cloud + +Use Anthropic Claude models through NeMo Gym's Responses API. + +cloud + + Use hosted providers like Fireworks, Together.ai, OpenRouter, and more. diff --git a/fern/versions/latest/pages/model-server/inference-providers.mdx b/fern/versions/latest/pages/model-server/inference-providers.mdx index aa3bdf818e..57f66275b4 100644 --- a/fern/versions/latest/pages/model-server/inference-providers.mdx +++ b/fern/versions/latest/pages/model-server/inference-providers.mdx @@ -1,7 +1,7 @@ --- title: "Inference Providers" description: "Use hosted inference providers like Fireworks, Together.ai, OpenRouter, and more for eval workloads" -position: 4 +position: 5 --- The `inference_provider` server connects NeMo Gym to any hosted inference provider. The server manages the conversion to and from the Responses API: it translates incoming Responses requests to Chat Completions for the provider and converts the reply back into a Responses object — so your agent code stays the same across backends. diff --git a/fern/versions/latest/pages/model-server/local-vllm-proxy.mdx b/fern/versions/latest/pages/model-server/local-vllm-proxy.mdx index 09e43857f6..589bc391a4 100644 --- a/fern/versions/latest/pages/model-server/local-vllm-proxy.mdx +++ b/fern/versions/latest/pages/model-server/local-vllm-proxy.mdx @@ -1,7 +1,7 @@ --- title: "Local vLLM Proxy" description: "Expose one Local vLLM deployment as multiple model servers" -position: 7 +position: 8 --- LocalVLLMModelProxy (in `responses_api_models/local_vllm_model_proxy`) is a lightweight model server that forwards requests to an existing [LocalVLLMModel](/model-server/local-vllm) instead of launching its own vLLM engine. diff --git a/fern/versions/latest/pages/model-server/local-vllm.mdx b/fern/versions/latest/pages/model-server/local-vllm.mdx index 419308c841..55f8b9e543 100644 --- a/fern/versions/latest/pages/model-server/local-vllm.mdx +++ b/fern/versions/latest/pages/model-server/local-vllm.mdx @@ -1,7 +1,7 @@ --- title: "Local vLLM" description: "Gym-managed vLLM server deployment" -position: 6 +position: 7 --- NeMo Gym can launch and manage the vLLM server for you using LocalVLLMModel (in `responses_api_models/local_vllm_model`). diff --git a/fern/versions/latest/pages/model-server/vllm.mdx b/fern/versions/latest/pages/model-server/vllm.mdx index 1fc3bb94d2..4e50cd4fc6 100644 --- a/fern/versions/latest/pages/model-server/vllm.mdx +++ b/fern/versions/latest/pages/model-server/vllm.mdx @@ -1,7 +1,7 @@ --- title: "vLLM" description: "Wrapper for an existing, external vLLM server" -position: 5 +position: 6 --- [vLLM](https://docs.vllm.ai/) is a popular LLM inference engine. The NeMo Gym VLLMModel server wraps vLLM's Chat Completions endpoint and converts requests and responses to NeMo Gym's native format, the OpenAI [Responses API](https://platform.openai.com/docs/api-reference/responses) schema. diff --git a/nemo_gym/anthropic_converter.py b/nemo_gym/anthropic_converter.py index ad3e7c6d82..c09e8a21e7 100644 --- a/nemo_gym/anthropic_converter.py +++ b/nemo_gym/anthropic_converter.py @@ -485,7 +485,7 @@ def responses_to_anthropic_response(self, response: NeMoGymResponse, model: str) if item_type == "message": content.extend(self._output_message_to_anthropic_blocks(item)) elif item_type == "reasoning": - content.extend(self._reasoning_item_to_anthropic_blocks(item)) + content.extend(self._reasoning_item_to_anthropic_blocks(item, default_empty_signature=True)) elif item_type == "function_call": content.append(self._function_call_to_tool_use(item)) has_tool_use = True @@ -712,16 +712,17 @@ def _content_to_text(self, content: Any) -> str: def _system_parts_to_anthropic_blocks(self, system_parts: List[str]) -> List[Dict[str, str]]: return [{"type": "text", "text": text} for text in system_parts if text] - def _reasoning_item_to_anthropic_blocks(self, item: Dict[str, Any]) -> List[Dict[str, Any]]: + def _reasoning_item_to_anthropic_blocks( + self, item: Dict[str, Any], default_empty_signature: bool = False + ) -> List[Dict[str, Any]]: blocks = [] for summary in item.get("summary", []): - # Anthropic's ThinkingBlock requires a signature; open-model backends don't - # produce one, so default to "" (the synthesized SSE never emits it anyway). - block = { - "type": "thinking", - "thinking": summary["text"], - "signature": item.get("encrypted_content") or "", - } + block: Dict[str, Any] = {"type": "thinking", "thinking": summary["text"]} + encrypted_content = item.get("encrypted_content") + if encrypted_content: + block["signature"] = encrypted_content + elif default_empty_signature: + block["signature"] = "" blocks.append(block) return blocks diff --git a/responses_api_models/anthropic_model/README.md b/responses_api_models/anthropic_model/README.md new file mode 100644 index 0000000000..d3487b3afe --- /dev/null +++ b/responses_api_models/anthropic_model/README.md @@ -0,0 +1,70 @@ +# Description + +`anthropic_model` is a native Anthropic Messages API model server behind NeMo Gym's `/v1/responses` interface. It translates NeMo Gym Responses API requests to Anthropic `/v1/messages` payloads and maps Anthropic responses back to NeMo Gym Responses API objects. + +It supports text messages, base64 image inputs, system/developer prompt extraction, function tools, previous tool calls/results, thinking blocks, usage mapping, and optional request concurrency limiting. It uses `nemo_gym.server_utils.request()` for raw aiohttp transport instead of the Anthropic Python SDK. + +# Usage + +Start with a resources server config and the Anthropic model config: + +```bash +ng_run "+config_paths=[resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml,responses_api_models/anthropic_model/configs/anthropic_model.yaml]" \ + +policy_base_url="$ANTHROPIC_BASE_URL" \ + +policy_api_key="$ANTHROPIC_API_KEY" \ + +policy_model_name="$ANTHROPIC_MODEL_NAME" +``` + +`anthropic_base_url` accepts either host-only or `/v1` style URLs. Both `https://api.anthropic.com` and `https://api.anthropic.com/v1` resolve to `/v1/messages`. + +This example uses the simple agent harness because it exercises `anthropic_model` as the policy model server through NeMo Gym's `/v1/responses` interface. `claude_code_agent` is a separate agent harness that invokes Claude Code/Anthropic directly, so it is useful for testing Claude Code workflows but does not validate this model server. + +For modern Claude models, prefer adaptive thinking with the typed `thinking` config: + +```yaml +thinking: + type: adaptive +``` + +`thinking_budget_tokens` remains available for older models that require manual `thinking: {type: enabled, budget_tokens: ...}`. + +Minimal direct smoke test once the model server is running: + +```bash +curl -s /v1/responses \ + -H 'Content-Type: application/json' \ + -d '{ + "input": "Say hello in one short sentence.", + "max_output_tokens": 64 + }' | python -m json.tool +``` + +Collect one rollout through the simple agent: + +```bash +mkdir -p results + +ng_collect_rollouts \ + +agent_name=example_single_tool_call_simple_agent \ + +input_jsonl_fpath=resources_servers/example_single_tool_call/data/example.jsonl \ + +output_jsonl_fpath=results/claude_example_single_tool_call_rollouts.jsonl \ + +limit=1 \ + +num_repeats=1 +``` + +# Notes + +Provider-specific Anthropic fields that are not modeled as typed config can be passed through `extra_body`. Some options are model-specific: Claude Opus 4.7 and 4.8 reject configurable sampling parameters (`temperature`, `top_p`, `top_k`), so omit them and use prompting or adaptive thinking/effort controls instead. + +Responses `input_image` parts are supported when `image_url` is a base64 data URL. Supported media types are `image/jpeg`, `image/png`, `image/gif`, and `image/webp`; remote image URLs are rejected with a 400. + +Anthropic `stop_reason` values are mapped to Responses-compatible `incomplete_details` when possible. `max_tokens` and `model_context_window_exceeded` map to `max_output_tokens`; `refusal` maps to `content_filter`. Other stop reasons such as `end_turn`, `tool_use`, and `pause_turn` remain complete responses. + +# Licensing information + +Code: Apache 2.0 + +Data: N/A + +Dependencies: +- nemo_gym: Apache 2.0 diff --git a/responses_api_models/anthropic_model/app.py b/responses_api_models/anthropic_model/app.py new file mode 100644 index 0000000000..7aaac346c0 --- /dev/null +++ b/responses_api_models/anthropic_model/app.py @@ -0,0 +1,125 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import asyncio +from contextlib import nullcontext +from typing import Any, Dict, Optional + +from fastapi import HTTPException, Request +from pydantic import Field + +from nemo_gym.anthropic_converter import ( + SUPPORTED_ANTHROPIC_IMAGE_MEDIA_TYPES, + AnthropicConverter, +) +from nemo_gym.base_responses_api_model import ( + BaseResponsesAPIModelConfig, + Body, + SimpleResponsesAPIModel, +) +from nemo_gym.openai_utils import ( + NeMoGymChatCompletion, + NeMoGymChatCompletionCreateParamsNonStreaming, + NeMoGymResponse, + NeMoGymResponseCreateParamsNonStreaming, +) +from nemo_gym.server_utils import get_response_json, raise_for_status +from nemo_gym.server_utils import request as aiohttp_request + + +# Re-exported for backwards-compatible imports; the converter now lives in nemo_gym core so it +# can be shared with the inverse-direction (ingress) Anthropic Messages proxy. +__all__ = ["AnthropicModel", "AnthropicModelConfig", "AnthropicConverter", "SUPPORTED_ANTHROPIC_IMAGE_MEDIA_TYPES"] + + +class AnthropicModelConfig(BaseResponsesAPIModelConfig): + anthropic_base_url: str = "https://api.anthropic.com/v1" + anthropic_api_key: str + anthropic_model: str + max_tokens: int + anthropic_version: str = "2023-06-01" + thinking: Optional[Dict[str, Any]] = None + thinking_budget_tokens: Optional[int] = None + max_concurrent_requests: Optional[int] = Field( + default=None, + description=( + "Cap on in-flight upstream requests from this server (per-process asyncio.Semaphore). None = unlimited." + ), + ) + extra_body: Dict[str, Any] = Field(default_factory=dict) + + +class AnthropicModel(SimpleResponsesAPIModel): + config: AnthropicModelConfig + + def model_post_init(self, context): + self._converter = AnthropicConverter() + self._semaphore = ( + asyncio.Semaphore(self.config.max_concurrent_requests) + if self.config.max_concurrent_requests is not None + else nullcontext() + ) + return super().model_post_init(context) + + async def responses( + self, request: Request, body: NeMoGymResponseCreateParamsNonStreaming = Body() + ) -> NeMoGymResponse: + try: + anthropic_body = self._converter.responses_to_anthropic( + body=body, + model=self.config.anthropic_model, + max_tokens=self.config.max_tokens, + thinking=self.config.thinking, + thinking_budget_tokens=self.config.thinking_budget_tokens, + extra_body=self.config.extra_body, + ) + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + async with self._semaphore: + anthropic_response = await self._messages_create(anthropic_body, cookies=request.cookies) + + return self._converter.anthropic_to_responses( + anthropic_response=anthropic_response, + request_body=body, + model=self.config.anthropic_model, + ) + + async def chat_completions( + self, body: NeMoGymChatCompletionCreateParamsNonStreaming = Body() + ) -> NeMoGymChatCompletion: + raise NotImplementedError("anthropic_model supports /v1/responses only") + + async def _messages_create(self, body: Dict[str, Any], cookies: Dict[str, str]) -> Dict[str, Any]: + request_kwargs = { + "url": self._messages_url(), + "json": body, + "headers": { + "x-api-key": self.config.anthropic_api_key, + "anthropic-version": self.config.anthropic_version, + }, + "cookies": cookies, + } + response = await aiohttp_request(method="POST", **request_kwargs) + await raise_for_status(response) + return await get_response_json(response) + + def _messages_url(self) -> str: + base_url = self.config.anthropic_base_url.rstrip("/") + if base_url.endswith("/v1"): + return f"{base_url}/messages" + return f"{base_url}/v1/messages" + + +if __name__ == "__main__": + AnthropicModel.run_webserver() diff --git a/responses_api_models/anthropic_model/configs/anthropic_model.yaml b/responses_api_models/anthropic_model/configs/anthropic_model.yaml new file mode 100644 index 0000000000..120dd218d9 --- /dev/null +++ b/responses_api_models/anthropic_model/configs/anthropic_model.yaml @@ -0,0 +1,13 @@ +policy_model: + responses_api_models: + anthropic_model: + entrypoint: app.py + anthropic_base_url: ${policy_base_url} + anthropic_api_key: ${policy_api_key} + anthropic_model: ${policy_model_name} + max_tokens: 32768 + anthropic_version: "2023-06-01" + thinking: null + thinking_budget_tokens: null + max_concurrent_requests: null + extra_body: {} diff --git a/responses_api_models/anthropic_model/requirements.txt b/responses_api_models/anthropic_model/requirements.txt new file mode 100644 index 0000000000..00ed83213e --- /dev/null +++ b/responses_api_models/anthropic_model/requirements.txt @@ -0,0 +1 @@ +-e nemo-gym[dev] @ ../../ diff --git a/responses_api_models/anthropic_model/tests/test_app.py b/responses_api_models/anthropic_model/tests/test_app.py new file mode 100644 index 0000000000..805d5691d4 --- /dev/null +++ b/responses_api_models/anthropic_model/tests/test_app.py @@ -0,0 +1,712 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import asyncio +import json +from contextlib import nullcontext +from unittest.mock import MagicMock + +import pytest +from fastapi.testclient import TestClient + +from nemo_gym.openai_utils import NeMoGymResponseCreateParamsNonStreaming +from nemo_gym.server_utils import ServerClient +from responses_api_models.anthropic_model.app import AnthropicConverter, AnthropicModel, AnthropicModelConfig + + +class TestAnthropicConverter: + def test_responses_to_anthropic_maps_messages_tools_and_thinking(self) -> None: + converter = AnthropicConverter() + body = NeMoGymResponseCreateParamsNonStreaming( + input=[ + { + "type": "message", + "role": "developer", + "content": "Be concise.", + }, + { + "type": "message", + "role": "user", + "content": [{"type": "input_text", "text": "What is the weather?"}], + }, + { + "type": "reasoning", + "id": "rs_123", + "summary": [{"type": "summary_text", "text": "Need weather data."}], + "encrypted_content": "signature_123", + }, + { + "type": "function_call", + "call_id": "toolu_123", + "name": "get_weather", + "arguments": '{"city": "San Francisco"}', + }, + { + "type": "function_call_output", + "call_id": "toolu_123", + "output": '{"temperature": 65}', + }, + ], + instructions="You are helpful.", + max_output_tokens=512, + temperature=0.2, + tools=[ + { + "type": "function", + "name": "get_weather", + "description": "Get weather.", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"], + }, + "strict": True, + } + ], + tool_choice={"type": "function", "name": "get_weather"}, + ) + + actual = converter.responses_to_anthropic( + body=body, + model="claude-sonnet-4-20250514", + max_tokens=4096, + thinking=None, + thinking_budget_tokens=1024, + extra_body={"metadata": {"request_id": "abc"}}, + ) + + assert actual == { + "metadata": {"request_id": "abc"}, + "model": "claude-sonnet-4-20250514", + "max_tokens": 512, + "messages": [ + { + "role": "user", + "content": [{"type": "text", "text": "What is the weather?"}], + }, + { + "role": "assistant", + "content": [ + { + "type": "thinking", + "thinking": "Need weather data.", + "signature": "signature_123", + }, + { + "type": "tool_use", + "id": "toolu_123", + "name": "get_weather", + "input": {"city": "San Francisco"}, + }, + ], + }, + { + "role": "user", + "content": [ + { + "type": "tool_result", + "tool_use_id": "toolu_123", + "content": '{"temperature": 65}', + } + ], + }, + ], + "system": [ + {"type": "text", "text": "You are helpful."}, + {"type": "text", "text": "Be concise."}, + ], + "temperature": 0.2, + "tools": [ + { + "name": "get_weather", + "description": "Get weather.", + "input_schema": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"], + }, + } + ], + "tool_choice": {"type": "tool", "name": "get_weather"}, + "thinking": {"type": "enabled", "budget_tokens": 1024}, + } + + def test_anthropic_to_responses_maps_text_thinking_tools_and_usage(self) -> None: + converter = AnthropicConverter() + request_body = NeMoGymResponseCreateParamsNonStreaming(input="hello") + + response = converter.anthropic_to_responses( + anthropic_response={ + "id": "msg_123", + "type": "message", + "role": "assistant", + "model": "claude-sonnet-4-20250514", + "content": [ + { + "type": "thinking", + "thinking": "I should call a tool.", + "signature": "signature_123", + }, + {"type": "text", "text": "Let me check."}, + { + "type": "tool_use", + "id": "toolu_123", + "name": "get_weather", + "input": {"city": "San Francisco"}, + }, + ], + "stop_reason": "tool_use", + "usage": {"input_tokens": 10, "output_tokens": 20, "cache_read_input_tokens": 3}, + }, + request_body=request_body, + model="claude-sonnet-4-20250514", + ) + + assert response.model == "claude-sonnet-4-20250514" + assert response.output[0].type == "reasoning" + assert response.output[0].summary[0].text == "I should call a tool." + assert response.output[0].encrypted_content == "signature_123" + assert response.output[1].type == "message" + assert response.output[1].content[0].text == "Let me check." + assert response.output[2].type == "function_call" + assert response.output[2].call_id == "toolu_123" + assert response.output[2].name == "get_weather" + assert json.loads(response.output[2].arguments) == {"city": "San Francisco"} + assert response.usage.input_tokens == 10 + assert response.usage.output_tokens == 20 + assert response.usage.total_tokens == 30 + assert response.usage.input_tokens_details.cached_tokens == 3 + + def test_anthropic_to_responses_maps_stop_reasons_to_incomplete_details(self) -> None: + converter = AnthropicConverter() + request_body = NeMoGymResponseCreateParamsNonStreaming(input="hello") + + base_response = { + "id": "msg_123", + "type": "message", + "role": "assistant", + "model": "claude-sonnet-4-20250514", + "content": [{"type": "text", "text": "Partial response."}], + } + + max_tokens_response = converter.anthropic_to_responses( + anthropic_response=base_response | {"stop_reason": "max_tokens"}, + request_body=request_body, + model="claude-sonnet-4-20250514", + ) + assert max_tokens_response.incomplete_details.reason == "max_output_tokens" + + context_response = converter.anthropic_to_responses( + anthropic_response=base_response | {"stop_reason": "model_context_window_exceeded"}, + request_body=request_body, + model="claude-sonnet-4-20250514", + ) + assert context_response.incomplete_details.reason == "max_output_tokens" + + refusal_response = converter.anthropic_to_responses( + anthropic_response=base_response | {"stop_reason": "refusal"}, + request_body=request_body, + model="claude-sonnet-4-20250514", + ) + assert refusal_response.incomplete_details.reason == "content_filter" + + tool_use_response = converter.anthropic_to_responses( + anthropic_response=base_response | {"stop_reason": "tool_use"}, + request_body=request_body, + model="claude-sonnet-4-20250514", + ) + assert tool_use_response.incomplete_details is None + + def test_responses_to_anthropic_maps_typed_adaptive_thinking(self) -> None: + converter = AnthropicConverter() + body = NeMoGymResponseCreateParamsNonStreaming(input="Hello") + + actual = converter.responses_to_anthropic( + body=body, + model="claude-opus-4-8", + max_tokens=1024, + thinking={"type": "adaptive"}, + thinking_budget_tokens=None, + extra_body={}, + ) + + assert actual["thinking"] == {"type": "adaptive"} + + def test_responses_to_anthropic_maps_input_image_data_url(self) -> None: + converter = AnthropicConverter() + body = NeMoGymResponseCreateParamsNonStreaming( + input=[ + { + "type": "message", + "role": "user", + "content": [ + {"type": "input_text", "text": "What is in this image?"}, + { + "type": "input_image", + "image_url": "data:image/png;base64,iVBORw0KGgo=", + "detail": "high", + }, + ], + } + ] + ) + + actual = converter.responses_to_anthropic( + body=body, + model="claude-sonnet-4-20250514", + max_tokens=1024, + thinking=None, + thinking_budget_tokens=None, + extra_body={}, + ) + + assert actual["messages"] == [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What is in this image?"}, + { + "type": "image", + "source": { + "type": "base64", + "media_type": "image/png", + "data": "iVBORw0KGgo=", + }, + }, + ], + } + ] + + def test_responses_to_anthropic_rejects_remote_image_url(self) -> None: + converter = AnthropicConverter() + body = NeMoGymResponseCreateParamsNonStreaming( + input=[ + { + "type": "message", + "role": "user", + "content": [ + { + "type": "input_image", + "image_url": "https://example.com/image.png", + "detail": "high", + } + ], + } + ] + ) + + with pytest.raises(ValueError, match="base64 data URLs"): + converter.responses_to_anthropic( + body=body, + model="claude-sonnet-4-20250514", + max_tokens=1024, + thinking=None, + thinking_budget_tokens=None, + extra_body={}, + ) + + def test_responses_to_anthropic_rejects_invalid_image_data_url(self) -> None: + converter = AnthropicConverter() + body = NeMoGymResponseCreateParamsNonStreaming( + input=[ + { + "type": "message", + "role": "user", + "content": [ + { + "type": "input_image", + "image_url": "data:image/png;base64,not valid base64", + "detail": "high", + } + ], + } + ] + ) + + with pytest.raises(ValueError, match="invalid base64"): + converter.responses_to_anthropic( + body=body, + model="claude-sonnet-4-20250514", + max_tokens=1024, + thinking=None, + thinking_budget_tokens=None, + extra_body={}, + ) + + def test_responses_to_anthropic_rejects_ambiguous_thinking_config(self) -> None: + converter = AnthropicConverter() + body = NeMoGymResponseCreateParamsNonStreaming(input="Hello") + + with pytest.raises(ValueError, match="Configure Anthropic thinking in only one place"): + converter.responses_to_anthropic( + body=body, + model="claude-opus-4-8", + max_tokens=1024, + thinking={"type": "adaptive"}, + thinking_budget_tokens=1024, + extra_body={}, + ) + + def test_responses_to_anthropic_rejects_opus_4_8_sampling_params(self) -> None: + converter = AnthropicConverter() + + with pytest.raises(ValueError, match="does not support configurable sampling"): + converter.responses_to_anthropic( + body=NeMoGymResponseCreateParamsNonStreaming(input="Hello", temperature=0.2), + model="claude-opus-4-8", + max_tokens=1024, + thinking={"type": "adaptive"}, + thinking_budget_tokens=None, + extra_body={}, + ) + + with pytest.raises(ValueError, match="does not support configurable sampling"): + converter.responses_to_anthropic( + body=NeMoGymResponseCreateParamsNonStreaming(input="Hello"), + model="us/aws/anthropic/eccn-claude-opus-4-8", + max_tokens=1024, + thinking={"type": "adaptive"}, + thinking_budget_tokens=None, + extra_body={"top_k": 5}, + ) + + def test_responses_to_anthropic_merges_consecutive_function_calls(self) -> None: + converter = AnthropicConverter() + body = NeMoGymResponseCreateParamsNonStreaming( + input=[ + {"type": "function_call", "call_id": "toolu_1", "name": "a", "arguments": "{}"}, + {"type": "function_call", "call_id": "toolu_2", "name": "b", "arguments": '{"x": 1}'}, + ] + ) + actual = converter.responses_to_anthropic( + body=body, + model="claude-sonnet-4-20250514", + max_tokens=1024, + thinking=None, + thinking_budget_tokens=None, + extra_body={}, + ) + assert actual["messages"] == [ + { + "role": "assistant", + "content": [ + {"type": "tool_use", "id": "toolu_1", "name": "a", "input": {}}, + {"type": "tool_use", "id": "toolu_2", "name": "b", "input": {"x": 1}}, + ], + } + ] + + def test_anthropic_to_responses_maps_multiple_tool_use_blocks(self) -> None: + converter = AnthropicConverter() + request_body = NeMoGymResponseCreateParamsNonStreaming(input="hi") + response = converter.anthropic_to_responses( + anthropic_response={ + "content": [ + {"type": "tool_use", "id": "toolu_1", "name": "get_weather", "input": {"city": "SF"}}, + {"type": "tool_use", "id": "toolu_2", "name": "get_time", "input": {"tz": "PST"}}, + ], + "stop_reason": "tool_use", + "usage": {"input_tokens": 1, "output_tokens": 2}, + }, + request_body=request_body, + model="claude-sonnet-4-20250514", + ) + assert [item.type for item in response.output] == ["function_call", "function_call"] + assert response.output[0].call_id == "toolu_1" + assert json.loads(response.output[1].arguments) == {"tz": "PST"} + + def test_anthropic_to_responses_preserves_text_tool_text_ordering(self) -> None: + converter = AnthropicConverter() + request_body = NeMoGymResponseCreateParamsNonStreaming(input="hi") + response = converter.anthropic_to_responses( + anthropic_response={ + "content": [ + {"type": "text", "text": "first"}, + {"type": "tool_use", "id": "toolu_1", "name": "f", "input": {}}, + {"type": "text", "text": "second"}, + ], + "stop_reason": "tool_use", + "usage": {"input_tokens": 1, "output_tokens": 2}, + }, + request_body=request_body, + model="claude-sonnet-4-20250514", + ) + assert [item.type for item in response.output] == ["message", "function_call", "message"] + assert response.output[2].content[0].text == "second" + + def test_responses_to_anthropic_reasoning_without_signature_omits_signature(self) -> None: + converter = AnthropicConverter() + body = NeMoGymResponseCreateParamsNonStreaming( + input=[{"type": "reasoning", "id": "rs_1", "summary": [{"type": "summary_text", "text": "thinking"}]}] + ) + actual = converter.responses_to_anthropic( + body=body, + model="claude-sonnet-4-20250514", + max_tokens=1024, + thinking=None, + thinking_budget_tokens=None, + extra_body={}, + ) + assert actual["messages"][0]["content"][0] == {"type": "thinking", "thinking": "thinking"} + assert "signature" not in actual["messages"][0]["content"][0] + + +class TestAnthropicModel: + def _setup_server( + self, + max_concurrent_requests=None, + thinking=None, + thinking_budget_tokens=None, + anthropic_model="claude-sonnet-4-20250514", + max_tokens=4096, + extra_body=None, + anthropic_base_url="https://api.anthropic.com/v1", + ) -> AnthropicModel: + config = AnthropicModelConfig( + host="0.0.0.0", + port=8081, + anthropic_base_url=anthropic_base_url, + anthropic_api_key="dummy_key", # pragma: allowlist secret + anthropic_model=anthropic_model, + max_tokens=max_tokens, + entrypoint="", + name="", + max_concurrent_requests=max_concurrent_requests, + thinking=thinking, + thinking_budget_tokens=thinking_budget_tokens, + extra_body=extra_body or {}, + ) + return AnthropicModel(config=config, server_client=MagicMock(spec=ServerClient)) + + async def test_sanity(self) -> None: + self._setup_server() + + def test_messages_url_accepts_host_or_v1_base_url(self) -> None: + assert self._setup_server(anthropic_base_url="https://api.anthropic.com")._messages_url() == ( + "https://api.anthropic.com/v1/messages" + ) + assert self._setup_server(anthropic_base_url="https://api.anthropic.com/v1")._messages_url() == ( + "https://api.anthropic.com/v1/messages" + ) + + def test_responses_endpoint_round_trip(self) -> None: + server = self._setup_server(thinking_budget_tokens=1024) + app = server.setup_webserver() + client = TestClient(app) + + called_body = {} + + async def mock_messages_create(body, cookies): + nonlocal called_body + called_body = body + return { + "id": "msg_123", + "type": "message", + "role": "assistant", + "model": "claude-sonnet-4-20250514", + "content": [{"type": "text", "text": "Hello from Claude."}], + "stop_reason": "end_turn", + "usage": {"input_tokens": 4, "output_tokens": 5}, + } + + server._messages_create = mock_messages_create + + response = client.post( + "/v1/responses", + json={ + "input": "hello", + "tools": [ + { + "type": "function", + "name": "finish", + "description": "Finish task.", + "parameters": {"type": "object", "properties": {}}, + "strict": True, + } + ], + }, + ) + + assert response.status_code == 200 + assert called_body["model"] == "claude-sonnet-4-20250514" + assert called_body["messages"] == [{"role": "user", "content": [{"type": "text", "text": "hello"}]}] + assert called_body["tools"] == [ + { + "name": "finish", + "description": "Finish task.", + "input_schema": {"type": "object", "properties": {}}, + } + ] + assert called_body["thinking"] == {"type": "enabled", "budget_tokens": 1024} + assert response.json()["output"][0]["content"][0]["text"] == "Hello from Claude." + + def test_responses_endpoint_propagates_upstream_error(self) -> None: + server = self._setup_server() + app = server.setup_webserver() + client = TestClient(app, raise_server_exceptions=False) + + async def boom(body, cookies): + raise RuntimeError("upstream 529 overloaded") + + server._messages_create = boom + + response = client.post("/v1/responses", json={"input": "hello"}) + assert response.status_code == 500 + + def test_responses_endpoint_sends_curl_shaped_anthropic_request_fields(self) -> None: + server = self._setup_server( + anthropic_model="claude-opus-4-6", + max_tokens=1024, + thinking={"type": "adaptive"}, + ) + app = server.setup_webserver() + client = TestClient(app) + + called_body = {} + + async def mock_messages_create(body, cookies): + nonlocal called_body + called_body = body + return { + "id": "msg_123", + "type": "message", + "role": "assistant", + "model": "claude-opus-4-6", + "content": [ + {"type": "thinking", "thinking": "Consider the greeting.", "signature": "signature_123"}, + {"type": "text", "text": "Hello!"}, + {"type": "tool_use", "id": "toolu_123", "name": "name", "input": {"location": "NYC"}}, + ], + "stop_reason": "tool_use", + "usage": {"input_tokens": 11, "output_tokens": 7}, + } + + server._messages_create = mock_messages_create + + response = client.post( + "/v1/responses", + json={ + "input": [{"content": "Hello, world", "role": "user", "type": "message"}], + "instructions": "Today's date is 2024-06-01.", + "temperature": 1, + "tools": [ + { + "type": "function", + "name": "name", + "parameters": { + "type": "object", + "properties": { + "location": {"type": "string"}, + "unit": {"type": "string"}, + }, + "required": ["location"], + }, + "strict": True, + } + ], + "top_p": 0.95, + }, + ) + + assert response.status_code == 200 + assert called_body == { + "thinking": {"type": "adaptive"}, + "model": "claude-opus-4-6", + "max_tokens": 1024, + "messages": [{"role": "user", "content": [{"type": "text", "text": "Hello, world"}]}], + "system": [{"type": "text", "text": "Today's date is 2024-06-01."}], + "temperature": 1.0, + "top_p": 0.95, + "tools": [ + { + "name": "name", + "input_schema": { + "type": "object", + "properties": { + "location": {"type": "string"}, + "unit": {"type": "string"}, + }, + "required": ["location"], + }, + } + ], + } + response_body = response.json() + assert response_body["output"][0]["type"] == "reasoning" + assert response_body["output"][0]["summary"][0]["text"] == "Consider the greeting." + assert response_body["output"][1]["content"][0]["text"] == "Hello!" + assert response_body["output"][2]["type"] == "function_call" + assert response_body["output"][2]["name"] == "name" + assert json.loads(response_body["output"][2]["arguments"]) == {"location": "NYC"} + assert response_body["usage"]["input_tokens"] == 11 + assert response_body["usage"]["output_tokens"] == 7 + + def test_responses_endpoint_rejects_opus_4_8_sampling_params(self) -> None: + server = self._setup_server(anthropic_model="claude-opus-4-8", thinking={"type": "adaptive"}) + app = server.setup_webserver() + client = TestClient(app) + + response = client.post("/v1/responses", json={"input": "hello", "temperature": 0.2}) + + assert response.status_code == 400 + assert "does not support configurable sampling" in response.json()["detail"] + + def test_responses_endpoint_rejects_remote_image_url(self) -> None: + server = self._setup_server() + app = server.setup_webserver() + client = TestClient(app) + + response = client.post( + "/v1/responses", + json={ + "input": [ + { + "type": "message", + "role": "user", + "content": [ + { + "type": "input_image", + "image_url": "https://example.com/image.png", + "detail": "high", + } + ], + } + ] + }, + ) + + assert response.status_code == 400 + assert "base64 data URLs" in response.json()["detail"] + + def test_semaphore_disabled_by_default(self) -> None: + server = self._setup_server() + assert isinstance(server._semaphore, type(nullcontext())) + + async def test_semaphore_caps_concurrency(self) -> None: + server = self._setup_server(max_concurrent_requests=2) + assert isinstance(server._semaphore, asyncio.Semaphore) + + in_flight = 0 + peak = 0 + + async def worker() -> None: + nonlocal in_flight, peak + async with server._semaphore: + in_flight += 1 + peak = max(peak, in_flight) + await asyncio.sleep(0.01) + in_flight -= 1 + + await asyncio.gather(*(worker() for _ in range(8))) + assert peak == 2 diff --git a/tests/unit_tests/test_anthropic_converter.py b/tests/unit_tests/test_anthropic_converter.py index 6fe997705a..e34ee842fe 100644 --- a/tests/unit_tests/test_anthropic_converter.py +++ b/tests/unit_tests/test_anthropic_converter.py @@ -75,6 +75,10 @@ def test_no_system_leaves_instructions_unset(self) -> None: ) assert params.instructions is None + def test_request_without_max_tokens_leaves_max_output_tokens_unset(self) -> None: + params = _converter().anthropic_request_to_responses({"messages": [{"role": "user", "content": "hi"}]}) + assert params.max_output_tokens is None + def test_system_list_without_text_leaves_instructions_unset(self) -> None: # A system list that contributes no usable text (empty-text blocks) yields no instructions. params = _converter().anthropic_request_to_responses(