From b4017d54ca70b4d28d75b14966ba410b67741e31 Mon Sep 17 00:00:00 2001 From: adil-a Date: Tue, 28 Jul 2026 03:32:01 +0000 Subject: [PATCH 01/25] =?UTF-8?q?feat(agents):=20RemoteAgent=20=E2=80=94?= =?UTF-8?q?=20thin=20proxy=20server=20for=20user-hosted=20remote=20agent?= =?UTF-8?q?=20services?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds responses_api_agents/remote_agent/: a thin agent server that brokers rollouts to an agent service running outside Gym's process tree. The remote service implements one endpoint (POST {agent_base_url}/v1/responses), receives only responses_create_params (verifier_metadata never leaves Gym), runs its own loop with its own model and tools, and returns one finished Responses API trajectory. The server owns seed_session, session cookies, strict trajectory validation, and /verify, and returns the verify response from /run. - failures never raise: every failure (remote endpoint down, timeout, malformed or interrupted reply, seed/verify errors, internal bugs) becomes a reward-0 sentinel verify-response routed to the failures sidecar and retried on resume; reused rollout/failures JSONL rows with stale result keys are sanitized rather than crashing or leaking routing - bounded outbound retries (3x, connect/disconnect only), per-request ClientTimeout, whole-run wallclock applied after semaphore acquire, redirects rejected, per-worker concurrency semaphore - optional forward_session: resources-server URL + session cookie sent as X-NeMo-Gym-* headers so the remote service can call Gym-hosted tools (stateful environments); declared-tools guard refuses silent-zero configurations; advertised_resources_url for off-host services - aggregate_metrics proxied to the resources server with a wallclock bound - 48 offline tests incl. an in-process stateful counter E2E (forwarded session cookie, real verify) and a collector round-trip pinning sidecar routing Part of the external-agent-integration epic #1396; supersedes the collector-side agent_url approach in #2006 (see PR description). Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- responses_api_agents/remote_agent/README.md | 39 + responses_api_agents/remote_agent/__init__.py | 0 responses_api_agents/remote_agent/app.py | 427 +++++++++ .../remote_agent/configs/remote_agent.yaml | 13 + .../remote_agent/requirements.txt | 1 + .../remote_agent/tests/__init__.py | 0 .../remote_agent/tests/test_app.py | 866 ++++++++++++++++++ 7 files changed, 1346 insertions(+) create mode 100644 responses_api_agents/remote_agent/README.md create mode 100644 responses_api_agents/remote_agent/__init__.py create mode 100644 responses_api_agents/remote_agent/app.py create mode 100644 responses_api_agents/remote_agent/configs/remote_agent.yaml create mode 100644 responses_api_agents/remote_agent/requirements.txt create mode 100644 responses_api_agents/remote_agent/tests/__init__.py create mode 100644 responses_api_agents/remote_agent/tests/test_app.py diff --git a/responses_api_agents/remote_agent/README.md b/responses_api_agents/remote_agent/README.md new file mode 100644 index 0000000000..1bef046ed5 --- /dev/null +++ b/responses_api_agents/remote_agent/README.md @@ -0,0 +1,39 @@ +# Remote Agent + +A thin agent server that brokers rollouts to an agent service you host yourself — in your own +repo, on your own infrastructure. Your service implements one endpoint, `POST /v1/responses`: +it receives the task's `responses_create_params`, runs its own agent loop (its own model and +tools, however many turns it needs), and returns one finished Responses API trajectory. + +Gym keeps everything else on its side: this server seeds the session, holds the session +cookies, verifies the trajectory on the resources server (`verifier_metadata` never leaves +Gym), and reports the verify response — so your rollouts land in the standard artifacts and +work with `gym eval profile`, aggregation, and (when your service routes its model calls +through a Gym model server) token-id capture for training. + +## Contract for your service + +- `POST {agent_base_url}/v1/responses` with the row's `responses_create_params` as the JSON body. +- Return a single finished Responses API object: the last output item is an assistant message, + no dangling tool calls, `usage` populated (`{input_tokens, output_tokens, total_tokens}`). +- Failures on Gym's side never crash a collection run: they are recorded as reward-0 rows in + the failures sidecar and retried on resume. + +## Gym-hosted tools (optional) + +With `forward_session: true`, each request to your service carries two headers: +`X-NeMo-Gym-Resources-Server-Url` and `X-NeMo-Gym-Session-Cookie`. Echo the cookie on every +tool call you make against that URL and stateful environments work end to end. If your service +runs on a different machine, set `advertised_resources_url` to an externally reachable URL — +the default resolves to the resources server's bind address, which is typically a loopback +address only valid on the Gym host. Without +forwarding, tasks that declare tools are refused up front (instead of silently scoring 0) +unless you set `assume_remote_tools: true` because your service implements the declared tools +itself. + +## Run + +```bash +gym env start --resources-server # plus this agent's config +gym eval run --no-serve +agent_name=remote_agent +input_jsonl_fpath=... +output_jsonl_fpath=... +``` diff --git a/responses_api_agents/remote_agent/__init__.py b/responses_api_agents/remote_agent/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/responses_api_agents/remote_agent/app.py b/responses_api_agents/remote_agent/app.py new file mode 100644 index 0000000000..4a77dbda48 --- /dev/null +++ b/responses_api_agents/remote_agent/app.py @@ -0,0 +1,427 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Thin agent server that brokers rollouts to a user-hosted remote agent service. + +The remote service implements ONE endpoint: ``POST {agent_base_url}/v1/responses``. +It receives the row's ``responses_create_params`` (never ``verifier_metadata`` — the +answer key stays inside Gym), runs its own agent loop with its own model and tools, +and returns a single finished Responses API trajectory. This server owns the Gym +side of the rollout: it seeds the session, holds the session cookies, verifies on +the resources server, and returns the verify response from ``/run``. + +Failures never raise out of ``/run``: every failure (remote endpoint down, timeout, +malformed reply, seed/verify errors) becomes a reward-0 verify response carrying the +``_ng_failure_class`` sentinel, which rollout collection routes to the failures +sidecar and retries on resume. +""" + +import asyncio +from typing import Any, Dict, Optional, Tuple +from urllib.parse import urlparse + +import orjson +from aiohttp import ClientOSError, ClientTimeout, ServerDisconnectedError +from fastapi import Body, Request +from pydantic import ConfigDict, PrivateAttr, field_validator +from pydantic import ValidationError as PydanticValidationError + +from nemo_gym.base_resources_server import ( + AggregateMetrics, + AggregateMetricsRequest, + BaseRunRequest, + BaseVerifyResponse, +) +from nemo_gym.base_responses_api_agent import BaseResponsesAPIAgentConfig, SimpleResponsesAPIAgent +from nemo_gym.config_types import ResourcesServerRef +from nemo_gym.global_config import SKILLS_REF_KEY_NAME +from nemo_gym.openai_utils import NeMoGymResponse +from nemo_gym.rollout_collection import NG_FAILURE_CLASS_KEY, NG_NO_PERSIST_KEY, NG_TERMINAL_KEY +from nemo_gym.server_utils import get_global_aiohttp_client, get_response_json, get_server_url, raise_for_status + + +REMOTE_AGENT_FAILURE_CLASS = "remote_agent_error" + +# Header names of the optional session-forwarding contract (forward_session=true): the +# remote service echoes the cookie on every resources-server tool call it makes. +RESOURCES_URL_HEADER = "X-NeMo-Gym-Resources-Server-Url" +SESSION_COOKIE_HEADER = "X-NeMo-Gym-Session-Cookie" + +_REMOTE_MAX_TRIES = 3 +_REMOTE_RETRY_SLEEP_SECS = 0.5 +_FAILURE_PRINT_HEAD = 5 +_FAILURE_PRINT_INTERVAL = 100 +_AGGREGATE_PROXY_TIMEOUT_SECS = 600.0 + +# Result/routing keys this server itself produces. Input rows may carry stale copies +# (e.g. a rollouts or failures JSONL re-fed as a dataset); they must never collide with +# the fresh values or leak through the verify echo into the dispatcher's routing. +_RESERVED_RESULT_KEYS = ("reward", "response", "error", NG_FAILURE_CLASS_KEY, NG_NO_PERSIST_KEY, NG_TERMINAL_KEY) + + +def normalize_remote_url(url: str) -> str: + """Validate the remote service URL and strip any trailing slash.""" + normalized = url.strip().rstrip("/") + parsed = urlparse(normalized) + if parsed.scheme not in ("http", "https") or not parsed.netloc: + raise ValueError(f"agent_base_url must be an absolute http:// or https:// URL, got {url!r}") + # "/v1/responses" is string-appended; anything after "?" or "#" would swallow it (bare + # delimiters parse as an empty query/fragment, so check the string itself). + if "?" in normalized or "#" in normalized or parsed.params: + raise ValueError( + f"agent_base_url must not carry a query string or fragment, got {url!r}. " + "Pass auth material via your service's own configuration instead." + ) + # Credentials would be stamped into logged configs and error messages; never echo the URL. + if parsed.username or parsed.password: + raise ValueError( + "agent_base_url must not embed credentials (user:pass@host). " + "Pass auth material via your service's own configuration instead." + ) + return normalized + + +def cookie_header_value(cookies: Any) -> Optional[str]: + """Serialize seed-session cookies (SimpleCookie morsels or a plain dict) into a Cookie header.""" + if not cookies: + return None + pairs = [] + for key, value in cookies.items(): + pairs.append(f"{key}={getattr(value, 'value', value)}") + return "; ".join(pairs) if pairs else None + + +class RemoteAgentConfig(BaseResponsesAPIAgentConfig): + agent_base_url: str + resources_server: ResourcesServerRef + concurrency: int = 32 + remote_responses_timeout_secs: float = 1800.0 + # Bound on the whole /run body (seed + remote call + verify), applied after the + # semaphore is acquired so queue wait does not count against it. The collector's + # named-agent hop carries no timeout of its own; this is the only wallclock bound. + run_timeout_secs: float = 2100.0 + # Forward the resources-server URL and session cookie to the remote service so its + # loop can call Gym-hosted tools (it must echo the cookie on every tool call). + forward_session: bool = False + # The dataset declares tools but the remote service implements them itself; skip the + # declared-tools guard without forwarding the session. + assume_remote_tools: bool = False + # The resources-server URL advertised to the remote service with forward_session. The + # default (resolved from the global config) is the BIND address — typically 127.0.0.1, + # unreachable from another machine. Set this to the externally reachable URL when the + # remote service runs off-host. + advertised_resources_url: Optional[str] = None + + @field_validator("agent_base_url") + @classmethod + def _normalize_agent_base_url(cls, value: str) -> str: + return normalize_remote_url(value) + + @field_validator("advertised_resources_url") + @classmethod + def _normalize_advertised_resources_url(cls, value: Optional[str]) -> Optional[str]: + return normalize_remote_url(value) if value else value + + +class RemoteAgentRunRequest(BaseRunRequest): + model_config = ConfigDict(extra="allow") + + +class RemoteAgentVerifyResponse(BaseVerifyResponse): + model_config = ConfigDict(extra="allow") + + +class RemoteAgent(SimpleResponsesAPIAgent): + config: RemoteAgentConfig + sem: Optional[asyncio.Semaphore] = None + _num_failures: int = PrivateAttr(default=0) + _warn_counts: Dict[str, int] = PrivateAttr(default_factory=dict) + model_config = ConfigDict(arbitrary_types_allowed=True) + + def model_post_init(self, __context: Any) -> None: + self.sem = asyncio.Semaphore(self.config.concurrency) + + async def responses(self, body=Body()) -> NeMoGymResponse: + raise NotImplementedError( + "RemoteAgent brokers a remote service; drive it through /run. The remote service's " + "own /v1/responses is called by run(), not exposed here." + ) + + async def run(self, request: Request, body: RemoteAgentRunRequest = Body()) -> RemoteAgentVerifyResponse: + record = self._sanitized_record(body) + async with self.sem: + try: + return await asyncio.wait_for( + self._run_once(request, body, record), timeout=self.config.run_timeout_secs + ) + except asyncio.TimeoutError: + return self._failure_response( + record, + f"/run exceeded run_timeout_secs={self.config.run_timeout_secs}s " + "(seed + remote /v1/responses + verify)", + ) + except Exception as e: # noqa: BLE001 -- never 500; one task must not abort the whole collection + return self._failure_response(record, f"unexpected error: {type(e).__name__}: {e}") + + def _sanitized_record(self, body: RemoteAgentRunRequest) -> Dict[str, Any]: + record = body.model_dump() + for key in _RESERVED_RESULT_KEYS: + record.pop(key, None) + return record + + async def _run_once( + self, request: Request, body: RemoteAgentRunRequest, record: Dict[str, Any] + ) -> RemoteAgentVerifyResponse: + guard_error = self._tools_guard_error(record) + if guard_error: + return self._failure_response(record, guard_error, terminal=True) + + if record.get(SKILLS_REF_KEY_NAME): + self._throttled_warn( + "skills_ref", + "WARNING: this run carries a skills_ref, but RemoteAgent cannot stage skills into a " + "remote service; the skills config is ignored.", + ) + + # 1. Seed the session; the cookies key all per-session state on the resources server. + cookies = request.cookies + try: + seed_response = await self.server_client.post( + server_name=self.config.resources_server.name, + url_path="/seed_session", + json=record, + cookies=cookies, + ) + await raise_for_status(seed_response) + cookies = seed_response.cookies + except Exception as e: + return self._failure_response( + record, f"/seed_session on the resources server failed: {type(e).__name__}: {e}" + ) + + # 2. One POST to the remote service: create-params in, finished trajectory out. + # exclude_unset keeps the wire payload to exactly what the dataset row carried. + remote_params = body.responses_create_params.model_dump(exclude_unset=True) + remote_result, remote_error, terminal = await self._post_remote_responses(remote_params, cookies) + if remote_error is not None: + return self._failure_response(record, remote_error, terminal=terminal) + + try: + response = NeMoGymResponse.model_validate(remote_result) + except PydanticValidationError as e: + # A shape error will not fix itself on retry. + return self._failure_response( + record, + f"remote service returned an invalid Responses API object: {str(e)[:500]}", + terminal=True, + ) + self._warn_on_response_quality(response) + + # 3. Verify on the SAME session; the verify response (reward included) is /run's result. + try: + verify_response = await self.server_client.post( + server_name=self.config.resources_server.name, + url_path="/verify", + json=record | {"response": response.model_dump(mode="json")}, + cookies=cookies, + ) + await raise_for_status(verify_response) + verify_json = await get_response_json(verify_response) + except Exception as e: + return self._failure_response(record, f"/verify on the resources server failed: {type(e).__name__}: {e}") + + return RemoteAgentVerifyResponse.model_validate(verify_json) + + def _tools_guard_error(self, record: Dict[str, Any]) -> Optional[str]: + """Refuse tool-declaring tasks the remote service cannot serve, instead of scoring silent zeros. + + A dataset that declares tools expects them to be called during the rollout. Without + forward_session the remote service has no session cookie, so any per-session state + those tools mutate stays untouched and verify() scores 0 on every row. + """ + declared_tools = (record.get("responses_create_params") or {}).get("tools") + if declared_tools and not (self.config.forward_session or self.config.assume_remote_tools): + return ( + "the task declares tools but forward_session=false and assume_remote_tools=false. " + "Set forward_session=true so the remote service can call Gym-hosted tools with the " + "session cookie, or assume_remote_tools=true if the service implements the declared " + "tools itself." + ) + return None + + async def _post_remote_responses( + self, remote_params: Dict[str, Any], cookies: Any + ) -> Tuple[Optional[Dict], Optional[str], bool]: + """POST create-params to the remote /v1/responses. Returns (result, error, terminal).""" + remote_url = f"{self.config.agent_base_url}/v1/responses" + client = get_global_aiohttp_client() + data = orjson.dumps(remote_params) + headers = {"Content-Type": "application/json"} + if self.config.forward_session: + resources_url = self.config.advertised_resources_url or get_server_url(self.config.resources_server.name) + advertised_host = urlparse(resources_url).hostname or "" + remote_host = urlparse(self.config.agent_base_url).hostname or "" + if advertised_host in ("127.0.0.1", "localhost") and remote_host not in ("127.0.0.1", "localhost"): + self._throttled_warn( + "loopback_resources_url", + f"WARNING: forwarding resources-server URL {resources_url} (a loopback address) to the " + f"off-host remote service at {self.config.agent_base_url}. Its tool calls will not reach " + "Gym; set advertised_resources_url to an externally reachable URL.", + ) + headers[RESOURCES_URL_HEADER] = resources_url + session_cookie = cookie_header_value(cookies) + if session_cookie: + headers[SESSION_COOKIE_HEADER] = session_cookie + timeout = ClientTimeout(total=self.config.remote_responses_timeout_secs) + + response = None + last_connect_error: Optional[BaseException] = None + for num_try in range(1, _REMOTE_MAX_TRIES + 1): + try: + # aiohttp follows a redirected POST as a body-less GET; fail with the 3xx instead. + response = await client.request( + "POST", remote_url, data=data, headers=headers, timeout=timeout, allow_redirects=False + ) + break + except (ClientOSError, ServerDisconnectedError) as e: + # Refused/reset (ClientOSError) and keepalive races (ServerDisconnectedError) + # are transient connection noise; everything else fails fast. + last_connect_error = e + if num_try < _REMOTE_MAX_TRIES: + await asyncio.sleep(_REMOTE_RETRY_SLEEP_SECS) + except asyncio.TimeoutError: + return ( + None, + f"remote /v1/responses timed out after {self.config.remote_responses_timeout_secs}s " + "(remote_responses_timeout_secs; raise it if rollouts legitimately run longer)", + False, + ) + except Exception as e: + return None, f"{type(e).__name__}: {e}", False + if response is None: + return ( + None, + f"could not reach the remote service after {_REMOTE_MAX_TRIES} tries " + f"({type(last_connect_error).__name__}: {last_connect_error}). " + f"Is your service running at {self.config.agent_base_url}?", + False, + ) + + # client.request() returns once headers arrive; the body read can still raise + # (mid-body disconnect, deadline) and must honor the same never-raise contract. + try: + content = await response.read() + except Exception as e: + return None, f"reading the response body failed: {type(e).__name__}: {e}", False + # response.ok is `status < 400`; reject 3xx explicitly (redirects are not followed). + if not response.ok or response.status >= 300: + location = response.headers.get("Location", "") + return ( + None, + f"HTTP {response.status}" + + (f" (redirect to {location}; fix agent_base_url to point at the final address)" if location else "") + + f": {content[:500].decode(errors='replace')}", + False, + ) + try: + result = orjson.loads(content) + except orjson.JSONDecodeError as e: + return None, f"response is not valid JSON: {e}", False + if not isinstance(result, dict): + return None, f"expected a JSON object from /v1/responses, got {type(result).__name__}", False + return result, None, False + + def _throttled_warn(self, key: str, message: str) -> None: + """Per-key sampled warning: the first few occurrences, then every 100th. At production + concurrency an unthrottled per-rollout print garbles the collector's progress bar.""" + n = self._warn_counts.get(key, 0) + 1 + self._warn_counts[key] = n + if n <= _FAILURE_PRINT_HEAD or n % _FAILURE_PRINT_INTERVAL == 0: + print(f"{message} (occurrence #{n})", flush=True) + + def _warn_on_response_quality(self, response: NeMoGymResponse) -> None: + last = response.output[-1] if response.output else None + finished_naturally = getattr(last, "type", None) == "message" and getattr(last, "role", None) == "assistant" + if not finished_naturally: + self._throttled_warn( + "non_terminal_trajectory", + "WARNING: the remote trajectory does not end with an assistant message; the contract " + "is one FINISHED trajectory per call (no dangling tool calls).", + ) + if response.usage is None: + self._throttled_warn( + "missing_usage", + "WARNING: the remote response carries no usage; token metrics for this agent will be " + "empty. Have your service report usage {input_tokens, output_tokens, total_tokens}.", + ) + + def _failure_response( + self, record: Dict[str, Any], error: str, terminal: bool = False + ) -> RemoteAgentVerifyResponse: + self._num_failures += 1 + n = self._num_failures + if n <= _FAILURE_PRINT_HEAD or n % _FAILURE_PRINT_INTERVAL == 0: + print(f"[remote_agent] rollout failed (failure #{n}): {error}", flush=True) + routing: Dict[str, Any] = {NG_FAILURE_CLASS_KEY: REMOTE_AGENT_FAILURE_CLASS, "error": error} + if terminal: + routing[NG_TERMINAL_KEY] = True + # Dict-merge with later keys winning: `record` is sanitized of reserved keys, but merge + # order still guarantees fresh reward/response/routing even if a caller passes a raw dump. + return RemoteAgentVerifyResponse.model_validate( + record | {"reward": 0.0, "response": self._empty_response().model_dump(mode="json")} | routing + ) + + def _empty_response(self) -> NeMoGymResponse: + """Minimal valid response for the failure path, so /run can return 200 with reward 0 + (never 500) even when the remote service produced nothing.""" + return NeMoGymResponse( + id="remote_agent_failure", + created_at=0.0, + model="remote_agent", + object="response", + output=[ + { + "type": "message", + "role": "assistant", + "status": "completed", + "id": "msg_0", + "content": [{"type": "output_text", "text": "", "annotations": []}], + } + ], + parallel_tool_calls=False, + tools=[], + tool_choice="auto", + ) + + async def aggregate_metrics(self, body: AggregateMetricsRequest = Body()) -> AggregateMetrics: + """Proxy aggregate_metrics to the resources server. + + Bounded: the ServerClient hop otherwise retries connection errors forever, and a dead + resources server at end-of-run would hang the collector after all rollouts are on disk. + """ + + async def _proxy() -> AggregateMetrics: + response = await self.server_client.post( + server_name=self.config.resources_server.name, + url_path="/aggregate_metrics", + json=body, + ) + await raise_for_status(response) + return AggregateMetrics.model_validate(await get_response_json(response)) + + return await asyncio.wait_for(_proxy(), timeout=_AGGREGATE_PROXY_TIMEOUT_SECS) + + +if __name__ == "__main__": + RemoteAgent.run_webserver() diff --git a/responses_api_agents/remote_agent/configs/remote_agent.yaml b/responses_api_agents/remote_agent/configs/remote_agent.yaml new file mode 100644 index 0000000000..ebbea879c8 --- /dev/null +++ b/responses_api_agents/remote_agent/configs/remote_agent.yaml @@ -0,0 +1,13 @@ +remote_agent: + responses_api_agents: + remote_agent: + entrypoint: app.py + agent_base_url: ??? + resources_server: + type: resources_servers + name: ??? + concurrency: 32 + remote_responses_timeout_secs: 1800.0 + run_timeout_secs: 2100.0 + forward_session: false + assume_remote_tools: false diff --git a/responses_api_agents/remote_agent/requirements.txt b/responses_api_agents/remote_agent/requirements.txt new file mode 100644 index 0000000000..00ed83213e --- /dev/null +++ b/responses_api_agents/remote_agent/requirements.txt @@ -0,0 +1 @@ +-e nemo-gym[dev] @ ../../ diff --git a/responses_api_agents/remote_agent/tests/__init__.py b/responses_api_agents/remote_agent/tests/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/responses_api_agents/remote_agent/tests/test_app.py b/responses_api_agents/remote_agent/tests/test_app.py new file mode 100644 index 0000000000..e5d4fb28a4 --- /dev/null +++ b/responses_api_agents/remote_agent/tests/test_app.py @@ -0,0 +1,866 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import asyncio +import json +from unittest.mock import AsyncMock, MagicMock + +import orjson +import pytest +from aiohttp import ClientConnectorError, ClientPayloadError, ServerDisconnectedError +from pydantic import ValidationError + +import responses_api_agents.remote_agent.app as remote_agent_app +from nemo_gym.config_types import ResourcesServerRef +from nemo_gym.rollout_collection import NG_FAILURE_CLASS_KEY, NG_NO_PERSIST_KEY, NG_TERMINAL_KEY +from nemo_gym.server_utils import ServerClient +from responses_api_agents.remote_agent.app import ( + REMOTE_AGENT_FAILURE_CLASS, + RESOURCES_URL_HEADER, + SESSION_COOKIE_HEADER, + RemoteAgent, + RemoteAgentConfig, + RemoteAgentRunRequest, + cookie_header_value, + normalize_remote_url, +) + + +_MINIMAL_TRAJECTORY = { + "id": "traj_1", + "created_at": 1.0, + "model": "their-model", + "object": "response", + "output": [ + { + "type": "message", + "role": "assistant", + "status": "completed", + "id": "msg_1", + "content": [{"type": "output_text", "text": "the answer is 42", "annotations": []}], + } + ], + "parallel_tool_calls": False, + "tools": [], + "tool_choice": "auto", + "usage": { + "input_tokens": 10, + "output_tokens": 5, + "total_tokens": 15, + "input_tokens_details": {"cached_tokens": 0}, + "output_tokens_details": {"reasoning_tokens": 0}, + }, +} + + +def make_config(**overrides) -> RemoteAgentConfig: + fields = dict( + host="0.0.0.0", + port=8080, + entrypoint="", + name="remote_agent", + agent_base_url="http://localhost:9000", + resources_server=ResourcesServerRef(type="resources_servers", name="my_env"), + ) + fields.update(overrides) + return RemoteAgentConfig(**fields) + + +def make_agent(server_client=None, **config_overrides) -> RemoteAgent: + return RemoteAgent( + config=make_config(**config_overrides), + server_client=server_client or MagicMock(spec=ServerClient), + ) + + +def make_row(tools=None, **extras) -> dict: + row = { + "responses_create_params": {"input": [{"role": "user", "content": "what is 6 x 7?"}]}, + "verifier_metadata": {"expected_answer": "42"}, + } + if tools is not None: + row["responses_create_params"]["tools"] = tools + row.update(extras) + return row + + +def make_request(cookies=None) -> MagicMock: + request = MagicMock() + request.cookies = cookies or {} + return request + + +class FakeRemoteResponse: + """Stands in for an aiohttp ClientResponse from the remote service.""" + + def __init__(self, status: int, content: bytes, headers=None, read_exc=None): + self.status = status + self._content = content + self.headers = headers or {} + self._read_exc = read_exc + + @property + def ok(self) -> bool: + return self.status < 400 + + async def read(self) -> bytes: + if self._read_exc is not None: + raise self._read_exc + return self._content + + +class FakeServerClientResponse: + """Stands in for an aiohttp ClientResponse from a Gym server via ServerClient.""" + + def __init__(self, body: dict, cookies=None, status: int = 200): + self._body = body + self.cookies = cookies or {} + self.status = status + + @property + def ok(self) -> bool: + return self.status < 400 + + @property + def content(self): + reader = MagicMock() + + async def _read(): + return orjson.dumps(self._body) + + reader.read = _read + return reader + + def raise_for_status(self): + if not self.ok: + raise RuntimeError(f"HTTP {self.status}") + + async def read(self) -> bytes: + return orjson.dumps(self._body) + + +def mock_remote(monkeypatch: pytest.MonkeyPatch, request_mock: AsyncMock) -> MagicMock: + client = MagicMock() + client.request = request_mock + monkeypatch.setattr(remote_agent_app, "get_global_aiohttp_client", lambda: client) + monkeypatch.setattr(remote_agent_app, "_REMOTE_RETRY_SLEEP_SECS", 0) + return client + + +def seed_verify_server_client(verify_body=None, seed_cookies=None, seed_status=200, verify_status=200): + """A ServerClient mock that answers /seed_session and /verify.""" + calls = [] + + async def _post(server_name, url_path, json=None, cookies=None, **kwargs): + calls.append({"server_name": server_name, "url_path": url_path, "json": json, "cookies": cookies}) + if url_path == "/seed_session": + return FakeServerClientResponse({}, cookies=seed_cookies or {"session": "abc123"}, status=seed_status) + if url_path == "/verify": + body = verify_body if verify_body is not None else (json | {"reward": 1.0}) + return FakeServerClientResponse(body, status=verify_status) + return FakeServerClientResponse({}, status=200) + + server_client = MagicMock(spec=ServerClient) + server_client.post = AsyncMock(side_effect=_post) + server_client.calls = calls + return server_client + + +class TestConfig: + def test_sanity_construct_and_semaphore(self) -> None: + agent = make_agent(concurrency=7) + assert agent.sem._value == 7 + + def test_agent_base_url_normalized(self) -> None: + assert make_config(agent_base_url="http://localhost:9000/").agent_base_url == "http://localhost:9000" + + @pytest.mark.parametrize( + "bad_url", + ["ftp://h:1", "localhost:9000", "http://h:1?token=abc", "http://h:1#frag", "http://user:pass@h:1"], + ) + def test_agent_base_url_rejected(self, bad_url: str) -> None: + with pytest.raises(ValidationError): + make_config(agent_base_url=bad_url) + + def test_normalize_remote_url_never_echoes_credentials(self) -> None: + with pytest.raises(ValueError) as exc_info: + normalize_remote_url("http://user:hunter2@h:1") + assert "hunter2" not in str(exc_info.value) + + def test_cookie_header_value_shapes(self) -> None: + assert cookie_header_value({}) is None + assert cookie_header_value({"a": "1", "b": "2"}) == "a=1; b=2" + morsel = MagicMock() + morsel.value = "xyz" + assert cookie_header_value({"session": morsel}) == "session=xyz" + + +class TestRunHappyPath: + async def test_seed_then_remote_then_verify(self, monkeypatch: pytest.MonkeyPatch) -> None: + request_mock = AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))) + client = mock_remote(monkeypatch, request_mock) + server_client = seed_verify_server_client(seed_cookies={"session": "s1"}) + agent = make_agent(server_client=server_client) + + row = make_row() + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row)) + + # Order and payloads: seed first, remote POST in between, verify last on the seed cookies + assert [c["url_path"] for c in server_client.calls] == ["/seed_session", "/verify"] + assert server_client.calls[1]["cookies"] == {"session": "s1"} + assert server_client.calls[1]["json"]["response"]["id"] == "traj_1" + assert server_client.calls[1]["json"]["verifier_metadata"] == {"expected_answer": "42"} + + args, kwargs = client.request.call_args + assert args == ("POST", "http://localhost:9000/v1/responses") + # The remote service receives ONLY create-params: no verifier_metadata, no row keys + remote_payload = orjson.loads(kwargs["data"]) + assert remote_payload == row["responses_create_params"] + assert kwargs["allow_redirects"] is False + assert kwargs["timeout"].total == 1800.0 + # No session forwarding by default + assert RESOURCES_URL_HEADER not in kwargs["headers"] + assert SESSION_COOKIE_HEADER not in kwargs["headers"] + + dumped = result.model_dump() + assert dumped["reward"] == 1.0 + assert NG_FAILURE_CLASS_KEY not in dumped + assert NG_NO_PERSIST_KEY not in dumped + assert NG_TERMINAL_KEY not in dumped + + async def test_verify_extras_pass_through(self, monkeypatch: pytest.MonkeyPatch) -> None: + mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)))) + row = make_row() + verify_body = row | {"response": _MINIMAL_TRAJECTORY, "reward": 0.5, "grading_notes": "close enough"} + agent = make_agent(server_client=seed_verify_server_client(verify_body=verify_body)) + + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row)) + + assert result.model_dump()["grading_notes"] == "close enough" + assert result.reward == 0.5 + + +class TestRemoteFailuresBecomeSentinelRows: + async def _run(self, monkeypatch, request_mock, **config_overrides): + client = mock_remote(monkeypatch, request_mock) + server_client = seed_verify_server_client() + agent = make_agent(server_client=server_client, **config_overrides) + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) + return client, server_client, result.model_dump() + + async def test_timeout_fails_once_without_retry(self, monkeypatch: pytest.MonkeyPatch) -> None: + client, _, result = await self._run(monkeypatch, AsyncMock(side_effect=asyncio.TimeoutError())) + assert client.request.call_count == 1 + assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + assert NG_TERMINAL_KEY not in result + assert "timed out after 1800.0s" in result["error"] + assert result["reward"] == 0.0 + + async def test_connect_exhaustion_after_bounded_retries(self, monkeypatch: pytest.MonkeyPatch) -> None: + connect_error = ClientConnectorError(MagicMock(), OSError("connection refused")) + client, server_client, result = await self._run(monkeypatch, AsyncMock(side_effect=connect_error)) + assert client.request.call_count == 3 + assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + assert "Is your service running at http://localhost:9000?" in result["error"] + # verify is never reached on a failed remote call + assert [c["url_path"] for c in server_client.calls] == ["/seed_session"] + + async def test_disconnect_then_success_retries(self, monkeypatch: pytest.MonkeyPatch) -> None: + request_mock = AsyncMock( + side_effect=[ServerDisconnectedError(), FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))] + ) + client, _, result = await self._run(monkeypatch, request_mock) + assert client.request.call_count == 2 + assert NG_FAILURE_CLASS_KEY not in result + + async def test_http_500_with_body_excerpt(self, monkeypatch: pytest.MonkeyPatch) -> None: + _, _, result = await self._run(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(500, b"kaboom"))) + assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + assert "HTTP 500" in result["error"] and "kaboom" in result["error"] + + async def test_redirect_rejected_with_location_hint(self, monkeypatch: pytest.MonkeyPatch) -> None: + response = FakeRemoteResponse(301, b"", headers={"Location": "https://elsewhere"}) + _, _, result = await self._run(monkeypatch, AsyncMock(return_value=response)) + assert "HTTP 301" in result["error"] and "https://elsewhere" in result["error"] + + async def test_invalid_json_body(self, monkeypatch: pytest.MonkeyPatch) -> None: + _, _, result = await self._run(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, b"not json"))) + assert "not valid JSON" in result["error"] + + async def test_non_object_body(self, monkeypatch: pytest.MonkeyPatch) -> None: + _, _, result = await self._run(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, b"[1, 2]"))) + assert "expected a JSON object" in result["error"] + + @pytest.mark.parametrize( + "read_exc", + [ClientPayloadError("Response payload is not completed"), asyncio.TimeoutError()], + ids=["mid-body disconnect", "deadline during body read"], + ) + async def test_body_read_failure(self, monkeypatch: pytest.MonkeyPatch, read_exc: Exception) -> None: + response = FakeRemoteResponse(200, b"", read_exc=read_exc) + _, _, result = await self._run(monkeypatch, AsyncMock(return_value=response)) + assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + assert "reading the response body failed" in result["error"] + + async def test_unexpected_exception_fails_fast_without_retry(self, monkeypatch: pytest.MonkeyPatch) -> None: + client, _, result = await self._run(monkeypatch, AsyncMock(side_effect=RuntimeError("surprise"))) + assert client.request.call_count == 1 + assert "RuntimeError: surprise" in result["error"] + + async def test_invalid_trajectory_shape_is_terminal_and_skips_verify( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + bad = {"id": "x", "object": "response"} # missing required Responses API fields + client, server_client, result = await self._run( + monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(bad))) + ) + assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + assert result[NG_TERMINAL_KEY] is True + assert "invalid Responses API object" in result["error"] + assert [c["url_path"] for c in server_client.calls] == ["/seed_session"] + + +class TestGymSideFailuresBecomeSentinelRows: + async def test_seed_failure_skips_remote_call(self, monkeypatch: pytest.MonkeyPatch) -> None: + request_mock = AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))) + client = mock_remote(monkeypatch, request_mock) + server_client = seed_verify_server_client(seed_status=500) + agent = make_agent(server_client=server_client) + + result = (await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row()))).model_dump() + + assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + assert "/seed_session" in result["error"] + assert client.request.call_count == 0 + + async def test_verify_failure(self, monkeypatch: pytest.MonkeyPatch) -> None: + mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)))) + server_client = seed_verify_server_client(verify_status=500) + agent = make_agent(server_client=server_client) + + result = (await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row()))).model_dump() + + assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + assert "/verify" in result["error"] + assert result["reward"] == 0.0 + + +class TestToolsGuardAndSessionForwarding: + _TOOLS = [ + { + "type": "function", + "name": "increment_counter", + "parameters": { + "type": "object", + "properties": {"count": {"type": "integer", "description": ""}}, + "required": ["count"], + "additionalProperties": False, + }, + "strict": True, + "description": "", + } + ] + + async def test_declared_tools_refused_by_default(self, monkeypatch: pytest.MonkeyPatch) -> None: + client = mock_remote(monkeypatch, AsyncMock()) + server_client = seed_verify_server_client() + agent = make_agent(server_client=server_client) + + row = make_row(tools=self._TOOLS) + result = (await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row))).model_dump() + + assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + assert result[NG_TERMINAL_KEY] is True + assert "forward_session" in result["error"] and "assume_remote_tools" in result["error"] + # Refused before any network traffic + assert client.request.call_count == 0 + assert server_client.calls == [] + + async def test_assume_remote_tools_skips_guard_without_headers(self, monkeypatch: pytest.MonkeyPatch) -> None: + request_mock = AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))) + client = mock_remote(monkeypatch, request_mock) + agent = make_agent(server_client=seed_verify_server_client(), assume_remote_tools=True) + + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row(tools=self._TOOLS))) + + assert NG_FAILURE_CLASS_KEY not in result.model_dump() + headers = client.request.call_args.kwargs["headers"] + assert RESOURCES_URL_HEADER not in headers + + async def test_forward_session_sends_url_and_cookie_headers(self, monkeypatch: pytest.MonkeyPatch) -> None: + request_mock = AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))) + client = mock_remote(monkeypatch, request_mock) + monkeypatch.setattr(remote_agent_app, "get_server_url", lambda name: f"http://resolved-{name}:1234") + server_client = seed_verify_server_client(seed_cookies={"session": "cookie-value"}) + agent = make_agent(server_client=server_client, forward_session=True) + + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row(tools=self._TOOLS))) + + assert NG_FAILURE_CLASS_KEY not in result.model_dump() + headers = client.request.call_args.kwargs["headers"] + assert headers[RESOURCES_URL_HEADER] == "http://resolved-my_env:1234" + assert headers[SESSION_COOKIE_HEADER] == "session=cookie-value" + + async def test_skills_ref_warns_and_continues( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: + mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)))) + agent = make_agent(server_client=seed_verify_server_client()) + + row = make_row(skills_ref={"path": "/skills", "hash": "abc", "skills": []}) + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row)) + + assert NG_FAILURE_CLASS_KEY not in result.model_dump() + assert "skills_ref" in capsys.readouterr().out + + +class TestResponseQualityWarnings: + async def _run_with_trajectory(self, monkeypatch, trajectory): + mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(trajectory)))) + agent = make_agent(server_client=seed_verify_server_client()) + return await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) + + async def test_non_terminal_trajectory_warns( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: + trajectory = dict(_MINIMAL_TRAJECTORY) + trajectory["output"] = [ + { + "type": "function_call", + "status": "completed", + "id": "fc_1", + "call_id": "call_1", + "name": "increment_counter", + "arguments": "{}", + } + ] + result = await self._run_with_trajectory(monkeypatch, trajectory) + assert NG_FAILURE_CLASS_KEY not in result.model_dump() + assert "does not end with an assistant message" in capsys.readouterr().out + + async def test_missing_usage_warns( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: + trajectory = dict(_MINIMAL_TRAJECTORY) + trajectory.pop("usage") + result = await self._run_with_trajectory(monkeypatch, trajectory) + assert NG_FAILURE_CLASS_KEY not in result.model_dump() + assert "no usage" in capsys.readouterr().out + + async def test_clean_trajectory_no_warnings( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: + await self._run_with_trajectory(monkeypatch, _MINIMAL_TRAJECTORY) + out = capsys.readouterr().out + assert "WARNING" not in out + + +class TestRunTimeoutAndSemaphore: + async def test_run_wallclock_bound_becomes_sentinel(self, monkeypatch: pytest.MonkeyPatch) -> None: + async def slow_request(*args, **kwargs): + await asyncio.sleep(30) + + mock_remote(monkeypatch, AsyncMock(side_effect=slow_request)) + agent = make_agent(server_client=seed_verify_server_client(), run_timeout_secs=0.05) + + result = (await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row()))).model_dump() + + assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + assert "run_timeout_secs" in result["error"] + + async def test_semaphore_bounds_in_flight_and_releases_on_failure(self, monkeypatch: pytest.MonkeyPatch) -> None: + in_flight = 0 + max_in_flight = 0 + release = asyncio.Event() + + async def gated_request(*args, **kwargs): + nonlocal in_flight, max_in_flight + in_flight += 1 + max_in_flight = max(max_in_flight, in_flight) + await release.wait() + in_flight -= 1 + return FakeRemoteResponse(500, b"boom") # failure path must release the permit too + + mock_remote(monkeypatch, AsyncMock(side_effect=gated_request)) + agent = make_agent(server_client=seed_verify_server_client(), concurrency=2) + + rows = [RemoteAgentRunRequest.model_validate(make_row()) for _ in range(4)] + tasks = [asyncio.create_task(agent.run(make_request(), row)) for row in rows] + await asyncio.sleep(0.05) + assert max_in_flight == 2 + release.set() + results = await asyncio.gather(*tasks) + + assert all(r.model_dump()[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS for r in results) + assert agent.sem._value == 2 # every permit released despite 4 failures + + +class TestRoutes: + def _client_and_mocks(self, monkeypatch, request_mock=None): + from fastapi.testclient import TestClient + + mock_remote( + monkeypatch, + request_mock or AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))), + ) + agent = make_agent(server_client=seed_verify_server_client()) + return TestClient(agent.setup_webserver(), raise_server_exceptions=False) + + def test_run_route_happy_path(self, monkeypatch: pytest.MonkeyPatch) -> None: + client = self._client_and_mocks(monkeypatch) + response = client.post("/run", json=make_row()) + assert response.status_code == 200 + assert response.json()["reward"] == 1.0 + + def test_run_route_failure_serializes_sentinel_with_http_200(self, monkeypatch: pytest.MonkeyPatch) -> None: + # The sentinel body must survive FastAPI response-model serialization: a 500 here + # would abort the entire collection run instead of routing to the failures sidecar. + client = self._client_and_mocks(monkeypatch, AsyncMock(side_effect=RuntimeError("remote exploded"))) + response = client.post("/run", json=make_row()) + assert response.status_code == 200 + body = response.json() + assert body[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + assert body["reward"] == 0.0 + assert body["response"]["output"][0]["type"] == "message" + + async def test_responses_not_implemented(self) -> None: + agent = make_agent() + with pytest.raises(NotImplementedError): + await agent.responses(body={}) + + +class TestStatefulToolsEndToEnd: + """The full session contract, in-process: RemoteAgent seeds the counter environment, + forwards the session to a fake remote service, the service calls the counter tools with + the forwarded cookie, and verify() scores the mutated session state.""" + + def _counter_client(self): + from fastapi.testclient import TestClient + + from resources_servers.example_session_state_mgmt.app import ( + StatefulCounterResourcesServer, + StatefulCounterResourcesServerConfig, + ) + + config = StatefulCounterResourcesServerConfig( + host="0.0.0.0", port=8081, entrypoint="", name="counter", domain="agent" + ) + server = StatefulCounterResourcesServer(config=config, server_client=MagicMock(spec=ServerClient)) + return TestClient(server.setup_webserver()) + + async def test_counter_env_reward_through_forwarded_session(self, monkeypatch: pytest.MonkeyPatch) -> None: + counter = self._counter_client() + + async def gym_post(server_name, url_path, json=None, cookies=None, **kwargs): + response = counter.post(url_path, json=json, cookies=dict(cookies or {})) + return FakeServerClientResponse( + response.json(), cookies=dict(response.cookies), status=response.status_code + ) + + server_client = MagicMock(spec=ServerClient) + server_client.post = AsyncMock(side_effect=gym_post) + + async def remote_service(method, url, data=None, headers=None, **kwargs): + # The remote service reads the forwarded session and calls the counter tools + # with the cookie echoed on every call — the contract under test. + cookie_pair = headers[SESSION_COOKIE_HEADER] + cookie_name, cookie_value = cookie_pair.split("=", 1) + tool_cookies = {cookie_name: cookie_value} + assert headers[RESOURCES_URL_HEADER].startswith("http://") + + assert counter.post("/increment_counter", json={"count": 1}, cookies=tool_cookies).status_code == 200 + assert counter.post("/increment_counter", json={"count": 2}, cookies=tool_cookies).status_code == 200 + count = counter.post("/get_counter_value", json={}, cookies=tool_cookies).json()["count"] + + trajectory = dict(_MINIMAL_TRAJECTORY) + trajectory["output"] = [ + { + "type": "message", + "role": "assistant", + "status": "completed", + "id": "msg_1", + "content": [{"type": "output_text", "text": f"final count is {count}", "annotations": []}], + } + ] + return FakeRemoteResponse(200, orjson.dumps(trajectory)) + + mock_remote(monkeypatch, AsyncMock(side_effect=remote_service)) + monkeypatch.setattr(remote_agent_app, "get_server_url", lambda name: "http://counter-in-process") + + agent = make_agent(server_client=server_client, forward_session=True) + row = { + "responses_create_params": { + "input": [{"role": "user", "content": "add 1 then add 2 then get the count"}], + "tools": [ + { + "type": "function", + "name": "increment_counter", + "parameters": { + "type": "object", + "properties": {"count": {"type": "integer", "description": ""}}, + "required": ["count"], + "additionalProperties": False, + }, + "strict": True, + "description": "", + } + ], + }, + "initial_count": 3, + "expected_count": 6, + } + + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row)) + + dumped = result.model_dump() + assert NG_FAILURE_CLASS_KEY not in dumped + # Reward 1.0 only if seed, both tool calls, and verify all shared ONE session + assert dumped["reward"] == 1.0 + assert "final count is 6" in dumped["response"]["output"][0]["content"][0]["text"] + + +class TestCollectorRoundTrip: + """Drive the real rollout-collection helper against this agent in-process and assert + the sidecar contract end to end: successes to the main jsonl, sentinel rows to the + failures sidecar.""" + + async def test_success_and_failure_routing(self, monkeypatch: pytest.MonkeyPatch, tmp_path) -> None: + from fastapi.testclient import TestClient + + import nemo_gym.rollout_collection + from nemo_gym.rollout_collection import RolloutCollectionConfig, RolloutCollectionHelper + + # The collector reads the global config for model-call capture dirs; neutralize the + # Hydra CLI parse it would otherwise attempt under pytest (same as the core tests). + monkeypatch.setattr(nemo_gym.rollout_collection, "get_global_config_dict", MagicMock(return_value={})) + + async def remote_service(method, url, data=None, headers=None, **kwargs): + params = orjson.loads(data) + if "fail" in params["input"][0]["content"]: + return FakeRemoteResponse(500, b"remote exploded") + return FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)) + + mock_remote(monkeypatch, AsyncMock(side_effect=remote_service)) + agent = make_agent(server_client=seed_verify_server_client()) + agent_http = TestClient(agent.setup_webserver(), raise_server_exceptions=False) + + class InProcessHelper(RolloutCollectionHelper): + def setup_server_client(self, *args, **kwargs): + async def _post(server_name, url_path, json=None, **kw): + response = agent_http.post(url_path, json=json) + return FakeServerClientResponse(response.json(), status=response.status_code) + + server_client = MagicMock(spec=ServerClient) + server_client.post = AsyncMock(side_effect=_post) + return server_client + + async def _call_aggregate_metrics(self, results, rows, output_fpath): + return None + + input_fpath = tmp_path / "input.jsonl" + rows = [ + {"responses_create_params": {"input": [{"role": "user", "content": "please succeed"}]}}, + {"responses_create_params": {"input": [{"role": "user", "content": "please fail"}]}}, + ] + input_fpath.write_text("\n".join(json.dumps(r) for r in rows) + "\n") + + config = RolloutCollectionConfig( + input_jsonl_fpath=str(input_fpath), + output_jsonl_fpath=str(tmp_path / "rollouts.jsonl"), + agent_name="remote_agent", + upload_rollouts_to_wandb=False, + ) + await InProcessHelper().run_from_config(config) + + main_rows = [json.loads(line) for line in (tmp_path / "rollouts.jsonl").open()] + assert len(main_rows) == 1 + assert main_rows[0]["reward"] == 1.0 + + sidecar_rows = [json.loads(line) for line in (tmp_path / "rollouts_failures.jsonl").open()] + assert len(sidecar_rows) == 1 + assert sidecar_rows[0][NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + assert "HTTP 500" in sidecar_rows[0]["error"] + + +class TestReviewFindingPins: + """Regression pins for the adversarial-review findings.""" + + _REUSED_ROW_EXTRAS = { + "reward": 0.75, + "response": {"stale": True}, + "error": "stale error", + NG_FAILURE_CLASS_KEY: "stale_class", + NG_NO_PERSIST_KEY: True, + NG_TERMINAL_KEY: True, + } + + async def test_failure_on_reused_rollout_row_still_returns_sentinel(self, monkeypatch: pytest.MonkeyPatch) -> None: + # A rollouts/failures JSONL re-fed as a dataset carries reward/response/error and stale + # routing keys; the failure path must not TypeError on them (the never-raise contract). + mock_remote(monkeypatch, AsyncMock(side_effect=RuntimeError("remote exploded"))) + agent = make_agent(server_client=seed_verify_server_client()) + + row = make_row(**self._REUSED_ROW_EXTRAS) + result = (await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row))).model_dump() + + assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + assert result["reward"] == 0.0 + assert result["response"]["output"][0]["type"] == "message" + assert "remote exploded" in result["error"] + # Stale no-persist/terminal flags from the input row must not survive + assert NG_NO_PERSIST_KEY not in result + assert NG_TERMINAL_KEY not in result + + def test_failure_on_reused_rollout_row_route_level_stays_200(self, monkeypatch: pytest.MonkeyPatch) -> None: + from fastapi.testclient import TestClient + + mock_remote(monkeypatch, AsyncMock(side_effect=RuntimeError("remote exploded"))) + agent = make_agent(server_client=seed_verify_server_client()) + client = TestClient(agent.setup_webserver(), raise_server_exceptions=False) + + response = client.post("/run", json=make_row(**self._REUSED_ROW_EXTRAS)) + + assert response.status_code == 200 + assert response.json()[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + + async def test_happy_path_reused_row_leaks_no_stale_sentinels(self, monkeypatch: pytest.MonkeyPatch) -> None: + # Stale routing keys on an input row must not echo through verify and misroute a + # SUCCESS into the failures sidecar. + mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)))) + agent = make_agent(server_client=seed_verify_server_client()) + + row = make_row(**self._REUSED_ROW_EXTRAS) + result = (await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row))).model_dump() + + assert result["reward"] == 1.0 + assert NG_FAILURE_CLASS_KEY not in result + assert NG_NO_PERSIST_KEY not in result + assert NG_TERMINAL_KEY not in result + + async def test_run_outer_backstop_never_raises(self, monkeypatch: pytest.MonkeyPatch) -> None: + agent = make_agent() + + async def explode(*args, **kwargs): + raise RuntimeError("internal bug") + + monkeypatch.setattr(agent, "_run_once", explode) + result = (await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row()))).model_dump() + + assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + assert "internal bug" in result["error"] + + async def test_aggregate_metrics_proxies_to_resources_server(self) -> None: + agg_body = { + "agent_metrics": {"mean/reward": 1.0}, + "key_metrics": {"mean/reward": 1.0}, + "group_level_metrics": [], + } + + async def _post(server_name, url_path, json=None, **kwargs): + assert url_path == "/aggregate_metrics" + assert server_name == "my_env" + return FakeServerClientResponse(agg_body) + + server_client = MagicMock(spec=ServerClient) + server_client.post = AsyncMock(side_effect=_post) + agent = make_agent(server_client=server_client) + + from nemo_gym.base_resources_server import AggregateMetricsRequest + + result = await agent.aggregate_metrics(AggregateMetricsRequest(verify_responses=[])) + assert result.key_metrics == {"mean/reward": 1.0} + + async def test_aggregate_metrics_bounded_when_resources_server_hangs( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + async def hang(*args, **kwargs): + await asyncio.sleep(60) + + server_client = MagicMock(spec=ServerClient) + server_client.post = AsyncMock(side_effect=hang) + agent = make_agent(server_client=server_client) + monkeypatch.setattr(remote_agent_app, "_AGGREGATE_PROXY_TIMEOUT_SECS", 0.05) + + from nemo_gym.base_resources_server import AggregateMetricsRequest + + with pytest.raises(asyncio.TimeoutError): + await agent.aggregate_metrics(AggregateMetricsRequest(verify_responses=[])) + + async def test_run_timeout_excludes_semaphore_queue_wait(self, monkeypatch: pytest.MonkeyPatch) -> None: + release_first = asyncio.Event() + + async def gated(*args, **kwargs): + await release_first.wait() + return FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)) + + mock_remote(monkeypatch, AsyncMock(side_effect=gated)) + agent = make_agent(server_client=seed_verify_server_client(), concurrency=1, run_timeout_secs=0.5) + + first = asyncio.create_task(agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row()))) + second = asyncio.create_task(agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row()))) + # Hold the only permit for most of the second task's would-be budget + await asyncio.sleep(0.4) + release_first.set() + results = [r.model_dump() for r in await asyncio.gather(first, second)] + + # If queue wait counted against run_timeout_secs, the second task would time out + assert all(NG_FAILURE_CLASS_KEY not in r for r in results) + + async def test_forward_session_loopback_warning_for_offhost_remote( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: + mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)))) + monkeypatch.setattr(remote_agent_app, "get_server_url", lambda name: "http://127.0.0.1:15022") + agent = make_agent( + server_client=seed_verify_server_client(), + forward_session=True, + agent_base_url="http://gpu-node-7:9000", + ) + + await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) + + assert "advertised_resources_url" in capsys.readouterr().out + + async def test_advertised_resources_url_overrides_header_and_silences_warning( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: + request_mock = AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))) + client = mock_remote(monkeypatch, request_mock) + monkeypatch.setattr(remote_agent_app, "get_server_url", lambda name: "http://127.0.0.1:15022") + agent = make_agent( + server_client=seed_verify_server_client(), + forward_session=True, + agent_base_url="http://gpu-node-7:9000", + advertised_resources_url="http://head-node.cluster:15022", + ) + + await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) + + headers = client.request.call_args.kwargs["headers"] + assert headers[RESOURCES_URL_HEADER] == "http://head-node.cluster:15022" + assert "advertised_resources_url" not in capsys.readouterr().out + + async def test_quality_warnings_are_throttled( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: + trajectory = dict(_MINIMAL_TRAJECTORY) + trajectory.pop("usage") + mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(trajectory)))) + agent = make_agent(server_client=seed_verify_server_client()) + + for _ in range(10): + await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) + + # Head of 5, then every 100th: 10 rollouts -> exactly 5 printed warnings + assert capsys.readouterr().out.count("no usage") == 5 From 33837c07c610eec5f40333080253097b4ad2f4d0 Mon Sep 17 00:00:00 2001 From: adil-a Date: Tue, 28 Jul 2026 19:32:58 +0000 Subject: [PATCH 02/25] refactor(remote_agent): tools_mode enum, review-thread cleanups - replace forward_session/assume_remote_tools with a single tools_mode: refuse|forward|remote enum, each choice documented - remove the speculative non-terminal-trajectory warning (and its test) - document the body/record duality on _run_once, the per-request URL forwarding rationale (random ports), and advertised_resources_url's bind-vs-advertise role directly in the code - README: off-host checklist (bind, route, verify from the remote machine, advertise) and the remote-side reachability self-check recommendation Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- responses_api_agents/remote_agent/README.md | 32 ++++++++--- responses_api_agents/remote_agent/app.py | 56 ++++++++++--------- .../remote_agent/configs/remote_agent.yaml | 4 +- .../remote_agent/tests/test_app.py | 36 +++--------- 4 files changed, 64 insertions(+), 64 deletions(-) diff --git a/responses_api_agents/remote_agent/README.md b/responses_api_agents/remote_agent/README.md index 1bef046ed5..554b078000 100644 --- a/responses_api_agents/remote_agent/README.md +++ b/responses_api_agents/remote_agent/README.md @@ -21,15 +21,29 @@ through a Gym model server) token-id capture for training. ## Gym-hosted tools (optional) -With `forward_session: true`, each request to your service carries two headers: -`X-NeMo-Gym-Resources-Server-Url` and `X-NeMo-Gym-Session-Cookie`. Echo the cookie on every -tool call you make against that URL and stateful environments work end to end. If your service -runs on a different machine, set `advertised_resources_url` to an externally reachable URL — -the default resolves to the resources server's bind address, which is typically a loopback -address only valid on the Gym host. Without -forwarding, tasks that declare tools are refused up front (instead of silently scoring 0) -unless you set `assume_remote_tools: true` because your service implements the declared tools -itself. +`tools_mode` decides who serves the tools a dataset declares: + +- `refuse` (default): tool-declaring tasks are rejected up front with a clear error instead of + silently scoring zero against untouched session state. +- `forward`: each request to your service carries two headers, `X-NeMo-Gym-Resources-Server-Url` + and `X-NeMo-Gym-Session-Cookie`. Echo the cookie on every tool call you make against that URL + and stateful environments work end to end. The URL is re-sent per request because Gym assigns + servers random ports on every start; the cookie is minted per rollout. +- `remote`: your service implements the declared tools itself; nothing is forwarded. + +### Running the service off-host (`tools_mode: forward`) + +The advertised URL must be reachable *from your service's machine* — Gym serves the tools and +tells your service where they are; making that address route to Gym is on you: + +1. Bind the resources server on all interfaces and pin its port (`host: 0.0.0.0`, `port: `). +2. Make the path route (internal DNS / firewall rule / SSH tunnel / load balancer — your infra). +3. Verify once from the remote machine: `curl http://
:/` should connect. +4. Set `advertised_resources_url: http://
:` on this agent. It changes only the + header string; the default advertises the bind address, which is typically a loopback address + other machines cannot reach (you'll see a warning for that combination). +5. Recommended: have your service probe the advertised URL on its first request and fail loudly — + reachability is only testable from your side (see the self-check in the docs example). ## Run diff --git a/responses_api_agents/remote_agent/app.py b/responses_api_agents/remote_agent/app.py index 4a77dbda48..e42ab56557 100644 --- a/responses_api_agents/remote_agent/app.py +++ b/responses_api_agents/remote_agent/app.py @@ -28,7 +28,7 @@ """ import asyncio -from typing import Any, Dict, Optional, Tuple +from typing import Any, Dict, Literal, Optional, Tuple from urllib.parse import urlparse import orjson @@ -53,8 +53,11 @@ REMOTE_AGENT_FAILURE_CLASS = "remote_agent_error" -# Header names of the optional session-forwarding contract (forward_session=true): the -# remote service echoes the cookie on every resources-server tool call it makes. +# Header names of the session-forwarding contract (tools_mode="forward"): the remote +# service echoes the cookie on every resources-server tool call it makes. The URL is +# re-sent per request (rather than configured remote-side) because Gym assigns servers +# random ports on every `gym env start` — a statically configured address goes stale on +# every restart; the cookie is minted per rollout and has no static equivalent at all. RESOURCES_URL_HEADER = "X-NeMo-Gym-Resources-Server-Url" SESSION_COOKIE_HEADER = "X-NeMo-Gym-Session-Cookie" @@ -111,16 +114,20 @@ class RemoteAgentConfig(BaseResponsesAPIAgentConfig): # semaphore is acquired so queue wait does not count against it. The collector's # named-agent hop carries no timeout of its own; this is the only wallclock bound. run_timeout_secs: float = 2100.0 - # Forward the resources-server URL and session cookie to the remote service so its - # loop can call Gym-hosted tools (it must echo the cookie on every tool call). - forward_session: bool = False - # The dataset declares tools but the remote service implements them itself; skip the - # declared-tools guard without forwarding the session. - assume_remote_tools: bool = False - # The resources-server URL advertised to the remote service with forward_session. The - # default (resolved from the global config) is the BIND address — typically 127.0.0.1, - # unreachable from another machine. Set this to the externally reachable URL when the - # remote service runs off-host. + # Who serves the tools a dataset declares: + # "refuse" — nobody can: reject tool-declaring tasks up front (terminal failure row) + # instead of letting verify() score silent zeros against untouched state. + # "forward" — Gym does: send the resources-server URL and session cookie as headers on + # every remote request; the service echoes the cookie on each tool call. + # "remote" — the service does: it implements the declared tools itself; nothing is + # forwarded and the guard stands down. + tools_mode: Literal["refuse", "forward", "remote"] = "refuse" + # The resources-server URL advertised to the remote service with tools_mode="forward". + # The default (resolved from the global config) is the BIND address — typically + # 127.0.0.1, unreachable from another machine. Set this to the externally reachable URL + # when the remote service runs off-host (bind vs. advertise can genuinely differ: NAT, + # tunnels, load balancers). This only changes the header string; making the address + # actually route to the resources server is the operator's job. advertised_resources_url: Optional[str] = None @field_validator("agent_base_url") @@ -183,6 +190,11 @@ def _sanitized_record(self, body: RemoteAgentRunRequest) -> Dict[str, Any]: async def _run_once( self, request: Request, body: RemoteAgentRunRequest, record: Dict[str, Any] ) -> RemoteAgentVerifyResponse: + # body and record are two views of the same row: `record` (sanitized dict, computed + # before run()'s try so failure rows can be built in ANY error state) feeds the Gym + # hops; `body` (typed model) is kept solely because exclude_unset information — which + # fields the dataset actually set — exists only on the model, and the remote wire + # payload must not carry materialized None defaults. guard_error = self._tools_guard_error(record) if guard_error: return self._failure_response(record, guard_error, terminal=True) @@ -251,11 +263,11 @@ def _tools_guard_error(self, record: Dict[str, Any]) -> Optional[str]: those tools mutate stays untouched and verify() scores 0 on every row. """ declared_tools = (record.get("responses_create_params") or {}).get("tools") - if declared_tools and not (self.config.forward_session or self.config.assume_remote_tools): + if declared_tools and self.config.tools_mode == "refuse": return ( - "the task declares tools but forward_session=false and assume_remote_tools=false. " - "Set forward_session=true so the remote service can call Gym-hosted tools with the " - "session cookie, or assume_remote_tools=true if the service implements the declared " + 'the task declares tools but tools_mode="refuse" (the default). Set ' + 'tools_mode="forward" so the remote service can call Gym-hosted tools with the ' + 'session cookie, or tools_mode="remote" if the service implements the declared ' "tools itself." ) return None @@ -268,7 +280,7 @@ async def _post_remote_responses( client = get_global_aiohttp_client() data = orjson.dumps(remote_params) headers = {"Content-Type": "application/json"} - if self.config.forward_session: + if self.config.tools_mode == "forward": resources_url = self.config.advertised_resources_url or get_server_url(self.config.resources_server.name) advertised_host = urlparse(resources_url).hostname or "" remote_host = urlparse(self.config.agent_base_url).hostname or "" @@ -351,14 +363,6 @@ def _throttled_warn(self, key: str, message: str) -> None: print(f"{message} (occurrence #{n})", flush=True) def _warn_on_response_quality(self, response: NeMoGymResponse) -> None: - last = response.output[-1] if response.output else None - finished_naturally = getattr(last, "type", None) == "message" and getattr(last, "role", None) == "assistant" - if not finished_naturally: - self._throttled_warn( - "non_terminal_trajectory", - "WARNING: the remote trajectory does not end with an assistant message; the contract " - "is one FINISHED trajectory per call (no dangling tool calls).", - ) if response.usage is None: self._throttled_warn( "missing_usage", diff --git a/responses_api_agents/remote_agent/configs/remote_agent.yaml b/responses_api_agents/remote_agent/configs/remote_agent.yaml index ebbea879c8..67a845d7c9 100644 --- a/responses_api_agents/remote_agent/configs/remote_agent.yaml +++ b/responses_api_agents/remote_agent/configs/remote_agent.yaml @@ -9,5 +9,5 @@ remote_agent: concurrency: 32 remote_responses_timeout_secs: 1800.0 run_timeout_secs: 2100.0 - forward_session: false - assume_remote_tools: false + tools_mode: refuse + advertised_resources_url: null diff --git a/responses_api_agents/remote_agent/tests/test_app.py b/responses_api_agents/remote_agent/tests/test_app.py index e5d4fb28a4..7e12f05b62 100644 --- a/responses_api_agents/remote_agent/tests/test_app.py +++ b/responses_api_agents/remote_agent/tests/test_app.py @@ -382,15 +382,15 @@ async def test_declared_tools_refused_by_default(self, monkeypatch: pytest.Monke assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS assert result[NG_TERMINAL_KEY] is True - assert "forward_session" in result["error"] and "assume_remote_tools" in result["error"] + assert "tools_mode" in result["error"] # Refused before any network traffic assert client.request.call_count == 0 assert server_client.calls == [] - async def test_assume_remote_tools_skips_guard_without_headers(self, monkeypatch: pytest.MonkeyPatch) -> None: + async def test_tools_mode_remote_skips_guard_without_headers(self, monkeypatch: pytest.MonkeyPatch) -> None: request_mock = AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))) client = mock_remote(monkeypatch, request_mock) - agent = make_agent(server_client=seed_verify_server_client(), assume_remote_tools=True) + agent = make_agent(server_client=seed_verify_server_client(), tools_mode="remote") result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row(tools=self._TOOLS))) @@ -398,12 +398,12 @@ async def test_assume_remote_tools_skips_guard_without_headers(self, monkeypatch headers = client.request.call_args.kwargs["headers"] assert RESOURCES_URL_HEADER not in headers - async def test_forward_session_sends_url_and_cookie_headers(self, monkeypatch: pytest.MonkeyPatch) -> None: + async def test_tools_mode_forward_sends_url_and_cookie_headers(self, monkeypatch: pytest.MonkeyPatch) -> None: request_mock = AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))) client = mock_remote(monkeypatch, request_mock) monkeypatch.setattr(remote_agent_app, "get_server_url", lambda name: f"http://resolved-{name}:1234") server_client = seed_verify_server_client(seed_cookies={"session": "cookie-value"}) - agent = make_agent(server_client=server_client, forward_session=True) + agent = make_agent(server_client=server_client, tools_mode="forward") result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row(tools=self._TOOLS))) @@ -431,24 +431,6 @@ async def _run_with_trajectory(self, monkeypatch, trajectory): agent = make_agent(server_client=seed_verify_server_client()) return await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) - async def test_non_terminal_trajectory_warns( - self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] - ) -> None: - trajectory = dict(_MINIMAL_TRAJECTORY) - trajectory["output"] = [ - { - "type": "function_call", - "status": "completed", - "id": "fc_1", - "call_id": "call_1", - "name": "increment_counter", - "arguments": "{}", - } - ] - result = await self._run_with_trajectory(monkeypatch, trajectory) - assert NG_FAILURE_CLASS_KEY not in result.model_dump() - assert "does not end with an assistant message" in capsys.readouterr().out - async def test_missing_usage_warns( self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] ) -> None: @@ -598,7 +580,7 @@ async def remote_service(method, url, data=None, headers=None, **kwargs): mock_remote(monkeypatch, AsyncMock(side_effect=remote_service)) monkeypatch.setattr(remote_agent_app, "get_server_url", lambda name: "http://counter-in-process") - agent = make_agent(server_client=server_client, forward_session=True) + agent = make_agent(server_client=server_client, tools_mode="forward") row = { "responses_create_params": { "input": [{"role": "user", "content": "add 1 then add 2 then get the count"}], @@ -817,14 +799,14 @@ async def gated(*args, **kwargs): # If queue wait counted against run_timeout_secs, the second task would time out assert all(NG_FAILURE_CLASS_KEY not in r for r in results) - async def test_forward_session_loopback_warning_for_offhost_remote( + async def test_tools_mode_forward_loopback_warning_for_offhost_remote( self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] ) -> None: mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)))) monkeypatch.setattr(remote_agent_app, "get_server_url", lambda name: "http://127.0.0.1:15022") agent = make_agent( server_client=seed_verify_server_client(), - forward_session=True, + tools_mode="forward", agent_base_url="http://gpu-node-7:9000", ) @@ -840,7 +822,7 @@ async def test_advertised_resources_url_overrides_header_and_silences_warning( monkeypatch.setattr(remote_agent_app, "get_server_url", lambda name: "http://127.0.0.1:15022") agent = make_agent( server_client=seed_verify_server_client(), - forward_session=True, + tools_mode="forward", agent_base_url="http://gpu-node-7:9000", advertised_resources_url="http://head-node.cluster:15022", ) From 47ac9a900eed568a707709ea88932e2e90c1997c Mon Sep 17 00:00:00 2001 From: adil-a Date: Tue, 28 Jul 2026 20:35:05 +0000 Subject: [PATCH 03/25] docs: add Drive a Remote Agent page Usage guide for the remote_agent server: the one-endpoint contract with request/response expectations, a minimal FastAPI quickstart, every config knob with defaults, the tools_mode choices with the off-host checklist for forwarded sessions, failure/resume semantics, and the gotchas list (redirects, missing usage, timeout retry semantics, skills, reused rollout files, stacked concurrency bounds). Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- .../pages/agent-server/agent-skills.mdx | 2 +- .../pages/agent-server/remote-agent.mdx | 145 ++++++++++++++++++ 2 files changed, 146 insertions(+), 1 deletion(-) create mode 100644 fern/versions/latest/pages/agent-server/remote-agent.mdx diff --git a/fern/versions/latest/pages/agent-server/agent-skills.mdx b/fern/versions/latest/pages/agent-server/agent-skills.mdx index 8431643002..b8c9e4ebc4 100644 --- a/fern/versions/latest/pages/agent-server/agent-skills.mdx +++ b/fern/versions/latest/pages/agent-server/agent-skills.mdx @@ -1,7 +1,7 @@ --- title: "Agent Skills" description: "Evaluate agent skills as a run-level variable, decoupled from the dataset" -position: 3 +position: 4 --- Skills are reusable units of operational knowledge an agent can load at runtime, following the open [Agent Skills standard](https://agentskills.io/specification) used by Claude Code and Codex CLI. A skill is a **directory** containing a `SKILL.md` file (YAML frontmatter + markdown body) plus optional supporting files. diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx new file mode 100644 index 0000000000..43bae60060 --- /dev/null +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -0,0 +1,145 @@ +--- +title: "Drive a Remote Agent" +description: "Evaluate an agent service you host yourself — one endpoint, verification stays in Gym" +position: 3 +--- + +# Drive a Remote Agent + +The `remote_agent` server lets an agent that runs as **its own HTTP service** — in your repo, on +your infrastructure — be driven by standard rollout collection. Your service implements one +endpoint; Gym keeps the session, the verification, and the task answer keys on its side, and your +results land in the standard artifacts (`gym eval profile`, aggregation, and training pipelines +all work unchanged). + +``` +collector ──/run──▶ remote_agent (Gym) ──POST /v1/responses──▶ your service + │ 1. seed_session (your model, your tools, + │ 4. verify (same session) returns one trajectory) + ▼ + resources server +``` + +## The contract for your service + +Implement `POST {agent_base_url}/v1/responses`: + +| | | +|---|---| +| **Request body** | The task's `responses_create_params` — the input messages and any declared tool schemas. Never `verifier_metadata` (the answer key stays inside Gym). | +| **Request headers** (only with `tools_mode: forward`) | `X-NeMo-Gym-Resources-Server-Url` and `X-NeMo-Gym-Session-Cookie` — see [Gym-hosted tools](#gym-hosted-tools-tools_mode). | +| **Response** | One **finished** Responses API object: run your whole loop (any number of model turns and tool calls) and return the merged trajectory, ending with an assistant message. Populate `usage` (`input_tokens`, `output_tokens`, `total_tokens`) — without it your token metrics are silently empty. | + + +The response is validated strictly against the Responses API schema. An invalid object is +recorded as a terminal failure (a schema bug will not fix itself on retry) and never reaches the +verifier. + + +## Quickstart + +A minimal service (FastAPI, ~20 lines): + +```python +from fastapi import FastAPI, Request + +app = FastAPI() + +@app.post("/v1/responses") +async def responses(request: Request): + params = await request.json() + answer = my_agent_loop(params["input"]) # your model, your tools, your turns + return { + "id": "my-service", "created_at": 0.0, "model": "my-model", "object": "response", + "output": [{ + "type": "message", "role": "assistant", "status": "completed", "id": "msg_0", + "content": [{"type": "output_text", "text": answer, "annotations": []}], + }], + "parallel_tool_calls": False, "tools": [], "tool_choice": "auto", + "usage": {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0}, + } +``` + +Wire it into Gym: + +```yaml +remote_agent: + responses_api_agents: + remote_agent: + entrypoint: app.py + agent_base_url: http://localhost:9000 # your service + resources_server: + type: resources_servers + name: my_env_resources_server # which environment verifies + tools_mode: refuse +``` + +```bash +gym env start --resources-server my_env "+config_paths=[.../remote_agent.yaml]" ... +gym eval run --no-serve +agent_name=remote_agent \ + +input_jsonl_fpath=data/tasks.jsonl +output_jsonl_fpath=results/rollouts.jsonl +``` + +## Configuration knobs + +| Field | Default | What it does | +|---|---|---| +| `agent_base_url` | required | Your service's base URL. Validated: `http(s)` only, no query string or fragment, no embedded credentials. | +| `resources_server` | required | The environment that seeds and verifies each rollout. Swap benchmarks by changing this ref — your service doesn't change. | +| `tools_mode` | `refuse` | Who serves the tools a dataset declares — see below. | +| `concurrency` | `32` | Maximum rollouts in flight against your service, enforced server-side. This is the *owner's* bound: it caps total pressure no matter how many collection runs call in. | +| `remote_responses_timeout_secs` | `1800` | Wallclock bound on one `/v1/responses` call. Raise it if your rollouts legitimately run longer than 30 minutes. | +| `run_timeout_secs` | `2100` | Bound on a whole `/run` (seed + your service + verify), started after the concurrency slot is acquired — queue wait doesn't count. | +| `advertised_resources_url` | unset | With `tools_mode: forward`: the resources-server URL told to your service. The default advertises the bind address (typically `127.0.0.1`), which another machine cannot reach — set this when your service runs off-host. It changes only the header string; making the address route to Gym is your infrastructure's job. | + +## Gym-hosted tools (`tools_mode`) + +- **`refuse`** (default) — tool-declaring tasks are rejected up front with a clear, terminal error. + This protects you from the silent failure mode: without session access, tools that mutate + per-session state never run, and every rollout scores 0 with no error anywhere. +- **`forward`** — each request to your service carries the resources-server URL and the rollout's + session cookie as headers. Echo the cookie on every tool call you make against that URL and + stateful environments work end to end. The URL is re-sent per request because Gym assigns + servers random ports on every start; the cookie is minted per rollout. +- **`remote`** — your service implements the declared tools itself; nothing is forwarded. + +### Running your service off-host with `tools_mode: forward` + +The advertised URL must be reachable *from your service's machine*: + +1. Bind the resources server on all interfaces and pin its port (`host: 0.0.0.0`, `port: `). +2. Make the path route — internal DNS, firewall rule, SSH tunnel, or load balancer (your infra). +3. Verify once **from the remote machine**: `curl http://
:/`. +4. Set `advertised_resources_url: http://
:`. +5. Recommended: probe the advertised URL from your service on first request and fail loudly — + reachability is only testable from your side. + + +Forgetting step 4 on an off-host deployment is the classic mistake: your service receives a +loopback address, every tool call misses Gym, and rewards are silently zero. The agent logs a +warning when it detects this combination, but it cannot verify reachability for you. + + +## Failure handling and resume + +Failures never crash a collection run. A down service (bounded connect retries), a timeout, a +malformed reply, or a verifier error becomes a reward-0 row with `_ng_failure_class: +"remote_agent_error"` in the failures sidecar (`_failures.jsonl`) — the main rollouts +file stays clean, and `+resume_from_cache=true` retries failed tasks up to the attempt cap. Error +messages name the URL and the timeout knob involved. + +## Gotchas + +- **Redirects are rejected, not followed.** A `301/302` from your service fails the rollout with + the `Location` shown — point `agent_base_url` at the final address. (Following a redirected + POST would silently convert it to a body-less GET.) +- **Missing `usage` = empty token metrics.** The run succeeds and warns; your cost/token + accounting is empty. Report real token counts. +- **Timeouts don't retry inline** — a timed-out rollout goes to the sidecar and is retried on + resume, so one hung request doesn't burn triple wallclock. +- **Skills are not forwarded.** A `+skills` config stamps rows but cannot stage files into a + remote process; the agent warns and ignores it. +- **Reused rollout files as input are safe** — rows carrying stale `reward`/`response`/routing + keys from previous runs are sanitized before processing. +- **`num_samples_in_parallel` and `concurrency` stack safely**: the collector bounds one run's + politeness, the agent bounds your service's total load; the tighter one wins. From ee78705af593d41dbdfa0c038a7e168896018dbc Mon Sep 17 00:00:00 2001 From: adil-a Date: Wed, 29 Jul 2026 05:29:27 +0000 Subject: [PATCH 04/25] fix: docs corrections, 0.0.0.0 advertise warning, debug-knob parity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Docs: quickstart usage example and contract table now show the full usage shape strict validation requires; placeholder/ref-name clarified; retry-cap and terminal-failure semantics stated; MCP forward-mode limitation and internet-exposure (no outbound auth, no HTTP_PROXY) notes added; dead README cross-reference removed. Code: the unroutable-advertise warning now also covers 0.0.0.0 (a bind address resolves to the remote machine's own loopback — confirmed live in an off-host container E2E where it produced silent zero rewards with no warning); remote-hop failures honor global_aiohttp_client_request_debug with full tracebacks/bodies, matching core request()/raise_for_status. Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- .../pages/agent-server/remote-agent.mdx | 36 ++++++++++++++----- responses_api_agents/remote_agent/README.md | 10 ++++-- responses_api_agents/remote_agent/app.py | 32 +++++++++++++---- .../remote_agent/tests/test_app.py | 17 +++++++++ 4 files changed, 77 insertions(+), 18 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 43bae60060..9b03bafac6 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -28,7 +28,7 @@ Implement `POST {agent_base_url}/v1/responses`: |---|---| | **Request body** | The task's `responses_create_params` — the input messages and any declared tool schemas. Never `verifier_metadata` (the answer key stays inside Gym). | | **Request headers** (only with `tools_mode: forward`) | `X-NeMo-Gym-Resources-Server-Url` and `X-NeMo-Gym-Session-Cookie` — see [Gym-hosted tools](#gym-hosted-tools-tools_mode). | -| **Response** | One **finished** Responses API object: run your whole loop (any number of model turns and tool calls) and return the merged trajectory, ending with an assistant message. Populate `usage` (`input_tokens`, `output_tokens`, `total_tokens`) — without it your token metrics are silently empty. | +| **Response** | One **finished** Responses API object: run your whole loop (any number of model turns and tool calls) and return the merged trajectory, ending with an assistant message. Populate the full `usage` object — `input_tokens`, `output_tokens`, `total_tokens`, `input_tokens_details: {cached_tokens}`, `output_tokens_details: {reasoning_tokens}` — or omit `usage` entirely (allowed; token metrics are then empty). A partial `usage` object fails validation. | The response is validated strictly against the Responses API schema. An invalid object is @@ -56,7 +56,11 @@ async def responses(request: Request): "content": [{"type": "output_text", "text": answer, "annotations": []}], }], "parallel_tool_calls": False, "tools": [], "tool_choice": "auto", - "usage": {"input_tokens": 0, "output_tokens": 0, "total_tokens": 0}, + "usage": { + "input_tokens": 0, "output_tokens": 0, "total_tokens": 0, + "input_tokens_details": {"cached_tokens": 0}, + "output_tokens_details": {"reasoning_tokens": 0}, + }, } ``` @@ -70,7 +74,7 @@ remote_agent: agent_base_url: http://localhost:9000 # your service resources_server: type: resources_servers - name: my_env_resources_server # which environment verifies + name: my_env_resources_server # the top-level key of that environment's config tools_mode: refuse ``` @@ -87,7 +91,7 @@ gym eval run --no-serve +agent_name=remote_agent \ | `agent_base_url` | required | Your service's base URL. Validated: `http(s)` only, no query string or fragment, no embedded credentials. | | `resources_server` | required | The environment that seeds and verifies each rollout. Swap benchmarks by changing this ref — your service doesn't change. | | `tools_mode` | `refuse` | Who serves the tools a dataset declares — see below. | -| `concurrency` | `32` | Maximum rollouts in flight against your service, enforced server-side. This is the *owner's* bound: it caps total pressure no matter how many collection runs call in. | +| `concurrency` | `32` | Maximum rollouts in flight against your service, enforced server-side. This is the *owner's* bound: it caps total pressure from every collection run calling into this agent process. (A second, independently started Gym stack has its own bound.) | | `remote_responses_timeout_secs` | `1800` | Wallclock bound on one `/v1/responses` call. Raise it if your rollouts legitimately run longer than 30 minutes. | | `run_timeout_secs` | `2100` | Bound on a whole `/run` (seed + your service + verify), started after the concurrency slot is acquired — queue wait doesn't count. | | `advertised_resources_url` | unset | With `tools_mode: forward`: the resources-server URL told to your service. The default advertises the bind address (typically `127.0.0.1`), which another machine cannot reach — set this when your service runs off-host. It changes only the header string; making the address route to Gym is your infrastructure's job. | @@ -103,6 +107,13 @@ gym eval run --no-serve +agent_name=remote_agent \ servers random ports on every start; the cookie is minted per rollout. - **`remote`** — your service implements the declared tools itself; nothing is forwarded. + +`forward` covers environments whose tools are plain HTTP routes on the resources server (the +session cookie is the only credential involved). Environments that expose their tools **over MCP** +mint additional per-session token headers that are not forwarded in this version — MCP-based +environments need a metadata-forwarding follow-up. + + ### Running your service off-host with `tools_mode: forward` The advertised URL must be reachable *from your service's machine*: @@ -115,9 +126,10 @@ The advertised URL must be reachable *from your service's machine*: reachability is only testable from your side. -Forgetting step 4 on an off-host deployment is the classic mistake: your service receives a -loopback address, every tool call misses Gym, and rewards are silently zero. The agent logs a -warning when it detects this combination, but it cannot verify reachability for you. +Forgetting step 4 on an off-host deployment is the classic mistake: your service receives an +unroutable bind address (`127.0.0.1` or `0.0.0.0`), every tool call misses Gym, and rewards are +silently zero. The agent logs a warning when it detects this combination, but it cannot verify +reachability for you. ## Failure handling and resume @@ -125,8 +137,9 @@ warning when it detects this combination, but it cannot verify reachability for Failures never crash a collection run. A down service (bounded connect retries), a timeout, a malformed reply, or a verifier error becomes a reward-0 row with `_ng_failure_class: "remote_agent_error"` in the failures sidecar (`_failures.jsonl`) — the main rollouts -file stays clean, and `+resume_from_cache=true` retries failed tasks up to the attempt cap. Error -messages name the URL and the timeout knob involved. +file stays clean, and `+resume_from_cache=true` retries failed tasks up to the attempt cap +(`NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`, default 3; terminal failures are not retried). Error messages +name the failing URL or the timeout knob involved. ## Gotchas @@ -143,3 +156,8 @@ messages name the URL and the timeout knob involved. keys from previous runs are sanitized before processing. - **`num_samples_in_parallel` and `concurrency` stack safely**: the collector bounds one run's politeness, the agent bounds your service's total load; the tighter one wins. +- **Exposing your service on the public internet needs network-level trust.** Gym sends no + credential on the `/v1/responses` call (`agent_base_url` rejects embedded credentials), so an + internet-exposed service should sit behind a VPN, private network, IP allowlist, or + authenticating tunnel. Corporate egress proxies are not honored either — Gym's HTTP client + ignores `HTTP_PROXY` environment variables. diff --git a/responses_api_agents/remote_agent/README.md b/responses_api_agents/remote_agent/README.md index 554b078000..f36bfca1e3 100644 --- a/responses_api_agents/remote_agent/README.md +++ b/responses_api_agents/remote_agent/README.md @@ -15,9 +15,13 @@ through a Gym model server) token-id capture for training. - `POST {agent_base_url}/v1/responses` with the row's `responses_create_params` as the JSON body. - Return a single finished Responses API object: the last output item is an assistant message, - no dangling tool calls, `usage` populated (`{input_tokens, output_tokens, total_tokens}`). + no dangling tool calls, `usage` populated with the full shape — `{input_tokens, output_tokens, + total_tokens, input_tokens_details: {cached_tokens}, output_tokens_details: {reasoning_tokens}}`. + (Omitting `usage` entirely is allowed and only warns; a partial `usage` object fails validation.) - Failures on Gym's side never crash a collection run: they are recorded as reward-0 rows in - the failures sidecar and retried on resume. + the failures sidecar and retried on resume up to the attempt cap + (`NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`, default 3). Failures marked terminal — an invalid response + shape, or the tools guard — are not retried. ## Gym-hosted tools (optional) @@ -43,7 +47,7 @@ tells your service where they are; making that address route to Gym is on you: header string; the default advertises the bind address, which is typically a loopback address other machines cannot reach (you'll see a warning for that combination). 5. Recommended: have your service probe the advertised URL on its first request and fail loudly — - reachability is only testable from your side (see the self-check in the docs example). + reachability is only testable from your side. ## Run diff --git a/responses_api_agents/remote_agent/app.py b/responses_api_agents/remote_agent/app.py index e42ab56557..2c879ec642 100644 --- a/responses_api_agents/remote_agent/app.py +++ b/responses_api_agents/remote_agent/app.py @@ -28,6 +28,7 @@ """ import asyncio +from traceback import print_exc from typing import Any, Dict, Literal, Optional, Tuple from urllib.parse import urlparse @@ -48,7 +49,13 @@ from nemo_gym.global_config import SKILLS_REF_KEY_NAME from nemo_gym.openai_utils import NeMoGymResponse from nemo_gym.rollout_collection import NG_FAILURE_CLASS_KEY, NG_NO_PERSIST_KEY, NG_TERMINAL_KEY -from nemo_gym.server_utils import get_global_aiohttp_client, get_response_json, get_server_url, raise_for_status +from nemo_gym.server_utils import ( + get_global_aiohttp_client, + get_response_json, + get_server_url, + is_global_aiohttp_client_request_debug_enabled, + raise_for_status, +) REMOTE_AGENT_FAILURE_CLASS = "remote_agent_error" @@ -232,6 +239,8 @@ async def _run_once( try: response = NeMoGymResponse.model_validate(remote_result) except PydanticValidationError as e: + if is_global_aiohttp_client_request_debug_enabled(): + print(f"[remote_agent] full validation error: {e}", flush=True) # A shape error will not fix itself on retry. return self._failure_response( record, @@ -284,12 +293,15 @@ async def _post_remote_responses( resources_url = self.config.advertised_resources_url or get_server_url(self.config.resources_server.name) advertised_host = urlparse(resources_url).hostname or "" remote_host = urlparse(self.config.agent_base_url).hostname or "" - if advertised_host in ("127.0.0.1", "localhost") and remote_host not in ("127.0.0.1", "localhost"): + # 0.0.0.0 is a bind address, not a routable one: from the remote machine it + # resolves to that machine's own loopback, exactly like 127.0.0.1 would. + local_only_hosts = ("127.0.0.1", "localhost", "0.0.0.0") + if advertised_host in local_only_hosts and remote_host not in local_only_hosts: self._throttled_warn( "loopback_resources_url", - f"WARNING: forwarding resources-server URL {resources_url} (a loopback address) to the " - f"off-host remote service at {self.config.agent_base_url}. Its tool calls will not reach " - "Gym; set advertised_resources_url to an externally reachable URL.", + f"WARNING: forwarding resources-server URL {resources_url} (an address other machines " + f"cannot reach) to the off-host remote service at {self.config.agent_base_url}. Its tool " + "calls will not reach Gym; set advertised_resources_url to an externally reachable URL.", ) headers[RESOURCES_URL_HEADER] = resources_url session_cookie = cookie_header_value(cookies) @@ -320,6 +332,8 @@ async def _post_remote_responses( False, ) except Exception as e: + if is_global_aiohttp_client_request_debug_enabled(): + print_exc() return None, f"{type(e).__name__}: {e}", False if response is None: return ( @@ -335,9 +349,13 @@ async def _post_remote_responses( try: content = await response.read() except Exception as e: + if is_global_aiohttp_client_request_debug_enabled(): + print_exc() return None, f"reading the response body failed: {type(e).__name__}: {e}", False # response.ok is `status < 400`; reject 3xx explicitly (redirects are not followed). if not response.ok or response.status >= 300: + if is_global_aiohttp_client_request_debug_enabled(): + print(f"[remote_agent] full HTTP {response.status} body: {content.decode(errors='replace')}", flush=True) location = response.headers.get("Location", "") return ( None, @@ -367,7 +385,9 @@ def _warn_on_response_quality(self, response: NeMoGymResponse) -> None: self._throttled_warn( "missing_usage", "WARNING: the remote response carries no usage; token metrics for this agent will be " - "empty. Have your service report usage {input_tokens, output_tokens, total_tokens}.", + "empty. Have your service report the full usage object: {input_tokens, output_tokens, " + "total_tokens, input_tokens_details: {cached_tokens}, output_tokens_details: " + "{reasoning_tokens}}.", ) def _failure_response( diff --git a/responses_api_agents/remote_agent/tests/test_app.py b/responses_api_agents/remote_agent/tests/test_app.py index 7e12f05b62..255ffa608c 100644 --- a/responses_api_agents/remote_agent/tests/test_app.py +++ b/responses_api_agents/remote_agent/tests/test_app.py @@ -814,6 +814,23 @@ async def test_tools_mode_forward_loopback_warning_for_offhost_remote( assert "advertised_resources_url" in capsys.readouterr().out + async def test_tools_mode_forward_warns_on_unroutable_bind_address_too( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: + # host: 0.0.0.0 makes get_server_url advertise 0.0.0.0 — from the remote machine that + # resolves to ITS OWN loopback, the same silent-zero failure as advertising 127.0.0.1. + mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)))) + monkeypatch.setattr(remote_agent_app, "get_server_url", lambda name: "http://0.0.0.0:15022") + agent = make_agent( + server_client=seed_verify_server_client(), + tools_mode="forward", + agent_base_url="http://gpu-node-7:9000", + ) + + await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) + + assert "advertised_resources_url" in capsys.readouterr().out + async def test_advertised_resources_url_overrides_header_and_silences_warning( self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] ) -> None: From 5b446d319eba98de8d1df15232e198b7e126f878 Mon Sep 17 00:00:00 2001 From: adil-a Date: Wed, 29 Jul 2026 05:51:01 +0000 Subject: [PATCH 05/25] docs: fix step numbering in remote_agent flow diagram Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- fern/versions/latest/pages/agent-server/remote-agent.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 9b03bafac6..788a34e513 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -13,9 +13,9 @@ results land in the standard artifacts (`gym eval profile`, aggregation, and tra all work unchanged). ``` -collector ──/run──▶ remote_agent (Gym) ──POST /v1/responses──▶ your service - │ 1. seed_session (your model, your tools, - │ 4. verify (same session) returns one trajectory) +collector ──/run──▶ remote_agent (Gym) ──2. POST /v1/responses──▶ your service + │ 1. seed_session (your model, your tools, + │ 3. verify (same session) returns one trajectory) ▼ resources server ``` From 68dcaa0fd01c1fad544f52418cb8c39f468a7df6 Mon Sep 17 00:00:00 2001 From: adil-a Date: Wed, 29 Jul 2026 06:18:53 +0000 Subject: [PATCH 06/25] docs: expand when to use tools_mode refuse; drop follow-up promise from MCP note Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- .../latest/pages/agent-server/remote-agent.mdx | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 788a34e513..37d1b5df6e 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -99,8 +99,13 @@ gym eval run --no-serve +agent_name=remote_agent \ ## Gym-hosted tools (`tools_mode`) - **`refuse`** (default) — tool-declaring tasks are rejected up front with a clear, terminal error. - This protects you from the silent failure mode: without session access, tools that mutate - per-session state never run, and every rollout scores 0 with no error anywhere. + Use this whenever your tasks don't declare Gym-hosted tools — text-only benchmarks, tasks whose + tools all live inside your own service — which is the common case. Tool-free rows run completely + normally under `refuse`; the guard only fires if a row *declares* tools. Keeping it on also acts + as a tripwire while you wire things up: if a tool-declaring dataset shows up before your service + echoes the session cookie, you get an immediate, named error instead of the silent failure mode — + without session access, tools that mutate per-session state never run, and every rollout scores 0 + with no error anywhere. - **`forward`** — each request to your service carries the resources-server URL and the rollout's session cookie as headers. Echo the cookie on every tool call you make against that URL and stateful environments work end to end. The URL is re-sent per request because Gym assigns @@ -110,8 +115,7 @@ gym eval run --no-serve +agent_name=remote_agent \ `forward` covers environments whose tools are plain HTTP routes on the resources server (the session cookie is the only credential involved). Environments that expose their tools **over MCP** -mint additional per-session token headers that are not forwarded in this version — MCP-based -environments need a metadata-forwarding follow-up. +mint additional per-session token headers that are not forwarded in this version. ### Running your service off-host with `tools_mode: forward` From ef6b635d9182eedeb41cb6213511f77eadf6a229 Mon Sep 17 00:00:00 2001 From: adil-a Date: Wed, 29 Jul 2026 06:19:34 +0000 Subject: [PATCH 07/25] docs: reword off-host warning; a new feature has no classic mistakes yet Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- fern/versions/latest/pages/agent-server/remote-agent.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 37d1b5df6e..2c12c1b4e0 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -130,8 +130,8 @@ The advertised URL must be reachable *from your service's machine*: reachability is only testable from your side. -Forgetting step 4 on an off-host deployment is the classic mistake: your service receives an -unroutable bind address (`127.0.0.1` or `0.0.0.0`), every tool call misses Gym, and rewards are +The easiest step to forget on an off-host deployment is step 4: without it, your service receives +an unroutable bind address (`127.0.0.1` or `0.0.0.0`), every tool call misses Gym, and rewards are silently zero. The agent logs a warning when it detects this combination, but it cannot verify reachability for you. From 3f6cf45eebeb5b48631b450daf8bcb497ab76bcc Mon Sep 17 00:00:00 2001 From: adil-a Date: Wed, 29 Jul 2026 06:21:24 +0000 Subject: [PATCH 08/25] docs: correct redirect claim; 307/308 preserve the POST body, 301/302/303 drop it Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- fern/versions/latest/pages/agent-server/remote-agent.mdx | 7 ++++--- responses_api_agents/remote_agent/app.py | 3 ++- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 2c12c1b4e0..50018f1f3d 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -147,9 +147,10 @@ name the failing URL or the timeout knob involved. ## Gotchas -- **Redirects are rejected, not followed.** A `301/302` from your service fails the rollout with - the `Location` shown — point `agent_base_url` at the final address. (Following a redirected - POST would silently convert it to a body-less GET.) +- **Redirects are rejected, not followed.** Any `3xx` from your service fails the rollout with + the `Location` shown — point `agent_base_url` at the final address. (Followed, a `301/302/303` + would silently re-issue the POST as a body-less GET; a `307/308` would silently re-send the + task to an address you never configured.) - **Missing `usage` = empty token metrics.** The run succeeds and warns; your cost/token accounting is empty. Report real token counts. - **Timeouts don't retry inline** — a timed-out rollout goes to the sidecar and is retried on diff --git a/responses_api_agents/remote_agent/app.py b/responses_api_agents/remote_agent/app.py index 2c879ec642..abf18904a8 100644 --- a/responses_api_agents/remote_agent/app.py +++ b/responses_api_agents/remote_agent/app.py @@ -313,7 +313,8 @@ async def _post_remote_responses( last_connect_error: Optional[BaseException] = None for num_try in range(1, _REMOTE_MAX_TRIES + 1): try: - # aiohttp follows a redirected POST as a body-less GET; fail with the 3xx instead. + # Never follow redirects: aiohttp re-issues 301/302/303 as a body-less GET and + # re-sends 307/308 to an address the user never configured; fail with the 3xx. response = await client.request( "POST", remote_url, data=data, headers=headers, timeout=timeout, allow_redirects=False ) From 08376e0f94aab964f09c7e241dcc6bb84e83e35a Mon Sep 17 00:00:00 2001 From: adil-a Date: Wed, 29 Jul 2026 06:21:58 +0000 Subject: [PATCH 09/25] docs: explain reused-rollout-files gotcha in plain terms Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- fern/versions/latest/pages/agent-server/remote-agent.mdx | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 50018f1f3d..5bd8ea98f4 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -157,8 +157,10 @@ name the failing URL or the timeout knob involved. resume, so one hung request doesn't burn triple wallclock. - **Skills are not forwarded.** A `+skills` config stamps rows but cannot stage files into a remote process; the agent warns and ignores it. -- **Reused rollout files as input are safe** — rows carrying stale `reward`/`response`/routing - keys from previous runs are sanitized before processing. +- **A previous run's output can be reused as the input dataset** (for example, re-running the + tasks in a failures sidecar). Output rows carry the old run's results (`reward`, `response`, + `error`, internal `_ng_*` flags); those fields are stripped from each row on the way in, so a + stale result can't collide with or leak into the new run's output. - **`num_samples_in_parallel` and `concurrency` stack safely**: the collector bounds one run's politeness, the agent bounds your service's total load; the tighter one wins. - **Exposing your service on the public internet needs network-level trust.** Gym sends no From 6406e504d48273531d87e0be2b34c1d48b5916c3 Mon Sep 17 00:00:00 2001 From: adil-a Date: Wed, 29 Jul 2026 06:22:25 +0000 Subject: [PATCH 10/25] docs: drop num_samples_in_parallel/concurrency stacking gotcha Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- fern/versions/latest/pages/agent-server/remote-agent.mdx | 2 -- 1 file changed, 2 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 5bd8ea98f4..470c960a8a 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -161,8 +161,6 @@ name the failing URL or the timeout knob involved. tasks in a failures sidecar). Output rows carry the old run's results (`reward`, `response`, `error`, internal `_ng_*` flags); those fields are stripped from each row on the way in, so a stale result can't collide with or leak into the new run's output. -- **`num_samples_in_parallel` and `concurrency` stack safely**: the collector bounds one run's - politeness, the agent bounds your service's total load; the tighter one wins. - **Exposing your service on the public internet needs network-level trust.** Gym sends no credential on the `/v1/responses` call (`agent_base_url` rejects embedded credentials), so an internet-exposed service should sit behind a VPN, private network, IP allowlist, or From 2a4abb213d3bb165488a12fd5e6ea2aa146adb2b Mon Sep 17 00:00:00 2001 From: adil-a Date: Wed, 29 Jul 2026 06:23:27 +0000 Subject: [PATCH 11/25] docs: scope proxy claim to env-var-configured proxies Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- fern/versions/latest/pages/agent-server/remote-agent.mdx | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 470c960a8a..883bd21770 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -164,5 +164,6 @@ name the failing URL or the timeout knob involved. - **Exposing your service on the public internet needs network-level trust.** Gym sends no credential on the `/v1/responses` call (`agent_base_url` rejects embedded credentials), so an internet-exposed service should sit behind a VPN, private network, IP allowlist, or - authenticating tunnel. Corporate egress proxies are not honored either — Gym's HTTP client - ignores `HTTP_PROXY` environment variables. + authenticating tunnel. Note that proxies configured via `HTTP_PROXY`/`HTTPS_PROXY` environment + variables are ignored (Gym's HTTP client does not read them), so a host that can only reach the + internet through such a proxy cannot reach your service. From 1fdc6d4ff6eff1f67a58402899dbafe7d2258af1 Mon Sep 17 00:00:00 2001 From: adil-a Date: Wed, 29 Jul 2026 06:27:11 +0000 Subject: [PATCH 12/25] ci: satisfy ruff-format and allowlist test-fixture credentials for secrets-detector Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- responses_api_agents/remote_agent/app.py | 4 +++- responses_api_agents/remote_agent/tests/test_app.py | 10 ++++++++-- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/responses_api_agents/remote_agent/app.py b/responses_api_agents/remote_agent/app.py index abf18904a8..6cf9fca414 100644 --- a/responses_api_agents/remote_agent/app.py +++ b/responses_api_agents/remote_agent/app.py @@ -356,7 +356,9 @@ async def _post_remote_responses( # response.ok is `status < 400`; reject 3xx explicitly (redirects are not followed). if not response.ok or response.status >= 300: if is_global_aiohttp_client_request_debug_enabled(): - print(f"[remote_agent] full HTTP {response.status} body: {content.decode(errors='replace')}", flush=True) + print( + f"[remote_agent] full HTTP {response.status} body: {content.decode(errors='replace')}", flush=True + ) location = response.headers.get("Location", "") return ( None, diff --git a/responses_api_agents/remote_agent/tests/test_app.py b/responses_api_agents/remote_agent/tests/test_app.py index 255ffa608c..4c5861ca76 100644 --- a/responses_api_agents/remote_agent/tests/test_app.py +++ b/responses_api_agents/remote_agent/tests/test_app.py @@ -187,7 +187,13 @@ def test_agent_base_url_normalized(self) -> None: @pytest.mark.parametrize( "bad_url", - ["ftp://h:1", "localhost:9000", "http://h:1?token=abc", "http://h:1#frag", "http://user:pass@h:1"], + [ + "ftp://h:1", + "localhost:9000", + "http://h:1?token=abc", + "http://h:1#frag", + "http://user:pass@h:1", # pragma: allowlist secret + ], ) def test_agent_base_url_rejected(self, bad_url: str) -> None: with pytest.raises(ValidationError): @@ -195,7 +201,7 @@ def test_agent_base_url_rejected(self, bad_url: str) -> None: def test_normalize_remote_url_never_echoes_credentials(self) -> None: with pytest.raises(ValueError) as exc_info: - normalize_remote_url("http://user:hunter2@h:1") + normalize_remote_url("http://user:hunter2@h:1") # pragma: allowlist secret assert "hunter2" not in str(exc_info.value) def test_cookie_header_value_shapes(self) -> None: From 97d019b636fa4f2c934e7b2104f45eac141a1b27 Mon Sep 17 00:00:00 2001 From: adil-a Date: Wed, 29 Jul 2026 23:45:37 +0000 Subject: [PATCH 13/25] refactor!: Gym drives the tool loop; resources server no longer exposed to the service Per review: the remote service is now called like a model. Each POST to {agent_base_url}/v1/responses carries the conversation so far; the service returns function_call items (unpaired = execute on the resources server, paired with its own function_call_output = internal record, passed through) or a final assistant message. The loop is simple_agent's responses() with the model hop swapped for the hardened remote POST; run() mirrors simple_agent's seed -> self-post /v1/responses -> verify inside the existing never-raise sentinel machinery. tools_mode, advertised_resources_url, the forwarding headers, and cookie serialization are removed: tool execution, session cookies, and verifier_metadata never leave Gym. The service's own cookies are round-tripped per call so it can keep per-rollout state. Terminal failure classification crosses the HTTP self-post boundary by exception name. Adds max_steps; 51 tests. Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- responses_api_agents/remote_agent/app.py | 459 ++++++----- .../remote_agent/tests/test_app.py | 739 ++++++++++-------- 2 files changed, 692 insertions(+), 506 deletions(-) diff --git a/responses_api_agents/remote_agent/app.py b/responses_api_agents/remote_agent/app.py index 6cf9fca414..28432cca3a 100644 --- a/responses_api_agents/remote_agent/app.py +++ b/responses_api_agents/remote_agent/app.py @@ -12,14 +12,20 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -"""Thin agent server that brokers rollouts to a user-hosted remote agent service. +"""Agent server that drives a user-hosted remote agent service through Gym's tool loop. -The remote service implements ONE endpoint: ``POST {agent_base_url}/v1/responses``. -It receives the row's ``responses_create_params`` (never ``verifier_metadata`` — the -answer key stays inside Gym), runs its own agent loop with its own model and tools, -and returns a single finished Responses API trajectory. This server owns the Gym -side of the rollout: it seeds the session, holds the session cookies, verifies on -the resources server, and returns the verify response from ``/run``. +The remote service implements ONE endpoint, ``POST {agent_base_url}/v1/responses``, and is +called like a model: each call it receives the conversation so far (the row's +``responses_create_params`` with the accumulated output and tool results appended to +``input``) and returns a Responses API object. To have Gym execute a tool from the +environment, it returns a ``function_call`` item WITHOUT a matching +``function_call_output``; tool calls it already answered itself (its own internal tools) +ride along as paired call+output items and are passed through untouched. Gym runs the +loop — copied from simple_agent — executing unpaired calls against the resources server +and re-posting until the service returns a final assistant message. + +The resources server is never exposed to the service: tool execution, session cookies, +and ``verifier_metadata`` all stay inside Gym. Failures never raise out of ``/run``: every failure (remote endpoint down, timeout, malformed reply, seed/verify errors) becomes a reward-0 verify response carrying the @@ -28,13 +34,14 @@ """ import asyncio +import json from traceback import print_exc -from typing import Any, Dict, Literal, Optional, Tuple +from typing import Any, Dict, List, Optional, Tuple from urllib.parse import urlparse import orjson from aiohttp import ClientOSError, ClientTimeout, ServerDisconnectedError -from fastapi import Body, Request +from fastapi import Body, Request, Response from pydantic import ConfigDict, PrivateAttr, field_validator from pydantic import ValidationError as PydanticValidationError @@ -47,12 +54,18 @@ from nemo_gym.base_responses_api_agent import BaseResponsesAPIAgentConfig, SimpleResponsesAPIAgent from nemo_gym.config_types import ResourcesServerRef from nemo_gym.global_config import SKILLS_REF_KEY_NAME -from nemo_gym.openai_utils import NeMoGymResponse +from nemo_gym.openai_utils import ( + NeMoGymEasyInputMessage, + NeMoGymFunctionCallOutput, + NeMoGymResponse, + NeMoGymResponseCreateParamsNonStreaming, + NeMoGymResponseFunctionToolCall, + NeMoGymResponseOutputMessage, +) from nemo_gym.rollout_collection import NG_FAILURE_CLASS_KEY, NG_NO_PERSIST_KEY, NG_TERMINAL_KEY from nemo_gym.server_utils import ( get_global_aiohttp_client, get_response_json, - get_server_url, is_global_aiohttp_client_request_debug_enabled, raise_for_status, ) @@ -60,14 +73,6 @@ REMOTE_AGENT_FAILURE_CLASS = "remote_agent_error" -# Header names of the session-forwarding contract (tools_mode="forward"): the remote -# service echoes the cookie on every resources-server tool call it makes. The URL is -# re-sent per request (rather than configured remote-side) because Gym assigns servers -# random ports on every `gym env start` — a statically configured address goes stale on -# every restart; the cookie is minted per rollout and has no static equivalent at all. -RESOURCES_URL_HEADER = "X-NeMo-Gym-Resources-Server-Url" -SESSION_COOKIE_HEADER = "X-NeMo-Gym-Session-Cookie" - _REMOTE_MAX_TRIES = 3 _REMOTE_RETRY_SLEEP_SECS = 0.5 _FAILURE_PRINT_HEAD = 5 @@ -80,6 +85,19 @@ _RESERVED_RESULT_KEYS = ("reward", "response", "error", NG_FAILURE_CLASS_KEY, NG_NO_PERSIST_KEY, NG_TERMINAL_KEY) +class RemoteAgentError(RuntimeError): + """A rollout-level failure from the remote hop; retryable on resume.""" + + +class RemoteAgentTerminalError(RemoteAgentError): + """A failure that will not fix itself on retry (e.g. an invalid response shape). + + run() reaches responses() over an HTTP self-post, so this class's NAME is the wire + contract: the exception middleware serializes it into the 500 body and run() matches + the name string to set the terminal routing flag. + """ + + def normalize_remote_url(url: str) -> str: """Validate the remote service URL and strip any trailing slash.""" normalized = url.strip().rstrip("/") @@ -102,51 +120,25 @@ def normalize_remote_url(url: str) -> str: return normalized -def cookie_header_value(cookies: Any) -> Optional[str]: - """Serialize seed-session cookies (SimpleCookie morsels or a plain dict) into a Cookie header.""" - if not cookies: - return None - pairs = [] - for key, value in cookies.items(): - pairs.append(f"{key}={getattr(value, 'value', value)}") - return "; ".join(pairs) if pairs else None - - class RemoteAgentConfig(BaseResponsesAPIAgentConfig): agent_base_url: str resources_server: ResourcesServerRef concurrency: int = 32 + # Per-call bound on one POST to the remote service; a rollout makes one call per loop step. remote_responses_timeout_secs: float = 1800.0 - # Bound on the whole /run body (seed + remote call + verify), applied after the - # semaphore is acquired so queue wait does not count against it. The collector's - # named-agent hop carries no timeout of its own; this is the only wallclock bound. + # Bound on the whole /run body (seed + the full agent/tool loop + verify), applied after + # the semaphore is acquired so queue wait does not count against it. The collector's + # named-agent hop carries no timeout of its own; this is the only whole-rollout bound. run_timeout_secs: float = 2100.0 - # Who serves the tools a dataset declares: - # "refuse" — nobody can: reject tool-declaring tasks up front (terminal failure row) - # instead of letting verify() score silent zeros against untouched state. - # "forward" — Gym does: send the resources-server URL and session cookie as headers on - # every remote request; the service echoes the cookie on each tool call. - # "remote" — the service does: it implements the declared tools itself; nothing is - # forwarded and the guard stands down. - tools_mode: Literal["refuse", "forward", "remote"] = "refuse" - # The resources-server URL advertised to the remote service with tools_mode="forward". - # The default (resolved from the global config) is the BIND address — typically - # 127.0.0.1, unreachable from another machine. Set this to the externally reachable URL - # when the remote service runs off-host (bind vs. advertise can genuinely differ: NAT, - # tunnels, load balancers). This only changes the header string; making the address - # actually route to the resources server is the operator's job. - advertised_resources_url: Optional[str] = None + # Maximum loop steps (remote calls) per rollout; None leaves run_timeout_secs as the + # only bound, matching simple_agent's default. + max_steps: Optional[int] = None @field_validator("agent_base_url") @classmethod def _normalize_agent_base_url(cls, value: str) -> str: return normalize_remote_url(value) - @field_validator("advertised_resources_url") - @classmethod - def _normalize_advertised_resources_url(cls, value: Optional[str]) -> Optional[str]: - return normalize_remote_url(value) if value else value - class RemoteAgentRunRequest(BaseRunRequest): model_config = ConfigDict(extra="allow") @@ -166,11 +158,202 @@ class RemoteAgent(SimpleResponsesAPIAgent): def model_post_init(self, __context: Any) -> None: self.sem = asyncio.Semaphore(self.config.concurrency) - async def responses(self, body=Body()) -> NeMoGymResponse: - raise NotImplementedError( - "RemoteAgent brokers a remote service; drive it through /run. The remote service's " - "own /v1/responses is called by run(), not exposed here." - ) + async def responses( + self, + request: Request, + response: Response, + body: NeMoGymResponseCreateParamsNonStreaming = Body(), + ) -> NeMoGymResponse: + # simple_agent's loop with the model hop swapped for the remote service. The service + # is called like a model: conversation in, Responses object out. Divergences from + # simple_agent are marked; everything else is kept verbatim. + body = body.model_copy(deep=True) + + if isinstance(body.input, str): + body.input = [NeMoGymEasyInputMessage(role="user", content=body.input)] + + new_outputs = [] + usage = None + step = 0 + agent_server_cookies = None # the service's own cookies, round-tripped so it can keep per-rollout state + resources_server_cookies = request.cookies # update the cookies on every resources server response + + while True: + step += 1 + new_body = body.model_copy(update={"input": body.input + new_outputs}) + + # Divergence: hardened POST to the external service instead of the model server. + agent_response, agent_server_cookies = await self._post_agent_responses(new_body, agent_server_cookies) + + output = agent_response.output + new_outputs.extend(output) + + if not usage: + usage = agent_response.usage + agent_response.usage = None + + if usage and agent_response.usage: + usage.input_tokens += agent_response.usage.input_tokens + usage.output_tokens += agent_response.usage.output_tokens + usage.total_tokens += agent_response.usage.total_tokens + + # TODO support more advanced token details + usage.input_tokens_details.cached_tokens = 0 + usage.output_tokens_details.reasoning_tokens = 0 + + if agent_response.incomplete_details: + break + + # Divergence: execute only UNPAIRED calls. A call the service already answered + # itself (matching function_call_output in the same response) is its own internal + # tool record — it passes through into the trajectory untouched. + answered_call_ids = {o.call_id for o in output if o.type == "function_call_output"} + all_fn_calls: List[NeMoGymResponseFunctionToolCall] = [ + o for o in output if o.type == "function_call" and o.call_id not in answered_call_ids + ] + all_output_messages: List[NeMoGymResponseOutputMessage] = [ + o for o in output if o.type == "message" and o.role == "assistant" + ] + if not all_fn_calls and all_output_messages: + break + + for output_function_call in all_fn_calls: + try: + parsed_arguments = json.loads(output_function_call.arguments) + except (json.JSONDecodeError, TypeError) as e: + # The service produced malformed tool-call arguments. Surface the + # error back as a tool response so the rollout can continue + # (or terminate with a low reward) instead of crashing the + # whole batch on json.loads. + tool_response = NeMoGymFunctionCallOutput( + type="function_call_output", + call_id=output_function_call.call_id, + # Use repr(e) so the exception type name is always + # included even when str(e) would be empty. + output=json.dumps({"error": f"Invalid tool call arguments: {e!r}"}), + ) + new_outputs.append(tool_response) + continue + + api_response = await self.server_client.post( + server_name=self.config.resources_server.name, + url_path=f"/{output_function_call.name}", + json=parsed_arguments, + cookies=resources_server_cookies, + ) + # We don't raise for status here since it's a valid return for the API to error e.g. if the service asks for an unknown tool or passes an invalid call. + resources_server_cookies = api_response.cookies + + tool_response = NeMoGymFunctionCallOutput( + type="function_call_output", + call_id=output_function_call.call_id, + output=(await api_response.content.read()).decode(), + ) + new_outputs.append(tool_response) + + # Check if max steps is not None and if we have exhausted it. + if self.config.max_steps and step >= self.config.max_steps: + break + + # Propagate any extra cookies necessary for downstream verification. The service's + # own cookies are its private session and deliberately stay out of the Gym side. + for k, v in resources_server_cookies.items(): + response.set_cookie(k, v) + + agent_response.output = new_outputs + agent_response.usage = usage + return agent_response + + async def _post_agent_responses( + self, new_body: NeMoGymResponseCreateParamsNonStreaming, cookies: Optional[Dict[str, str]] + ) -> Tuple[NeMoGymResponse, Dict[str, str]]: + """One hardened POST to the remote service. Returns (validated response, its cookies).""" + remote_url = f"{self.config.agent_base_url}/v1/responses" + client = get_global_aiohttp_client() + # exclude_unset keeps the wire payload to the fields the dataset row (and the loop) + # actually set, never materialized None defaults. + data = orjson.dumps(new_body.model_dump(exclude_unset=True)) + headers = {"Content-Type": "application/json"} + timeout = ClientTimeout(total=self.config.remote_responses_timeout_secs) + + response = None + last_connect_error: Optional[BaseException] = None + for num_try in range(1, _REMOTE_MAX_TRIES + 1): + try: + # Never follow redirects: aiohttp re-issues 301/302/303 as a body-less GET and + # re-sends 307/308 to an address the user never configured; fail with the 3xx. + response = await client.request( + "POST", + remote_url, + data=data, + headers=headers, + cookies=cookies or {}, + timeout=timeout, + allow_redirects=False, + ) + break + except (ClientOSError, ServerDisconnectedError) as e: + # Refused/reset (ClientOSError) and keepalive races (ServerDisconnectedError) + # are transient connection noise; everything else fails fast. + last_connect_error = e + if num_try < _REMOTE_MAX_TRIES: + await asyncio.sleep(_REMOTE_RETRY_SLEEP_SECS) + except asyncio.TimeoutError: + raise RemoteAgentError( + f"remote /v1/responses timed out after {self.config.remote_responses_timeout_secs}s " + "(remote_responses_timeout_secs; raise it if agent calls legitimately run longer)" + ) from None + except Exception as e: + if is_global_aiohttp_client_request_debug_enabled(): + print_exc() + raise RemoteAgentError(f"{type(e).__name__}: {e}") from e + if response is None: + raise RemoteAgentError( + f"could not reach the remote service after {_REMOTE_MAX_TRIES} tries " + f"({type(last_connect_error).__name__}: {last_connect_error}). " + f"Is your service running at {self.config.agent_base_url}?" + ) + + # client.request() returns once headers arrive; the body read can still raise + # (mid-body disconnect, deadline) and must honor the same never-raise contract. + try: + content = await response.read() + except Exception as e: + if is_global_aiohttp_client_request_debug_enabled(): + print_exc() + raise RemoteAgentError(f"reading the response body failed: {type(e).__name__}: {e}") from e + # response.ok is `status < 400`; reject 3xx explicitly (redirects are not followed). + if not response.ok or response.status >= 300: + if is_global_aiohttp_client_request_debug_enabled(): + print( + f"[remote_agent] full HTTP {response.status} body: {content.decode(errors='replace')}", flush=True + ) + location = response.headers.get("Location", "") + raise RemoteAgentError( + f"HTTP {response.status}" + + (f" (redirect to {location}; fix agent_base_url to point at the final address)" if location else "") + + f": {content[:500].decode(errors='replace')}" + ) + try: + result = orjson.loads(content) + except orjson.JSONDecodeError as e: + raise RemoteAgentError(f"response is not valid JSON: {e}") from e + if not isinstance(result, dict): + raise RemoteAgentError(f"expected a JSON object from /v1/responses, got {type(result).__name__}") + + try: + validated = NeMoGymResponse.model_validate(result) + except PydanticValidationError as e: + if is_global_aiohttp_client_request_debug_enabled(): + print(f"[remote_agent] full validation error: {e}", flush=True) + # A shape error will not fix itself on retry. + raise RemoteAgentTerminalError( + f"remote service returned an invalid Responses API object: {str(e)[:500]}" + ) from e + + merged_cookies = dict(cookies or {}) + merged_cookies.update({k: morsel.value for k, morsel in response.cookies.items()}) + return validated, merged_cookies async def run(self, request: Request, body: RemoteAgentRunRequest = Body()) -> RemoteAgentVerifyResponse: record = self._sanitized_record(body) @@ -183,7 +366,7 @@ async def run(self, request: Request, body: RemoteAgentRunRequest = Body()) -> R return self._failure_response( record, f"/run exceeded run_timeout_secs={self.config.run_timeout_secs}s " - "(seed + remote /v1/responses + verify)", + "(seed + agent/tool loop + verify)", ) except Exception as e: # noqa: BLE001 -- never 500; one task must not abort the whole collection return self._failure_response(record, f"unexpected error: {type(e).__name__}: {e}") @@ -200,12 +383,7 @@ async def _run_once( # body and record are two views of the same row: `record` (sanitized dict, computed # before run()'s try so failure rows can be built in ANY error state) feeds the Gym # hops; `body` (typed model) is kept solely because exclude_unset information — which - # fields the dataset actually set — exists only on the model, and the remote wire - # payload must not carry materialized None defaults. - guard_error = self._tools_guard_error(record) - if guard_error: - return self._failure_response(record, guard_error, terminal=True) - + # fields the dataset actually set — exists only on the model. if record.get(SKILLS_REF_KEY_NAME): self._throttled_warn( "skills_ref", @@ -229,32 +407,34 @@ async def _run_once( record, f"/seed_session on the resources server failed: {type(e).__name__}: {e}" ) - # 2. One POST to the remote service: create-params in, finished trajectory out. - # exclude_unset keeps the wire payload to exactly what the dataset row carried. - remote_params = body.responses_create_params.model_dump(exclude_unset=True) - remote_result, remote_error, terminal = await self._post_remote_responses(remote_params, cookies) - if remote_error is not None: - return self._failure_response(record, remote_error, terminal=terminal) - + # 2. Self-post to our own /v1/responses, which drives the agent/tool loop. try: - response = NeMoGymResponse.model_validate(remote_result) - except PydanticValidationError as e: - if is_global_aiohttp_client_request_debug_enabled(): - print(f"[remote_agent] full validation error: {e}", flush=True) - # A shape error will not fix itself on retry. - return self._failure_response( - record, - f"remote service returned an invalid Responses API object: {str(e)[:500]}", - terminal=True, + loop_response = await self.server_client.post( + server_name=self.config.name, + url_path=self.url_path_for_run("/v1/responses", body), + json=body.responses_create_params, + cookies=cookies, ) - self._warn_on_response_quality(response) + await raise_for_status(loop_response) + response_json = await get_response_json(loop_response) + cookies = loop_response.cookies + except Exception as e: + content = getattr(e, "response_content", b"") + text = content.decode(errors="replace") if isinstance(content, (bytes, bytearray)) else str(content) + # Terminal classification crosses the HTTP self-post boundary by exception NAME: + # the middleware serialized the raised RemoteAgentTerminalError into the 500 body. + terminal = "RemoteAgentTerminalError" in text + detail = text or f"{type(e).__name__}: {e}" + return self._failure_response(record, f"agent loop failed: {detail[:500]}", terminal=terminal) + + self._warn_on_response_quality(response_json) # 3. Verify on the SAME session; the verify response (reward included) is /run's result. try: verify_response = await self.server_client.post( server_name=self.config.resources_server.name, url_path="/verify", - json=record | {"response": response.model_dump(mode="json")}, + json=record | {"response": response_json}, cookies=cookies, ) await raise_for_status(verify_response) @@ -264,117 +444,6 @@ async def _run_once( return RemoteAgentVerifyResponse.model_validate(verify_json) - def _tools_guard_error(self, record: Dict[str, Any]) -> Optional[str]: - """Refuse tool-declaring tasks the remote service cannot serve, instead of scoring silent zeros. - - A dataset that declares tools expects them to be called during the rollout. Without - forward_session the remote service has no session cookie, so any per-session state - those tools mutate stays untouched and verify() scores 0 on every row. - """ - declared_tools = (record.get("responses_create_params") or {}).get("tools") - if declared_tools and self.config.tools_mode == "refuse": - return ( - 'the task declares tools but tools_mode="refuse" (the default). Set ' - 'tools_mode="forward" so the remote service can call Gym-hosted tools with the ' - 'session cookie, or tools_mode="remote" if the service implements the declared ' - "tools itself." - ) - return None - - async def _post_remote_responses( - self, remote_params: Dict[str, Any], cookies: Any - ) -> Tuple[Optional[Dict], Optional[str], bool]: - """POST create-params to the remote /v1/responses. Returns (result, error, terminal).""" - remote_url = f"{self.config.agent_base_url}/v1/responses" - client = get_global_aiohttp_client() - data = orjson.dumps(remote_params) - headers = {"Content-Type": "application/json"} - if self.config.tools_mode == "forward": - resources_url = self.config.advertised_resources_url or get_server_url(self.config.resources_server.name) - advertised_host = urlparse(resources_url).hostname or "" - remote_host = urlparse(self.config.agent_base_url).hostname or "" - # 0.0.0.0 is a bind address, not a routable one: from the remote machine it - # resolves to that machine's own loopback, exactly like 127.0.0.1 would. - local_only_hosts = ("127.0.0.1", "localhost", "0.0.0.0") - if advertised_host in local_only_hosts and remote_host not in local_only_hosts: - self._throttled_warn( - "loopback_resources_url", - f"WARNING: forwarding resources-server URL {resources_url} (an address other machines " - f"cannot reach) to the off-host remote service at {self.config.agent_base_url}. Its tool " - "calls will not reach Gym; set advertised_resources_url to an externally reachable URL.", - ) - headers[RESOURCES_URL_HEADER] = resources_url - session_cookie = cookie_header_value(cookies) - if session_cookie: - headers[SESSION_COOKIE_HEADER] = session_cookie - timeout = ClientTimeout(total=self.config.remote_responses_timeout_secs) - - response = None - last_connect_error: Optional[BaseException] = None - for num_try in range(1, _REMOTE_MAX_TRIES + 1): - try: - # Never follow redirects: aiohttp re-issues 301/302/303 as a body-less GET and - # re-sends 307/308 to an address the user never configured; fail with the 3xx. - response = await client.request( - "POST", remote_url, data=data, headers=headers, timeout=timeout, allow_redirects=False - ) - break - except (ClientOSError, ServerDisconnectedError) as e: - # Refused/reset (ClientOSError) and keepalive races (ServerDisconnectedError) - # are transient connection noise; everything else fails fast. - last_connect_error = e - if num_try < _REMOTE_MAX_TRIES: - await asyncio.sleep(_REMOTE_RETRY_SLEEP_SECS) - except asyncio.TimeoutError: - return ( - None, - f"remote /v1/responses timed out after {self.config.remote_responses_timeout_secs}s " - "(remote_responses_timeout_secs; raise it if rollouts legitimately run longer)", - False, - ) - except Exception as e: - if is_global_aiohttp_client_request_debug_enabled(): - print_exc() - return None, f"{type(e).__name__}: {e}", False - if response is None: - return ( - None, - f"could not reach the remote service after {_REMOTE_MAX_TRIES} tries " - f"({type(last_connect_error).__name__}: {last_connect_error}). " - f"Is your service running at {self.config.agent_base_url}?", - False, - ) - - # client.request() returns once headers arrive; the body read can still raise - # (mid-body disconnect, deadline) and must honor the same never-raise contract. - try: - content = await response.read() - except Exception as e: - if is_global_aiohttp_client_request_debug_enabled(): - print_exc() - return None, f"reading the response body failed: {type(e).__name__}: {e}", False - # response.ok is `status < 400`; reject 3xx explicitly (redirects are not followed). - if not response.ok or response.status >= 300: - if is_global_aiohttp_client_request_debug_enabled(): - print( - f"[remote_agent] full HTTP {response.status} body: {content.decode(errors='replace')}", flush=True - ) - location = response.headers.get("Location", "") - return ( - None, - f"HTTP {response.status}" - + (f" (redirect to {location}; fix agent_base_url to point at the final address)" if location else "") - + f": {content[:500].decode(errors='replace')}", - False, - ) - try: - result = orjson.loads(content) - except orjson.JSONDecodeError as e: - return None, f"response is not valid JSON: {e}", False - if not isinstance(result, dict): - return None, f"expected a JSON object from /v1/responses, got {type(result).__name__}", False - return result, None, False - def _throttled_warn(self, key: str, message: str) -> None: """Per-key sampled warning: the first few occurrences, then every 100th. At production concurrency an unthrottled per-rollout print garbles the collector's progress bar.""" @@ -383,8 +452,8 @@ def _throttled_warn(self, key: str, message: str) -> None: if n <= _FAILURE_PRINT_HEAD or n % _FAILURE_PRINT_INTERVAL == 0: print(f"{message} (occurrence #{n})", flush=True) - def _warn_on_response_quality(self, response: NeMoGymResponse) -> None: - if response.usage is None: + def _warn_on_response_quality(self, response_json: Dict[str, Any]) -> None: + if not response_json.get("usage"): self._throttled_warn( "missing_usage", "WARNING: the remote response carries no usage; token metrics for this agent will be " diff --git a/responses_api_agents/remote_agent/tests/test_app.py b/responses_api_agents/remote_agent/tests/test_app.py index 4c5861ca76..974af3ccd5 100644 --- a/responses_api_agents/remote_agent/tests/test_app.py +++ b/responses_api_agents/remote_agent/tests/test_app.py @@ -14,54 +14,87 @@ # limitations under the License. import asyncio import json +from http.cookies import SimpleCookie from unittest.mock import AsyncMock, MagicMock import orjson import pytest -from aiohttp import ClientConnectorError, ClientPayloadError, ServerDisconnectedError -from pydantic import ValidationError +from aiohttp import ClientConnectorError, ClientPayloadError, ClientResponseError, ServerDisconnectedError +from fastapi import Response +from pydantic import BaseModel, ValidationError import responses_api_agents.remote_agent.app as remote_agent_app from nemo_gym.config_types import ResourcesServerRef +from nemo_gym.openai_utils import NeMoGymResponseCreateParamsNonStreaming from nemo_gym.rollout_collection import NG_FAILURE_CLASS_KEY, NG_NO_PERSIST_KEY, NG_TERMINAL_KEY from nemo_gym.server_utils import ServerClient from responses_api_agents.remote_agent.app import ( REMOTE_AGENT_FAILURE_CLASS, - RESOURCES_URL_HEADER, - SESSION_COOKIE_HEADER, RemoteAgent, RemoteAgentConfig, RemoteAgentRunRequest, - cookie_header_value, normalize_remote_url, ) -_MINIMAL_TRAJECTORY = { - "id": "traj_1", - "created_at": 1.0, - "model": "their-model", - "object": "response", - "output": [ - { - "type": "message", - "role": "assistant", - "status": "completed", - "id": "msg_1", - "content": [{"type": "output_text", "text": "the answer is 42", "annotations": []}], +def msg(text: str, item_id: str = "msg_1") -> dict: + return { + "type": "message", + "role": "assistant", + "status": "completed", + "id": item_id, + "content": [{"type": "output_text", "text": text, "annotations": []}], + } + + +def fn_call(call_id: str, name: str, arguments: str) -> dict: + return {"type": "function_call", "call_id": call_id, "name": name, "arguments": arguments, "id": f"fc_{call_id}"} + + +def fn_output(call_id: str, output: str) -> dict: + return {"type": "function_call_output", "call_id": call_id, "output": output} + + +def traj(output: list, usage: dict | None = "default") -> dict: + t = { + "id": "traj_1", + "created_at": 1.0, + "model": "their-model", + "object": "response", + "output": output, + "parallel_tool_calls": False, + "tools": [], + "tool_choice": "auto", + } + if usage == "default": + usage = { + "input_tokens": 10, + "output_tokens": 5, + "total_tokens": 15, + "input_tokens_details": {"cached_tokens": 0}, + "output_tokens_details": {"reasoning_tokens": 0}, } - ], - "parallel_tool_calls": False, - "tools": [], - "tool_choice": "auto", - "usage": { - "input_tokens": 10, - "output_tokens": 5, - "total_tokens": 15, - "input_tokens_details": {"cached_tokens": 0}, - "output_tokens_details": {"reasoning_tokens": 0}, - }, -} + if usage is not None: + t["usage"] = usage + return t + + +_MINIMAL_TRAJECTORY = traj([msg("the answer is 42")]) + +_COUNTER_TOOLS = [ + { + "type": "function", + "name": "increment_counter", + "parameters": { + "type": "object", + "properties": {"count": {"type": "integer", "description": ""}}, + "required": ["count"], + "additionalProperties": False, + }, + "strict": True, + "description": "", + } +] def make_config(**overrides) -> RemoteAgentConfig: @@ -77,13 +110,6 @@ def make_config(**overrides) -> RemoteAgentConfig: return RemoteAgentConfig(**fields) -def make_agent(server_client=None, **config_overrides) -> RemoteAgent: - return RemoteAgent( - config=make_config(**config_overrides), - server_client=server_client or MagicMock(spec=ServerClient), - ) - - def make_row(tools=None, **extras) -> dict: row = { "responses_create_params": {"input": [{"role": "user", "content": "what is 6 x 7?"}]}, @@ -104,11 +130,14 @@ def make_request(cookies=None) -> MagicMock: class FakeRemoteResponse: """Stands in for an aiohttp ClientResponse from the remote service.""" - def __init__(self, status: int, content: bytes, headers=None, read_exc=None): + def __init__(self, status: int, content: bytes, headers=None, read_exc=None, set_cookies=None): self.status = status self._content = content self.headers = headers or {} self._read_exc = read_exc + self.cookies = SimpleCookie() + for k, v in (set_cookies or {}).items(): + self.cookies[k] = v @property def ok(self) -> bool: @@ -143,8 +172,10 @@ async def _read(): return reader def raise_for_status(self): + # Mirror aiohttp so nemo_gym.raise_for_status attaches response_content, the + # channel run() reads middleware-serialized errors (incl. terminal names) from. if not self.ok: - raise RuntimeError(f"HTTP {self.status}") + raise ClientResponseError(request_info=MagicMock(), history=(), status=self.status, message="error") async def read(self) -> bytes: return orjson.dumps(self._body) @@ -158,8 +189,41 @@ def mock_remote(monkeypatch: pytest.MonkeyPatch, request_mock: AsyncMock) -> Mag return client -def seed_verify_server_client(verify_body=None, seed_cookies=None, seed_status=200, verify_status=200): - """A ServerClient mock that answers /seed_session and /verify.""" +def scripted_service(*turns): + """AsyncMock remote service returning one canned trajectory per call, recording payloads.""" + received = [] + + async def handler(method, url, data=None, headers=None, cookies=None, **kwargs): + received.append({"payload": orjson.loads(data), "cookies": dict(cookies or {})}) + turn = turns[min(len(received) - 1, len(turns) - 1)] + if isinstance(turn, FakeRemoteResponse): + return turn + return FakeRemoteResponse(200, orjson.dumps(turn)) + + request_mock = AsyncMock(side_effect=handler) + request_mock.received = received + return request_mock + + +def make_agent(server_client=None, **config_overrides) -> RemoteAgent: + return RemoteAgent( + config=make_config(**config_overrides), + server_client=server_client or MagicMock(spec=ServerClient), + ) + + +def wire_gym( + agent: RemoteAgent, + verify_body=None, + seed_cookies=None, + seed_status=200, + verify_status=200, + tool_handler=None, +): + """A ServerClient mock that emulates the Gym side: seed_session and verify on the + resources server, tool routes via tool_handler, and — the load-bearing part — the + /v1/responses self-post routed into the agent's REAL responses() with the exception + middleware emulated (exceptions become a 500 body carrying repr(e)).""" calls = [] async def _post(server_name, url_path, json=None, cookies=None, **kwargs): @@ -169,14 +233,47 @@ async def _post(server_name, url_path, json=None, cookies=None, **kwargs): if url_path == "/verify": body = verify_body if verify_body is not None else (json | {"reward": 1.0}) return FakeServerClientResponse(body, status=verify_status) + if url_path.endswith("/v1/responses"): + wire = json.model_dump(exclude_unset=True) if isinstance(json, BaseModel) else json + params = NeMoGymResponseCreateParamsNonStreaming.model_validate(wire) + fastapi_response = Response() + try: + result = await agent.responses(make_request(dict(cookies or {})), fastapi_response, params) + except Exception as e: # noqa: BLE001 -- emulate SimpleServer's exception middleware + return FakeServerClientResponse({"error": repr(e)}, status=500) + out_cookies = SimpleCookie() + for header_value in fastapi_response.headers.getlist("set-cookie"): + out_cookies.load(header_value) + return FakeServerClientResponse( + result.model_dump(mode="json"), cookies={k: m.value for k, m in out_cookies.items()} + ) + if tool_handler is not None: + return await tool_handler(url_path, json, cookies) return FakeServerClientResponse({}, status=200) server_client = MagicMock(spec=ServerClient) server_client.post = AsyncMock(side_effect=_post) server_client.calls = calls + agent.server_client = server_client return server_client +def make_wired_agent(monkeypatch, request_mock, *, tool_handler=None, verify_body=None, **kwargs): + client = mock_remote(monkeypatch, request_mock) + agent = make_agent( + **{k: v for k, v in kwargs.items() if k not in ("seed_cookies", "seed_status", "verify_status")} + ) + server_client = wire_gym( + agent, + verify_body=verify_body, + seed_cookies=kwargs.get("seed_cookies"), + seed_status=kwargs.get("seed_status", 200), + verify_status=kwargs.get("verify_status", 200), + tool_handler=tool_handler, + ) + return agent, client, server_client + + class TestConfig: def test_sanity_construct_and_semaphore(self) -> None: agent = make_agent(concurrency=7) @@ -185,6 +282,9 @@ def test_sanity_construct_and_semaphore(self) -> None: def test_agent_base_url_normalized(self) -> None: assert make_config(agent_base_url="http://localhost:9000/").agent_base_url == "http://localhost:9000" + def test_max_steps_defaults_to_none(self) -> None: + assert make_config().max_steps is None + @pytest.mark.parametrize( "bad_url", [ @@ -204,40 +304,28 @@ def test_normalize_remote_url_never_echoes_credentials(self) -> None: normalize_remote_url("http://user:hunter2@h:1") # pragma: allowlist secret assert "hunter2" not in str(exc_info.value) - def test_cookie_header_value_shapes(self) -> None: - assert cookie_header_value({}) is None - assert cookie_header_value({"a": "1", "b": "2"}) == "a=1; b=2" - morsel = MagicMock() - morsel.value = "xyz" - assert cookie_header_value({"session": morsel}) == "session=xyz" - class TestRunHappyPath: - async def test_seed_then_remote_then_verify(self, monkeypatch: pytest.MonkeyPatch) -> None: - request_mock = AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))) - client = mock_remote(monkeypatch, request_mock) - server_client = seed_verify_server_client(seed_cookies={"session": "s1"}) - agent = make_agent(server_client=server_client) + async def test_seed_then_loop_then_verify(self, monkeypatch: pytest.MonkeyPatch) -> None: + service = scripted_service(_MINIMAL_TRAJECTORY) + agent, client, server_client = make_wired_agent(monkeypatch, service, seed_cookies={"session": "s1"}) row = make_row() result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row)) - # Order and payloads: seed first, remote POST in between, verify last on the seed cookies - assert [c["url_path"] for c in server_client.calls] == ["/seed_session", "/verify"] + paths = [c["url_path"] for c in server_client.calls] + assert paths == ["/seed_session", "/v1/responses", "/verify"] + # Seeded cookies reach the loop and verify; verify carries the trajectory + row keys assert server_client.calls[1]["cookies"] == {"session": "s1"} - assert server_client.calls[1]["json"]["response"]["id"] == "traj_1" - assert server_client.calls[1]["json"]["verifier_metadata"] == {"expected_answer": "42"} + assert server_client.calls[2]["json"]["response"]["id"] == "traj_1" + assert server_client.calls[2]["json"]["verifier_metadata"] == {"expected_answer": "42"} - args, kwargs = client.request.call_args - assert args == ("POST", "http://localhost:9000/v1/responses") # The remote service receives ONLY create-params: no verifier_metadata, no row keys - remote_payload = orjson.loads(kwargs["data"]) - assert remote_payload == row["responses_create_params"] - assert kwargs["allow_redirects"] is False - assert kwargs["timeout"].total == 1800.0 - # No session forwarding by default - assert RESOURCES_URL_HEADER not in kwargs["headers"] - assert SESSION_COOKIE_HEADER not in kwargs["headers"] + assert service.received[0]["payload"] == row["responses_create_params"] + args, request_kwargs = client.request.call_args + assert args == ("POST", "http://localhost:9000/v1/responses") + assert request_kwargs["allow_redirects"] is False + assert request_kwargs["timeout"].total == 1800.0 dumped = result.model_dump() assert dumped["reward"] == 1.0 @@ -246,10 +334,9 @@ async def test_seed_then_remote_then_verify(self, monkeypatch: pytest.MonkeyPatc assert NG_TERMINAL_KEY not in dumped async def test_verify_extras_pass_through(self, monkeypatch: pytest.MonkeyPatch) -> None: - mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)))) row = make_row() verify_body = row | {"response": _MINIMAL_TRAJECTORY, "reward": 0.5, "grading_notes": "close enough"} - agent = make_agent(server_client=seed_verify_server_client(verify_body=verify_body)) + agent, _, _ = make_wired_agent(monkeypatch, scripted_service(_MINIMAL_TRAJECTORY), verify_body=verify_body) result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row)) @@ -257,11 +344,150 @@ async def test_verify_extras_pass_through(self, monkeypatch: pytest.MonkeyPatch) assert result.reward == 0.5 +class TestAgentToolLoop: + """The Gym-driven loop: the service returns unpaired function_calls as asks; Gym + executes them on the resources server and re-posts the grown conversation.""" + + @staticmethod + async def counter_tool_handler(url_path, body, cookies): + if url_path == "/increment_counter": + return FakeServerClientResponse({"success": True}, cookies={"tool_session": "t1"}) + if url_path == "/get_counter_value": + return FakeServerClientResponse({"count": 6}) + return FakeServerClientResponse({"detail": f"Not Found: {url_path}"}, status=404) + + async def test_multi_turn_tool_execution(self, monkeypatch: pytest.MonkeyPatch) -> None: + service = scripted_service( + traj([fn_call("c1", "increment_counter", '{"count": 3}')]), + traj([msg("done, counter incremented")]), + ) + agent, _, server_client = make_wired_agent(monkeypatch, service, tool_handler=self.counter_tool_handler) + + row = make_row(tools=_COUNTER_TOOLS) + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row)) + + dumped = result.model_dump() + assert NG_FAILURE_CLASS_KEY not in dumped + assert dumped["reward"] == 1.0 + + # Gym executed the tool: one resources-server POST to /increment_counter with parsed args + tool_calls = [c for c in server_client.calls if c["url_path"] == "/increment_counter"] + assert len(tool_calls) == 1 + assert tool_calls[0]["json"] == {"count": 3} + + # Turn 2 payload = original input + call + tool output + second_input = service.received[1]["payload"]["input"] + assert [i.get("type", "message") for i in second_input] == ["message", "function_call", "function_call_output"] + assert second_input[2]["output"] == '{"success":true}' + + # The final trajectory carries the merged conversation + types = [o["type"] for o in dumped["response"]["output"]] + assert types == ["function_call", "function_call_output", "message"] + + async def test_paired_calls_pass_through_unexecuted(self, monkeypatch: pytest.MonkeyPatch) -> None: + # call_a is the service's own internal tool record (paired); call_b is the ask. + service = scripted_service( + traj( + [ + fn_call("call_a", "web_search", '{"q": "counters"}'), + fn_output("call_a", '{"results": []}'), + fn_call("call_b", "increment_counter", '{"count": 3}'), + ] + ), + traj([msg("done")]), + ) + agent, _, server_client = make_wired_agent(monkeypatch, service, tool_handler=self.counter_tool_handler) + + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row(tools=_COUNTER_TOOLS))) + + dumped = result.model_dump() + assert dumped["reward"] == 1.0 + executed = [c["url_path"] for c in server_client.calls if c["url_path"].startswith("/increment")] + assert executed == ["/increment_counter"] + # web_search was never sent to the resources server + assert not any(c["url_path"] == "/web_search" for c in server_client.calls) + # The paired record survives in the final trajectory + types = [o["type"] for o in dumped["response"]["output"]] + assert types.count("function_call") == 2 and types.count("function_call_output") == 2 + + async def test_unknown_tool_error_fed_back_not_raised(self, monkeypatch: pytest.MonkeyPatch) -> None: + service = scripted_service( + traj([fn_call("c1", "web_search", "{}")]), # unpaired ask for a tool the env doesn't serve + traj([msg("recovered")]), + ) + agent, _, _ = make_wired_agent(monkeypatch, service, tool_handler=self.counter_tool_handler) + + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row(tools=_COUNTER_TOOLS))) + + assert NG_FAILURE_CLASS_KEY not in result.model_dump() + # The 404 body came back to the service as the tool output + second_input = service.received[1]["payload"]["input"] + assert "Not Found" in second_input[-1]["output"] + + async def test_malformed_arguments_fed_back(self, monkeypatch: pytest.MonkeyPatch) -> None: + service = scripted_service( + traj([fn_call("c1", "increment_counter", "not json")]), + traj([msg("ok")]), + ) + agent, _, server_client = make_wired_agent(monkeypatch, service, tool_handler=self.counter_tool_handler) + + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row(tools=_COUNTER_TOOLS))) + + assert NG_FAILURE_CLASS_KEY not in result.model_dump() + assert not any(c["url_path"] == "/increment_counter" for c in server_client.calls) + second_input = service.received[1]["payload"]["input"] + assert "Invalid tool call arguments" in second_input[-1]["output"] + + async def test_max_steps_bounds_the_loop(self, monkeypatch: pytest.MonkeyPatch) -> None: + always_ask = traj([fn_call("c1", "increment_counter", '{"count": 1}')]) + service = scripted_service(always_ask, always_ask, always_ask, always_ask) + agent, _, _ = make_wired_agent(monkeypatch, service, tool_handler=self.counter_tool_handler, max_steps=2) + + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row(tools=_COUNTER_TOOLS))) + + assert len(service.received) == 2 + assert NG_FAILURE_CLASS_KEY not in result.model_dump() + + async def test_service_cookies_round_trip(self, monkeypatch: pytest.MonkeyPatch) -> None: + turn1 = FakeRemoteResponse( + 200, + orjson.dumps(traj([fn_call("c1", "increment_counter", '{"count": 1}')])), + set_cookies={"svc_session": "svc1"}, + ) + service = scripted_service(turn1, traj([msg("done")])) + agent, _, _ = make_wired_agent(monkeypatch, service, tool_handler=self.counter_tool_handler) + + await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row(tools=_COUNTER_TOOLS))) + + assert service.received[0]["cookies"] == {} + assert service.received[1]["cookies"] == {"svc_session": "svc1"} + + async def test_usage_accumulates_across_turns(self, monkeypatch: pytest.MonkeyPatch) -> None: + service = scripted_service( + traj([fn_call("c1", "increment_counter", '{"count": 1}')]), + traj([msg("done")]), + ) + agent, _, _ = make_wired_agent(monkeypatch, service, tool_handler=self.counter_tool_handler) + + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row(tools=_COUNTER_TOOLS))) + + usage = result.model_dump()["response"]["usage"] + assert usage["input_tokens"] == 20 and usage["output_tokens"] == 10 and usage["total_tokens"] == 30 + + async def test_string_input_coerced_to_message(self, monkeypatch: pytest.MonkeyPatch) -> None: + service = scripted_service(_MINIMAL_TRAJECTORY) + agent, _, _ = make_wired_agent(monkeypatch, service) + + row = {"responses_create_params": {"input": "just a string"}} + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row)) + + assert NG_FAILURE_CLASS_KEY not in result.model_dump() + assert service.received[0]["payload"]["input"][0]["content"] == "just a string" + + class TestRemoteFailuresBecomeSentinelRows: async def _run(self, monkeypatch, request_mock, **config_overrides): - client = mock_remote(monkeypatch, request_mock) - server_client = seed_verify_server_client() - agent = make_agent(server_client=server_client, **config_overrides) + agent, client, server_client = make_wired_agent(monkeypatch, request_mock, **config_overrides) result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) return client, server_client, result.model_dump() @@ -279,8 +505,8 @@ async def test_connect_exhaustion_after_bounded_retries(self, monkeypatch: pytes assert client.request.call_count == 3 assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS assert "Is your service running at http://localhost:9000?" in result["error"] - # verify is never reached on a failed remote call - assert [c["url_path"] for c in server_client.calls] == ["/seed_session"] + # verify is never reached on a failed loop + assert [c["url_path"] for c in server_client.calls] == ["/seed_session", "/v1/responses"] async def test_disconnect_then_success_retries(self, monkeypatch: pytest.MonkeyPatch) -> None: request_mock = AsyncMock( @@ -332,17 +558,30 @@ async def test_invalid_trajectory_shape_is_terminal_and_skips_verify( monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(bad))) ) assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + # Terminal classification survives the HTTP self-post boundary (exception-name match) assert result[NG_TERMINAL_KEY] is True assert "invalid Responses API object" in result["error"] - assert [c["url_path"] for c in server_client.calls] == ["/seed_session"] + assert [c["url_path"] for c in server_client.calls] == ["/seed_session", "/v1/responses"] + + async def test_mid_loop_failure_becomes_sentinel(self, monkeypatch: pytest.MonkeyPatch) -> None: + # Turn 1 succeeds with a tool ask; turn 2 the service dies — the whole rollout + # must land in the sidecar, not crash the loop. + service = scripted_service( + traj([fn_call("c1", "increment_counter", '{"count": 1}')]), + FakeRemoteResponse(500, b"died mid-rollout"), + ) + agent, _, _ = make_wired_agent(monkeypatch, service, tool_handler=TestAgentToolLoop.counter_tool_handler) + result = ( + await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row(tools=_COUNTER_TOOLS))) + ).model_dump() + assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS + assert "died mid-rollout" in result["error"] class TestGymSideFailuresBecomeSentinelRows: async def test_seed_failure_skips_remote_call(self, monkeypatch: pytest.MonkeyPatch) -> None: - request_mock = AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))) - client = mock_remote(monkeypatch, request_mock) - server_client = seed_verify_server_client(seed_status=500) - agent = make_agent(server_client=server_client) + service = scripted_service(_MINIMAL_TRAJECTORY) + agent, client, _ = make_wired_agent(monkeypatch, service, seed_status=500) result = (await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row()))).model_dump() @@ -351,9 +590,7 @@ async def test_seed_failure_skips_remote_call(self, monkeypatch: pytest.MonkeyPa assert client.request.call_count == 0 async def test_verify_failure(self, monkeypatch: pytest.MonkeyPatch) -> None: - mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)))) - server_client = seed_verify_server_client(verify_status=500) - agent = make_agent(server_client=server_client) + agent, _, _ = make_wired_agent(monkeypatch, scripted_service(_MINIMAL_TRAJECTORY), verify_status=500) result = (await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row()))).model_dump() @@ -361,68 +598,10 @@ async def test_verify_failure(self, monkeypatch: pytest.MonkeyPatch) -> None: assert "/verify" in result["error"] assert result["reward"] == 0.0 - -class TestToolsGuardAndSessionForwarding: - _TOOLS = [ - { - "type": "function", - "name": "increment_counter", - "parameters": { - "type": "object", - "properties": {"count": {"type": "integer", "description": ""}}, - "required": ["count"], - "additionalProperties": False, - }, - "strict": True, - "description": "", - } - ] - - async def test_declared_tools_refused_by_default(self, monkeypatch: pytest.MonkeyPatch) -> None: - client = mock_remote(monkeypatch, AsyncMock()) - server_client = seed_verify_server_client() - agent = make_agent(server_client=server_client) - - row = make_row(tools=self._TOOLS) - result = (await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row))).model_dump() - - assert result[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS - assert result[NG_TERMINAL_KEY] is True - assert "tools_mode" in result["error"] - # Refused before any network traffic - assert client.request.call_count == 0 - assert server_client.calls == [] - - async def test_tools_mode_remote_skips_guard_without_headers(self, monkeypatch: pytest.MonkeyPatch) -> None: - request_mock = AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))) - client = mock_remote(monkeypatch, request_mock) - agent = make_agent(server_client=seed_verify_server_client(), tools_mode="remote") - - result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row(tools=self._TOOLS))) - - assert NG_FAILURE_CLASS_KEY not in result.model_dump() - headers = client.request.call_args.kwargs["headers"] - assert RESOURCES_URL_HEADER not in headers - - async def test_tools_mode_forward_sends_url_and_cookie_headers(self, monkeypatch: pytest.MonkeyPatch) -> None: - request_mock = AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))) - client = mock_remote(monkeypatch, request_mock) - monkeypatch.setattr(remote_agent_app, "get_server_url", lambda name: f"http://resolved-{name}:1234") - server_client = seed_verify_server_client(seed_cookies={"session": "cookie-value"}) - agent = make_agent(server_client=server_client, tools_mode="forward") - - result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row(tools=self._TOOLS))) - - assert NG_FAILURE_CLASS_KEY not in result.model_dump() - headers = client.request.call_args.kwargs["headers"] - assert headers[RESOURCES_URL_HEADER] == "http://resolved-my_env:1234" - assert headers[SESSION_COOKIE_HEADER] == "session=cookie-value" - async def test_skills_ref_warns_and_continues( self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] ) -> None: - mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)))) - agent = make_agent(server_client=seed_verify_server_client()) + agent, _, _ = make_wired_agent(monkeypatch, scripted_service(_MINIMAL_TRAJECTORY)) row = make_row(skills_ref={"path": "/skills", "hash": "abc", "skills": []}) result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row)) @@ -432,26 +611,31 @@ async def test_skills_ref_warns_and_continues( class TestResponseQualityWarnings: - async def _run_with_trajectory(self, monkeypatch, trajectory): - mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(trajectory)))) - agent = make_agent(server_client=seed_verify_server_client()) - return await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) - async def test_missing_usage_warns( self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] ) -> None: - trajectory = dict(_MINIMAL_TRAJECTORY) - trajectory.pop("usage") - result = await self._run_with_trajectory(monkeypatch, trajectory) + agent, _, _ = make_wired_agent(monkeypatch, scripted_service(traj([msg("hi")], usage=None))) + result = await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) assert NG_FAILURE_CLASS_KEY not in result.model_dump() assert "no usage" in capsys.readouterr().out async def test_clean_trajectory_no_warnings( self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] ) -> None: - await self._run_with_trajectory(monkeypatch, _MINIMAL_TRAJECTORY) - out = capsys.readouterr().out - assert "WARNING" not in out + agent, _, _ = make_wired_agent(monkeypatch, scripted_service(_MINIMAL_TRAJECTORY)) + await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) + assert "WARNING" not in capsys.readouterr().out + + async def test_quality_warnings_are_throttled( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: + agent, _, _ = make_wired_agent(monkeypatch, scripted_service(traj([msg("hi")], usage=None))) + + for _ in range(10): + await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) + + # Head of 5, then every 100th: 10 rollouts -> exactly 5 printed warnings + assert capsys.readouterr().out.count("no usage") == 5 class TestRunTimeoutAndSemaphore: @@ -459,8 +643,7 @@ async def test_run_wallclock_bound_becomes_sentinel(self, monkeypatch: pytest.Mo async def slow_request(*args, **kwargs): await asyncio.sleep(30) - mock_remote(monkeypatch, AsyncMock(side_effect=slow_request)) - agent = make_agent(server_client=seed_verify_server_client(), run_timeout_secs=0.05) + agent, _, _ = make_wired_agent(monkeypatch, AsyncMock(side_effect=slow_request), run_timeout_secs=0.05) result = (await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row()))).model_dump() @@ -480,8 +663,7 @@ async def gated_request(*args, **kwargs): in_flight -= 1 return FakeRemoteResponse(500, b"boom") # failure path must release the permit too - mock_remote(monkeypatch, AsyncMock(side_effect=gated_request)) - agent = make_agent(server_client=seed_verify_server_client(), concurrency=2) + agent, _, _ = make_wired_agent(monkeypatch, AsyncMock(side_effect=gated_request), concurrency=2) rows = [RemoteAgentRunRequest.model_validate(make_row()) for _ in range(4)] tasks = [asyncio.create_task(agent.run(make_request(), row)) for row in rows] @@ -493,20 +675,44 @@ async def gated_request(*args, **kwargs): assert all(r.model_dump()[NG_FAILURE_CLASS_KEY] == REMOTE_AGENT_FAILURE_CLASS for r in results) assert agent.sem._value == 2 # every permit released despite 4 failures + async def test_queue_wait_does_not_count_against_run_timeout(self, monkeypatch: pytest.MonkeyPatch) -> None: + release_first = asyncio.Event() + first_seen = asyncio.Event() + call_count = 0 + + async def gated_request(*args, **kwargs): + nonlocal call_count + call_count += 1 + if call_count == 1: + first_seen.set() + await release_first.wait() + return FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)) + + agent, _, _ = make_wired_agent( + monkeypatch, AsyncMock(side_effect=gated_request), concurrency=1, run_timeout_secs=0.5 + ) + + first = asyncio.create_task(agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row()))) + second = asyncio.create_task(agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row()))) + await first_seen.wait() + # Hold the only permit for most of the second task's would-be budget + await asyncio.sleep(0.4) + release_first.set() + results = [r.model_dump() for r in await asyncio.gather(first, second)] + + # If queue wait counted against run_timeout_secs, the second task would time out + assert all(NG_FAILURE_CLASS_KEY not in r for r in results) + class TestRoutes: - def _client_and_mocks(self, monkeypatch, request_mock=None): + def _client(self, monkeypatch, request_mock=None): from fastapi.testclient import TestClient - mock_remote( - monkeypatch, - request_mock or AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))), - ) - agent = make_agent(server_client=seed_verify_server_client()) + agent, _, _ = make_wired_agent(monkeypatch, request_mock or scripted_service(_MINIMAL_TRAJECTORY)) return TestClient(agent.setup_webserver(), raise_server_exceptions=False) def test_run_route_happy_path(self, monkeypatch: pytest.MonkeyPatch) -> None: - client = self._client_and_mocks(monkeypatch) + client = self._client(monkeypatch) response = client.post("/run", json=make_row()) assert response.status_code == 200 assert response.json()["reward"] == 1.0 @@ -514,7 +720,7 @@ def test_run_route_happy_path(self, monkeypatch: pytest.MonkeyPatch) -> None: def test_run_route_failure_serializes_sentinel_with_http_200(self, monkeypatch: pytest.MonkeyPatch) -> None: # The sentinel body must survive FastAPI response-model serialization: a 500 here # would abort the entire collection run instead of routing to the failures sidecar. - client = self._client_and_mocks(monkeypatch, AsyncMock(side_effect=RuntimeError("remote exploded"))) + client = self._client(monkeypatch, AsyncMock(side_effect=RuntimeError("remote exploded"))) response = client.post("/run", json=make_row()) assert response.status_code == 200 body = response.json() @@ -522,16 +728,18 @@ def test_run_route_failure_serializes_sentinel_with_http_200(self, monkeypatch: assert body["reward"] == 0.0 assert body["response"]["output"][0]["type"] == "message" - async def test_responses_not_implemented(self) -> None: - agent = make_agent() - with pytest.raises(NotImplementedError): - await agent.responses(body={}) + def test_responses_route_is_live(self, monkeypatch: pytest.MonkeyPatch) -> None: + # /v1/responses is a real route now: create-params in, finished trajectory out. + client = self._client(monkeypatch) + response = client.post("/v1/responses", json={"input": [{"role": "user", "content": "hi"}]}) + assert response.status_code == 200 + assert response.json()["id"] == "traj_1" class TestStatefulToolsEndToEnd: - """The full session contract, in-process: RemoteAgent seeds the counter environment, - forwards the session to a fake remote service, the service calls the counter tools with - the forwarded cookie, and verify() scores the mutated session state.""" + """The full session contract, in-process: RemoteAgent seeds the real counter environment, + the service asks for tools via unpaired function_calls, GYM executes them against the + counter server on the seeded session, and verify() scores the mutated state.""" def _counter_client(self): from fastapi.testclient import TestClient @@ -547,10 +755,43 @@ def _counter_client(self): server = StatefulCounterResourcesServer(config=config, server_client=MagicMock(spec=ServerClient)) return TestClient(server.setup_webserver()) - async def test_counter_env_reward_through_forwarded_session(self, monkeypatch: pytest.MonkeyPatch) -> None: + async def test_counter_env_reward_through_gym_executed_tools(self, monkeypatch: pytest.MonkeyPatch) -> None: counter = self._counter_client() + # The service never sees the counter server: it only returns asks and reads outputs. + def turn2(received): + return traj([fn_call("c3", "get_counter_value", "{}")]) + + service = scripted_service( + traj( + [ + fn_call("c1", "increment_counter", '{"count": 1}'), + fn_call("c2", "increment_counter", '{"count": 2}'), + ] + ), + traj([fn_call("c3", "get_counter_value", "{}")]), + # Final turn: read the count Gym fed back and answer with it + traj([msg("final count is 6")]), + ) + + agent = make_agent() + async def gym_post(server_name, url_path, json=None, cookies=None, **kwargs): + if url_path.endswith("/v1/responses"): + wire = json.model_dump(exclude_unset=True) if isinstance(json, BaseModel) else json + params = NeMoGymResponseCreateParamsNonStreaming.model_validate(wire) + fastapi_response = Response() + try: + result = await agent.responses(make_request(dict(cookies or {})), fastapi_response, params) + except Exception as e: # noqa: BLE001 + return FakeServerClientResponse({"error": repr(e)}, status=500) + out_cookies = SimpleCookie() + for header_value in fastapi_response.headers.getlist("set-cookie"): + out_cookies.load(header_value) + return FakeServerClientResponse( + result.model_dump(mode="json"), cookies={k: m.value for k, m in out_cookies.items()} + ) + # Everything else — seed, tools, verify — hits the REAL counter server response = counter.post(url_path, json=json, cookies=dict(cookies or {})) return FakeServerClientResponse( response.json(), cookies=dict(response.cookies), status=response.status_code @@ -558,52 +799,13 @@ async def gym_post(server_name, url_path, json=None, cookies=None, **kwargs): server_client = MagicMock(spec=ServerClient) server_client.post = AsyncMock(side_effect=gym_post) + agent.server_client = server_client + mock_remote(monkeypatch, service) - async def remote_service(method, url, data=None, headers=None, **kwargs): - # The remote service reads the forwarded session and calls the counter tools - # with the cookie echoed on every call — the contract under test. - cookie_pair = headers[SESSION_COOKIE_HEADER] - cookie_name, cookie_value = cookie_pair.split("=", 1) - tool_cookies = {cookie_name: cookie_value} - assert headers[RESOURCES_URL_HEADER].startswith("http://") - - assert counter.post("/increment_counter", json={"count": 1}, cookies=tool_cookies).status_code == 200 - assert counter.post("/increment_counter", json={"count": 2}, cookies=tool_cookies).status_code == 200 - count = counter.post("/get_counter_value", json={}, cookies=tool_cookies).json()["count"] - - trajectory = dict(_MINIMAL_TRAJECTORY) - trajectory["output"] = [ - { - "type": "message", - "role": "assistant", - "status": "completed", - "id": "msg_1", - "content": [{"type": "output_text", "text": f"final count is {count}", "annotations": []}], - } - ] - return FakeRemoteResponse(200, orjson.dumps(trajectory)) - - mock_remote(monkeypatch, AsyncMock(side_effect=remote_service)) - monkeypatch.setattr(remote_agent_app, "get_server_url", lambda name: "http://counter-in-process") - - agent = make_agent(server_client=server_client, tools_mode="forward") row = { "responses_create_params": { "input": [{"role": "user", "content": "add 1 then add 2 then get the count"}], - "tools": [ - { - "type": "function", - "name": "increment_counter", - "parameters": { - "type": "object", - "properties": {"count": {"type": "integer", "description": ""}}, - "required": ["count"], - "additionalProperties": False, - }, - "strict": True, - "description": "", - } - ], + "tools": _COUNTER_TOOLS, }, "initial_count": 3, "expected_count": 6, @@ -613,9 +815,11 @@ async def remote_service(method, url, data=None, headers=None, **kwargs): dumped = result.model_dump() assert NG_FAILURE_CLASS_KEY not in dumped - # Reward 1.0 only if seed, both tool calls, and verify all shared ONE session + # Reward 1.0 only if seed, both Gym-executed tool calls, and verify shared ONE session assert dumped["reward"] == 1.0 - assert "final count is 6" in dumped["response"]["output"][0]["content"][0]["text"] + # The service really was fed the counter value Gym read back + third_input = service.received[2]["payload"]["input"] + assert any('"count":6' in i.get("output", "") for i in third_input if isinstance(i, dict)) class TestCollectorRoundTrip: @@ -633,14 +837,13 @@ async def test_success_and_failure_routing(self, monkeypatch: pytest.MonkeyPatch # Hydra CLI parse it would otherwise attempt under pytest (same as the core tests). monkeypatch.setattr(nemo_gym.rollout_collection, "get_global_config_dict", MagicMock(return_value={})) - async def remote_service(method, url, data=None, headers=None, **kwargs): + async def remote_service(method, url, data=None, headers=None, cookies=None, **kwargs): params = orjson.loads(data) if "fail" in params["input"][0]["content"]: return FakeRemoteResponse(500, b"remote exploded") return FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)) - mock_remote(monkeypatch, AsyncMock(side_effect=remote_service)) - agent = make_agent(server_client=seed_verify_server_client()) + agent, _, _ = make_wired_agent(monkeypatch, AsyncMock(side_effect=remote_service)) agent_http = TestClient(agent.setup_webserver(), raise_server_exceptions=False) class InProcessHelper(RolloutCollectionHelper): @@ -696,8 +899,7 @@ class TestReviewFindingPins: async def test_failure_on_reused_rollout_row_still_returns_sentinel(self, monkeypatch: pytest.MonkeyPatch) -> None: # A rollouts/failures JSONL re-fed as a dataset carries reward/response/error and stale # routing keys; the failure path must not TypeError on them (the never-raise contract). - mock_remote(monkeypatch, AsyncMock(side_effect=RuntimeError("remote exploded"))) - agent = make_agent(server_client=seed_verify_server_client()) + agent, _, _ = make_wired_agent(monkeypatch, AsyncMock(side_effect=RuntimeError("remote exploded"))) row = make_row(**self._REUSED_ROW_EXTRAS) result = (await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row))).model_dump() @@ -713,8 +915,7 @@ async def test_failure_on_reused_rollout_row_still_returns_sentinel(self, monkey def test_failure_on_reused_rollout_row_route_level_stays_200(self, monkeypatch: pytest.MonkeyPatch) -> None: from fastapi.testclient import TestClient - mock_remote(monkeypatch, AsyncMock(side_effect=RuntimeError("remote exploded"))) - agent = make_agent(server_client=seed_verify_server_client()) + agent, _, _ = make_wired_agent(monkeypatch, AsyncMock(side_effect=RuntimeError("remote exploded"))) client = TestClient(agent.setup_webserver(), raise_server_exceptions=False) response = client.post("/run", json=make_row(**self._REUSED_ROW_EXTRAS)) @@ -725,8 +926,7 @@ def test_failure_on_reused_rollout_row_route_level_stays_200(self, monkeypatch: async def test_happy_path_reused_row_leaks_no_stale_sentinels(self, monkeypatch: pytest.MonkeyPatch) -> None: # Stale routing keys on an input row must not echo through verify and misroute a # SUCCESS into the failures sidecar. - mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)))) - agent = make_agent(server_client=seed_verify_server_client()) + agent, _, _ = make_wired_agent(monkeypatch, scripted_service(_MINIMAL_TRAJECTORY)) row = make_row(**self._REUSED_ROW_EXTRAS) result = (await agent.run(make_request(), RemoteAgentRunRequest.model_validate(row))).model_dump() @@ -780,92 +980,9 @@ async def hang(*args, **kwargs): agent = make_agent(server_client=server_client) monkeypatch.setattr(remote_agent_app, "_AGGREGATE_PROXY_TIMEOUT_SECS", 0.05) - from nemo_gym.base_resources_server import AggregateMetricsRequest - with pytest.raises(asyncio.TimeoutError): - await agent.aggregate_metrics(AggregateMetricsRequest(verify_responses=[])) - - async def test_run_timeout_excludes_semaphore_queue_wait(self, monkeypatch: pytest.MonkeyPatch) -> None: - release_first = asyncio.Event() - - async def gated(*args, **kwargs): - await release_first.wait() - return FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)) - - mock_remote(monkeypatch, AsyncMock(side_effect=gated)) - agent = make_agent(server_client=seed_verify_server_client(), concurrency=1, run_timeout_secs=0.5) - - first = asyncio.create_task(agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row()))) - second = asyncio.create_task(agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row()))) - # Hold the only permit for most of the second task's would-be budget - await asyncio.sleep(0.4) - release_first.set() - results = [r.model_dump() for r in await asyncio.gather(first, second)] - - # If queue wait counted against run_timeout_secs, the second task would time out - assert all(NG_FAILURE_CLASS_KEY not in r for r in results) - - async def test_tools_mode_forward_loopback_warning_for_offhost_remote( - self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] - ) -> None: - mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)))) - monkeypatch.setattr(remote_agent_app, "get_server_url", lambda name: "http://127.0.0.1:15022") - agent = make_agent( - server_client=seed_verify_server_client(), - tools_mode="forward", - agent_base_url="http://gpu-node-7:9000", - ) - - await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) - - assert "advertised_resources_url" in capsys.readouterr().out - - async def test_tools_mode_forward_warns_on_unroutable_bind_address_too( - self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] - ) -> None: - # host: 0.0.0.0 makes get_server_url advertise 0.0.0.0 — from the remote machine that - # resolves to ITS OWN loopback, the same silent-zero failure as advertising 127.0.0.1. - mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY)))) - monkeypatch.setattr(remote_agent_app, "get_server_url", lambda name: "http://0.0.0.0:15022") - agent = make_agent( - server_client=seed_verify_server_client(), - tools_mode="forward", - agent_base_url="http://gpu-node-7:9000", - ) - - await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) - - assert "advertised_resources_url" in capsys.readouterr().out - - async def test_advertised_resources_url_overrides_header_and_silences_warning( - self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] - ) -> None: - request_mock = AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(_MINIMAL_TRAJECTORY))) - client = mock_remote(monkeypatch, request_mock) - monkeypatch.setattr(remote_agent_app, "get_server_url", lambda name: "http://127.0.0.1:15022") - agent = make_agent( - server_client=seed_verify_server_client(), - tools_mode="forward", - agent_base_url="http://gpu-node-7:9000", - advertised_resources_url="http://head-node.cluster:15022", - ) - - await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) - - headers = client.request.call_args.kwargs["headers"] - assert headers[RESOURCES_URL_HEADER] == "http://head-node.cluster:15022" - assert "advertised_resources_url" not in capsys.readouterr().out - - async def test_quality_warnings_are_throttled( - self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] - ) -> None: - trajectory = dict(_MINIMAL_TRAJECTORY) - trajectory.pop("usage") - mock_remote(monkeypatch, AsyncMock(return_value=FakeRemoteResponse(200, orjson.dumps(trajectory)))) - agent = make_agent(server_client=seed_verify_server_client()) - - for _ in range(10): - await agent.run(make_request(), RemoteAgentRunRequest.model_validate(make_row())) - - # Head of 5, then every 100th: 10 rollouts -> exactly 5 printed warnings - assert capsys.readouterr().out.count("no usage") == 5 + await agent.aggregate_metrics( + __import__( + "nemo_gym.base_resources_server", fromlist=["AggregateMetricsRequest"] + ).AggregateMetricsRequest(verify_responses=[]) + ) From 9a0bf2cda58fd05479e7e096874b92d8de6a82d6 Mon Sep 17 00:00:00 2001 From: adil-a Date: Wed, 29 Jul 2026 23:45:59 +0000 Subject: [PATCH 14/25] chore: align remote_agent.yaml with the loop architecture (drop tools_mode/advertised, add max_steps) Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- responses_api_agents/remote_agent/configs/remote_agent.yaml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/responses_api_agents/remote_agent/configs/remote_agent.yaml b/responses_api_agents/remote_agent/configs/remote_agent.yaml index 67a845d7c9..8ff0354256 100644 --- a/responses_api_agents/remote_agent/configs/remote_agent.yaml +++ b/responses_api_agents/remote_agent/configs/remote_agent.yaml @@ -9,5 +9,4 @@ remote_agent: concurrency: 32 remote_responses_timeout_secs: 1800.0 run_timeout_secs: 2100.0 - tools_mode: refuse - advertised_resources_url: null + max_steps: null From 4105a08f0f6a37063768dde87f1dbd8033d64ccd Mon Sep 17 00:00:00 2001 From: adil-a Date: Wed, 29 Jul 2026 23:52:56 +0000 Subject: [PATCH 15/25] docs: rewrite remote_agent docs for the Gym-driven loop; explicit user-vs-Gym responsibilities Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- .../pages/agent-server/remote-agent.mdx | 177 +++++++++--------- responses_api_agents/remote_agent/README.md | 82 ++++---- 2 files changed, 125 insertions(+), 134 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 883bd21770..8b249e642e 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -1,6 +1,6 @@ --- title: "Drive a Remote Agent" -description: "Evaluate an agent service you host yourself — one endpoint, verification stays in Gym" +description: "Evaluate an agent service you host yourself — Gym drives the loop, your service is called like a model" position: 3 --- @@ -8,37 +8,80 @@ position: 3 The `remote_agent` server lets an agent that runs as **its own HTTP service** — in your repo, on your infrastructure — be driven by standard rollout collection. Your service implements one -endpoint; Gym keeps the session, the verification, and the task answer keys on its side, and your -results land in the standard artifacts (`gym eval profile`, aggregation, and training pipelines -all work unchanged). +endpoint and is **called like a model**: it receives the conversation so far and returns what it +wants to do next. Gym runs the loop, executes environment tools, holds the session, and verifies. +Your results land in the standard artifacts (`gym eval profile`, aggregation, and training +pipelines all work unchanged). ``` -collector ──/run──▶ remote_agent (Gym) ──2. POST /v1/responses──▶ your service - │ 1. seed_session (your model, your tools, - │ 3. verify (same session) returns one trajectory) +collector ──/run──▶ remote_agent (Gym) ──POST /v1/responses──▶ your service + │ ▲ (returns tool asks + │ └── tool results appended, or a final answer) + │ loop repeats ▼ - resources server + resources server (seed / tools / verify — never exposed to your service) ``` -## The contract for your service +## Who does what -Implement `POST {agent_base_url}/v1/responses`: +**Gym is responsible for:** -| | | -|---|---| -| **Request body** | The task's `responses_create_params` — the input messages and any declared tool schemas. Never `verifier_metadata` (the answer key stays inside Gym). | -| **Request headers** (only with `tools_mode: forward`) | `X-NeMo-Gym-Resources-Server-Url` and `X-NeMo-Gym-Session-Cookie` — see [Gym-hosted tools](#gym-hosted-tools-tools_mode). | -| **Response** | One **finished** Responses API object: run your whole loop (any number of model turns and tool calls) and return the merged trajectory, ending with an assistant message. Populate the full `usage` object — `input_tokens`, `output_tokens`, `total_tokens`, `input_tokens_details: {cached_tokens}`, `output_tokens_details: {reasoning_tokens}` — or omit `usage` entirely (allowed; token metrics are then empty). A partial `usage` object fails validation. | +1. Seeding a fresh environment session per rollout and holding its cookies — session state + and the task's answer key (`verifier_metadata`) never reach your service. +2. Driving the loop: calling your service, executing the tool calls it asks for against the + environment, appending the results, and calling again. +3. All failure handling: bounded connect retries, per-call and whole-rollout timeouts, and + converting every failure into a reward-0 row in the failures sidecar (a bad rollout never + crashes the collection run). +4. Verifying the finished trajectory and reporting the reward. + +**Your service is responsible for:** + +1. Implementing `POST {agent_base_url}/v1/responses` and answering every call with a valid + Responses API object (the schema is validated strictly; an invalid object is a terminal, + non-retried failure). +2. Deciding what to do next on each call: ask for environment tools, run its own internal + tools, or finish with an assistant message. +3. Being callable N times per rollout. Each call carries the full conversation so far, so a + stateless service needs nothing extra; if you want per-rollout state, set a cookie — Gym + echoes your cookies back on every subsequent call of the same rollout. +4. Reporting `usage` (or omitting it entirely — allowed, but your token metrics are empty). + +## The contract, exactly - -The response is validated strictly against the Responses API schema. An invalid object is -recorded as a terminal failure (a schema bug will not fix itself on retry) and never reaches the -verifier. - +**Every request you receive** is the task's `responses_create_params` with the conversation +accumulated in `input`: + +1. `input` — the task messages, plus (from turn 2 onward) everything so far: your previous + output items and one `function_call_output` per tool call Gym executed for you. +2. `tools` — the tool schemas this environment serves, verbatim from the dataset row. These + are the only tools you may ask Gym to execute. +3. Your own cookies from earlier calls of this rollout, echoed back. + +**Every response you return** is one Responses API object whose `output` decides the next step: + +| You return | Gym does | +|---|---| +| One or more `function_call` items **without** a matching `function_call_output` (same `call_id`, same response) | Executes each against the resources server, appends each result as a `function_call_output`, and calls you again. | +| `function_call` + `function_call_output` **pairs** (same `call_id`) | Nothing — that's your own internal tool record; it passes into the trajectory untouched. | +| An assistant `message` and no unpaired calls | The rollout is done: Gym merges the full conversation into one trajectory and verifies it. | +| `incomplete_details` set | The loop stops and the trajectory so far is verified. | + +**Expectations and edge semantics:** + +1. Ask only for tools the request's `tools` declared. An unknown tool name is not an error — + Gym sends the resources server's error text (e.g. a 404) back to you as that call's + `function_call_output`, and the rollout continues. +2. Malformed `arguments` (not valid JSON) likewise come back to you as an error output, + never a crash. +3. Never return an unpaired `function_call` for a tool you already executed yourself — + unpaired means "Gym, run this." +4. One rollout = one conversation. Requests of the same rollout share your cookies; + different rollouts (including retries of the same task) start clean. ## Quickstart -A minimal service (FastAPI, ~20 lines): +A minimal service — ask for a tool once, then answer (FastAPI, ~25 lines): ```python from fastapi import FastAPI, Request @@ -48,13 +91,17 @@ app = FastAPI() @app.post("/v1/responses") async def responses(request: Request): params = await request.json() - answer = my_agent_loop(params["input"]) # your model, your tools, your turns + tool_results = [i for i in params["input"] if i.get("type") == "function_call_output"] + if not tool_results: # turn 1: ask Gym to run a tool + output = [{"type": "function_call", "id": "fc_1", "call_id": "c1", + "name": params["tools"][0]["name"], "arguments": "{}"}] + else: # turn 2: answer from the tool result + output = [{"type": "message", "role": "assistant", "status": "completed", "id": "msg_1", + "content": [{"type": "output_text", "text": tool_results[-1]["output"], + "annotations": []}]}] return { "id": "my-service", "created_at": 0.0, "model": "my-model", "object": "response", - "output": [{ - "type": "message", "role": "assistant", "status": "completed", "id": "msg_0", - "content": [{"type": "output_text", "text": answer, "annotations": []}], - }], + "output": output, "parallel_tool_calls": False, "tools": [], "tool_choice": "auto", "usage": { "input_tokens": 0, "output_tokens": 0, "total_tokens": 0, @@ -75,7 +122,6 @@ remote_agent: resources_server: type: resources_servers name: my_env_resources_server # the top-level key of that environment's config - tools_mode: refuse ``` ```bash @@ -88,75 +134,34 @@ gym eval run --no-serve +agent_name=remote_agent \ | Field | Default | What it does | |---|---|---| -| `agent_base_url` | required | Your service's base URL. Validated: `http(s)` only, no query string or fragment, no embedded credentials. | -| `resources_server` | required | The environment that seeds and verifies each rollout. Swap benchmarks by changing this ref — your service doesn't change. | -| `tools_mode` | `refuse` | Who serves the tools a dataset declares — see below. | -| `concurrency` | `32` | Maximum rollouts in flight against your service, enforced server-side. This is the *owner's* bound: it caps total pressure from every collection run calling into this agent process. (A second, independently started Gym stack has its own bound.) | -| `remote_responses_timeout_secs` | `1800` | Wallclock bound on one `/v1/responses` call. Raise it if your rollouts legitimately run longer than 30 minutes. | -| `run_timeout_secs` | `2100` | Bound on a whole `/run` (seed + your service + verify), started after the concurrency slot is acquired — queue wait doesn't count. | -| `advertised_resources_url` | unset | With `tools_mode: forward`: the resources-server URL told to your service. The default advertises the bind address (typically `127.0.0.1`), which another machine cannot reach — set this when your service runs off-host. It changes only the header string; making the address route to Gym is your infrastructure's job. | - -## Gym-hosted tools (`tools_mode`) - -- **`refuse`** (default) — tool-declaring tasks are rejected up front with a clear, terminal error. - Use this whenever your tasks don't declare Gym-hosted tools — text-only benchmarks, tasks whose - tools all live inside your own service — which is the common case. Tool-free rows run completely - normally under `refuse`; the guard only fires if a row *declares* tools. Keeping it on also acts - as a tripwire while you wire things up: if a tool-declaring dataset shows up before your service - echoes the session cookie, you get an immediate, named error instead of the silent failure mode — - without session access, tools that mutate per-session state never run, and every rollout scores 0 - with no error anywhere. -- **`forward`** — each request to your service carries the resources-server URL and the rollout's - session cookie as headers. Echo the cookie on every tool call you make against that URL and - stateful environments work end to end. The URL is re-sent per request because Gym assigns - servers random ports on every start; the cookie is minted per rollout. -- **`remote`** — your service implements the declared tools itself; nothing is forwarded. - - -`forward` covers environments whose tools are plain HTTP routes on the resources server (the -session cookie is the only credential involved). Environments that expose their tools **over MCP** -mint additional per-session token headers that are not forwarded in this version. - - -### Running your service off-host with `tools_mode: forward` - -The advertised URL must be reachable *from your service's machine*: - -1. Bind the resources server on all interfaces and pin its port (`host: 0.0.0.0`, `port: `). -2. Make the path route — internal DNS, firewall rule, SSH tunnel, or load balancer (your infra). -3. Verify once **from the remote machine**: `curl http://
:/`. -4. Set `advertised_resources_url: http://
:`. -5. Recommended: probe the advertised URL from your service on first request and fail loudly — - reachability is only testable from your side. - - -The easiest step to forget on an off-host deployment is step 4: without it, your service receives -an unroutable bind address (`127.0.0.1` or `0.0.0.0`), every tool call misses Gym, and rewards are -silently zero. The agent logs a warning when it detects this combination, but it cannot verify -reachability for you. - +| `agent_base_url` | required | Your service's base URL. Validated: `http(s)` only, no query string or fragment, no embedded credentials. This is the only network direction — Gym calls you; your service never needs to reach Gym. | +| `resources_server` | required | The environment that seeds, serves tools for, and verifies each rollout. Swap benchmarks by changing this ref — your service doesn't change. | +| `concurrency` | `32` | Maximum rollouts in flight against your service, enforced server-side. | +| `remote_responses_timeout_secs` | `1800` | Wallclock bound on ONE call to your service (a rollout makes one call per loop step). | +| `run_timeout_secs` | `2100` | Bound on a whole rollout (seed + every loop step + verify), started after the concurrency slot is acquired — queue wait doesn't count. | +| `max_steps` | unset | Maximum loop steps per rollout. Unset leaves `run_timeout_secs` as the only bound. | ## Failure handling and resume -Failures never crash a collection run. A down service (bounded connect retries), a timeout, a -malformed reply, or a verifier error becomes a reward-0 row with `_ng_failure_class: -"remote_agent_error"` in the failures sidecar (`_failures.jsonl`) — the main rollouts -file stays clean, and `+resume_from_cache=true` retries failed tasks up to the attempt cap -(`NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`, default 3; terminal failures are not retried). Error messages -name the failing URL or the timeout knob involved. +Failures never crash a collection run. A down service (3 connection attempts, then fail), a +timed-out call, a malformed reply, or a verifier error becomes a reward-0 row with +`_ng_failure_class: "remote_agent_error"` in the failures sidecar (`_failures.jsonl`) — +the main rollouts file stays clean, and `+resume_from_cache=true` retries failed tasks up to the +attempt cap (`NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`, default 3; terminal failures — an invalid response +shape — are not retried). Error messages name the failing URL or the timeout knob involved. ## Gotchas +- **Your service is called multiple times per rollout.** Design for it: everything you need is + in each request's `input`, or in your own cookies. - **Redirects are rejected, not followed.** Any `3xx` from your service fails the rollout with the `Location` shown — point `agent_base_url` at the final address. (Followed, a `301/302/303` would silently re-issue the POST as a body-less GET; a `307/308` would silently re-send the task to an address you never configured.) -- **Missing `usage` = empty token metrics.** The run succeeds and warns; your cost/token - accounting is empty. Report real token counts. -- **Timeouts don't retry inline** — a timed-out rollout goes to the sidecar and is retried on - resume, so one hung request doesn't burn triple wallclock. -- **Skills are not forwarded.** A `+skills` config stamps rows but cannot stage files into a - remote process; the agent warns and ignores it. +- **Partial `usage` fails validation.** Report the full object — `input_tokens`, + `output_tokens`, `total_tokens`, `input_tokens_details: {cached_tokens}`, + `output_tokens_details: {reasoning_tokens}` — or omit `usage` entirely (allowed; token + metrics are then empty, with a warning). - **A previous run's output can be reused as the input dataset** (for example, re-running the tasks in a failures sidecar). Output rows carry the old run's results (`reward`, `response`, `error`, internal `_ng_*` flags); those fields are stripped from each row on the way in, so a diff --git a/responses_api_agents/remote_agent/README.md b/responses_api_agents/remote_agent/README.md index f36bfca1e3..8d1de57946 100644 --- a/responses_api_agents/remote_agent/README.md +++ b/responses_api_agents/remote_agent/README.md @@ -1,53 +1,37 @@ # Remote Agent -A thin agent server that brokers rollouts to an agent service you host yourself — in your own -repo, on your own infrastructure. Your service implements one endpoint, `POST /v1/responses`: -it receives the task's `responses_create_params`, runs its own agent loop (its own model and -tools, however many turns it needs), and returns one finished Responses API trajectory. - -Gym keeps everything else on its side: this server seeds the session, holds the session -cookies, verifies the trajectory on the resources server (`verifier_metadata` never leaves -Gym), and reports the verify response — so your rollouts land in the standard artifacts and -work with `gym eval profile`, aggregation, and (when your service routes its model calls -through a Gym model server) token-id capture for training. - -## Contract for your service - -- `POST {agent_base_url}/v1/responses` with the row's `responses_create_params` as the JSON body. -- Return a single finished Responses API object: the last output item is an assistant message, - no dangling tool calls, `usage` populated with the full shape — `{input_tokens, output_tokens, - total_tokens, input_tokens_details: {cached_tokens}, output_tokens_details: {reasoning_tokens}}`. - (Omitting `usage` entirely is allowed and only warns; a partial `usage` object fails validation.) -- Failures on Gym's side never crash a collection run: they are recorded as reward-0 rows in - the failures sidecar and retried on resume up to the attempt cap - (`NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`, default 3). Failures marked terminal — an invalid response - shape, or the tools guard — are not retried. - -## Gym-hosted tools (optional) - -`tools_mode` decides who serves the tools a dataset declares: - -- `refuse` (default): tool-declaring tasks are rejected up front with a clear error instead of - silently scoring zero against untouched session state. -- `forward`: each request to your service carries two headers, `X-NeMo-Gym-Resources-Server-Url` - and `X-NeMo-Gym-Session-Cookie`. Echo the cookie on every tool call you make against that URL - and stateful environments work end to end. The URL is re-sent per request because Gym assigns - servers random ports on every start; the cookie is minted per rollout. -- `remote`: your service implements the declared tools itself; nothing is forwarded. - -### Running the service off-host (`tools_mode: forward`) - -The advertised URL must be reachable *from your service's machine* — Gym serves the tools and -tells your service where they are; making that address route to Gym is on you: - -1. Bind the resources server on all interfaces and pin its port (`host: 0.0.0.0`, `port: `). -2. Make the path route (internal DNS / firewall rule / SSH tunnel / load balancer — your infra). -3. Verify once from the remote machine: `curl http://
:/` should connect. -4. Set `advertised_resources_url: http://
:` on this agent. It changes only the - header string; the default advertises the bind address, which is typically a loopback address - other machines cannot reach (you'll see a warning for that combination). -5. Recommended: have your service probe the advertised URL on its first request and fail loudly — - reachability is only testable from your side. +An agent server that drives an agent service you host yourself — in your own repo, on your own +infrastructure. Your service implements one endpoint, `POST /v1/responses`, and is **called like +a model**: each call it receives the conversation so far and returns what it wants to do next. +Gym runs the loop. + +## Who does what + +Gym: seeds a fresh environment session per rollout and holds its cookies (session state and +`verifier_metadata` never reach your service), executes the tool calls your service asks for +against the resources server, appends the results and calls your service again, converts every +failure into a reward-0 sidecar row (never a crashed run), and verifies the finished trajectory. + +Your service: answers each call with a valid Responses API object; decides whether to ask for +environment tools, run its own internal tools, or finish; tolerates being called N times per +rollout (each request carries the full conversation; set a cookie if you want per-rollout state — +Gym echoes your cookies back within the rollout); reports full `usage` or omits it. + +## The response contract + +- `function_call` items **without** a matching `function_call_output` (same `call_id`, same + response) are asks: Gym executes them on the resources server and feeds the results back as + `function_call_output` items on the next call. +- `function_call` + `function_call_output` **pairs** are your own internal tool records; they + pass into the trajectory untouched. +- An assistant `message` with no unpaired calls finishes the rollout; Gym merges the whole + conversation into one trajectory and verifies it. +- Unknown tool names and malformed arguments are not crashes: the error text comes back to you + as that call's output and the rollout continues. An invalid Responses object is a terminal, + non-retried failure. + +The tool schemas your service may ask for arrive in every request's `tools` field, verbatim from +the dataset row. ## Run @@ -55,3 +39,5 @@ tells your service where they are; making that address route to Gym is on you: gym env start --resources-server # plus this agent's config gym eval run --no-serve +agent_name=remote_agent +input_jsonl_fpath=... +output_jsonl_fpath=... ``` + +Knobs and the full contract: see the "Drive a Remote Agent" docs page. From 8990b949918129c5fdc4fd66af9fc26d03b037e3 Mon Sep 17 00:00:00 2001 From: adil-a Date: Thu, 30 Jul 2026 00:22:42 +0000 Subject: [PATCH 16/25] =?UTF-8?q?docs:=20runnable=20quickstart=20=E2=80=94?= =?UTF-8?q?=20counter-env=20service,=20as-run=20config,=20expected=20outpu?= =?UTF-8?q?t?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- .../pages/agent-server/remote-agent.mdx | 62 +++++++++++++++---- 1 file changed, 50 insertions(+), 12 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 8b249e642e..83d4e9589d 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -81,9 +81,13 @@ accumulated in `input`: ## Quickstart -A minimal service — ask for a tool once, then answer (FastAPI, ~25 lines): +A complete, runnable example against the in-repo stateful counter environment +(`example_session_state_mgmt`; tasks read "add 1 then add 2 then get the count"). The service +asks Gym for every increment, then for the value, then answers with it — three loop turns: ```python +# service.py +import json, re from fastapi import FastAPI, Request app = FastAPI() @@ -91,14 +95,22 @@ app = FastAPI() @app.post("/v1/responses") async def responses(request: Request): params = await request.json() - tool_results = [i for i in params["input"] if i.get("type") == "function_call_output"] - if not tool_results: # turn 1: ask Gym to run a tool - output = [{"type": "function_call", "id": "fc_1", "call_id": "c1", - "name": params["tools"][0]["name"], "arguments": "{}"}] - else: # turn 2: answer from the tool result + task = next(i["content"] for i in params["input"] if i.get("role") == "user") + increments = [int(n) for n in re.findall(r"add (\d+)", task)] + results = [i for i in params["input"] if i.get("type") == "function_call_output"] + + if not results: # turn 1: ask Gym to run every increment + output = [{"type": "function_call", "id": f"fc_{k}", "call_id": f"c{k}", + "name": "increment_counter", "arguments": json.dumps({"count": n})} + for k, n in enumerate(increments)] + elif len(results) == len(increments): # turn 2: ask for the final value + output = [{"type": "function_call", "id": "fc_get", "call_id": "c_get", + "name": "get_counter_value", "arguments": "{}"}] + else: # turn 3: answer with it + count = json.loads(results[-1]["output"])["count"] output = [{"type": "message", "role": "assistant", "status": "completed", "id": "msg_1", - "content": [{"type": "output_text", "text": tool_results[-1]["output"], - "annotations": []}]}] + "content": [{"type": "output_text", "text": str(count), "annotations": []}]}] + return { "id": "my-service", "created_at": 0.0, "model": "my-model", "object": "response", "output": output, @@ -111,9 +123,18 @@ async def responses(request: Request): } ``` -Wire it into Gym: +One config file wires the environment and the agent together (this exact shape drove the live +end-to-end run — the only thing that changes for your own benchmark is the resources-server +block and the ref name): ```yaml +# counter_remote.yaml +example_session_state_mgmt_resources_server: + resources_servers: + example_session_state_mgmt: + entrypoint: app.py + domain: agent + remote_agent: responses_api_agents: remote_agent: @@ -121,15 +142,32 @@ remote_agent: agent_base_url: http://localhost:9000 # your service resources_server: type: resources_servers - name: my_env_resources_server # the top-level key of that environment's config + name: example_session_state_mgmt_resources_server # the top-level key above + concurrency: 4 + max_steps: 8 ``` +Run all three pieces: + ```bash -gym env start --resources-server my_env "+config_paths=[.../remote_agent.yaml]" ... +uvicorn service:app --port 9000 # your service +gym env start "+config_paths=[counter_remote.yaml]" # Gym: environment + remote_agent gym eval run --no-serve +agent_name=remote_agent \ - +input_jsonl_fpath=data/tasks.jsonl +output_jsonl_fpath=results/rollouts.jsonl + +input_jsonl_fpath=resources_servers/example_session_state_mgmt/data/example.jsonl \ + +output_jsonl_fpath=results/rollouts.jsonl ``` +Expected: 5 rollouts, `mean/reward: 1.0`, and each trajectory reads +`function_call → function_call_output → ... → message`. + + +A real service replaces the regex with an actual agent: render the incoming conversation into +your agent's context, let it decide, and translate its decision into unpaired `function_call` +items or a final message. The live end-to-end test of this page ran exactly that — a +containerized service using the Claude CLI as the brain — against the same counter environment, +scoring 5/5. + + ## Configuration knobs | Field | Default | What it does | From 4442f97485d93a6481ef3f7f8b50048898c98c23 Mon Sep 17 00:00:00 2001 From: adil-a Date: Thu, 30 Jul 2026 00:31:53 +0000 Subject: [PATCH 17/25] docs: external agent_base_url in quickstart; fix four audit imprecisions Quickstart config now shows an external host as the primary form (localhost demoted to a first-try footnote); validation wording no longer says strict (extras are tolerated); sidecar naming shown by example; error-message claim scoped to connection failures and timeouts; E2E tip phrased as a test of the contract. The quickstart was verified by execution: the doc's literal service.py driven by the real RemoteAgent against the real counter server scores 5/5 reward 1.0 on the example dataset. Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- .../latest/pages/agent-server/remote-agent.mdx | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 83d4e9589d..322180a851 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -38,8 +38,8 @@ collector ──/run──▶ remote_agent (Gym) ──POST /v1/responses── **Your service is responsible for:** 1. Implementing `POST {agent_base_url}/v1/responses` and answering every call with a valid - Responses API object (the schema is validated strictly; an invalid object is a terminal, - non-retried failure). + Responses API object (required fields and types are enforced, unknown extra fields are + tolerated; an invalid object is a terminal, non-retried failure). 2. Deciding what to do next on each call: ask for environment tools, run its own internal tools, or finish with an assistant message. 3. Being callable N times per rollout. Each call carries the full conversation so far, so a @@ -139,7 +139,9 @@ remote_agent: responses_api_agents: remote_agent: entrypoint: app.py - agent_base_url: http://localhost:9000 # your service + # Your service's address — typically another machine, a container, or a cloud box. + # (For a first try with everything on one machine, http://localhost:9000 works too.) + agent_base_url: http://your-agent-host:9000 resources_server: type: resources_servers name: example_session_state_mgmt_resources_server # the top-level key above @@ -150,7 +152,7 @@ remote_agent: Run all three pieces: ```bash -uvicorn service:app --port 9000 # your service +uvicorn service:app --host 0.0.0.0 --port 9000 # your service, on ITS machine gym env start "+config_paths=[counter_remote.yaml]" # Gym: environment + remote_agent gym eval run --no-serve +agent_name=remote_agent \ +input_jsonl_fpath=resources_servers/example_session_state_mgmt/data/example.jsonl \ @@ -163,7 +165,7 @@ Expected: 5 rollouts, `mean/reward: 1.0`, and each trajectory reads A real service replaces the regex with an actual agent: render the incoming conversation into your agent's context, let it decide, and translate its decision into unpaired `function_call` -items or a final message. The live end-to-end test of this page ran exactly that — a +items or a final message. A live end-to-end test of this contract ran exactly that — a containerized service using the Claude CLI as the brain — against the same counter environment, scoring 5/5. @@ -183,10 +185,12 @@ scoring 5/5. Failures never crash a collection run. A down service (3 connection attempts, then fail), a timed-out call, a malformed reply, or a verifier error becomes a reward-0 row with -`_ng_failure_class: "remote_agent_error"` in the failures sidecar (`_failures.jsonl`) — +`_ng_failure_class: "remote_agent_error"` in the failures sidecar (for +`results/rollouts.jsonl` the sidecar is `results/rollouts_failures.jsonl`) — the main rollouts file stays clean, and `+resume_from_cache=true` retries failed tasks up to the attempt cap (`NEMO_GYM_MAX_ROLLOUT_ATTEMPTS`, default 3; terminal failures — an invalid response -shape — are not retried). Error messages name the failing URL or the timeout knob involved. +shape — are not retried). Connection failures and timeouts name the failing URL or the timeout +knob involved. ## Gotchas From 2ddb865ef7af2cb21088d5c3e7e76f4545e81dc7 Mon Sep 17 00:00:00 2001 From: adil-a Date: Thu, 30 Jul 2026 00:37:34 +0000 Subject: [PATCH 18/25] docs: label the quickstart regex as the toy stand-in for the agent's model Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- fern/versions/latest/pages/agent-server/remote-agent.mdx | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 322180a851..2f5791d219 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -96,6 +96,9 @@ app = FastAPI() async def responses(request: Request): params = await request.json() task = next(i["content"] for i in params["input"] if i.get("role") == "user") + # The "brain" of this toy service: reading what the task wants out of the natural-language + # instruction. A real service does this with its model/agent (see the tip below); the tools + # field only carries the callable schemas, never the task-specific values. increments = [int(n) for n in re.findall(r"add (\d+)", task)] results = [i for i in params["input"] if i.get("type") == "function_call_output"] From 797ea4d023f8798aa690702ce8a22d96712381d7 Mon Sep 17 00:00:00 2001 From: adil-a Date: Thu, 30 Jul 2026 00:54:36 +0000 Subject: [PATCH 19/25] docs: Claude-CLI quickstart (self-contained, execution-verified); two-reasons-to-return contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The quickstart service now uses the Claude Code CLI as the agent brain — render conversation, let Claude decide, translate to Responses format (the CLI envelope note added where the translation happens). Verified by execution: the doc's literal block driven by the real RemoteAgent against the real counter server with real haiku scored 5/5 reward 1.0, including batched-vs-incremental ask patterns and fenced-JSON model output. The contract docs now state explicitly that the service runs its own tools freely and returns only to ask for a Gym tool or to finish. Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- .../pages/agent-server/remote-agent.mdx | 124 ++++++++++++------ responses_api_agents/remote_agent/README.md | 10 +- 2 files changed, 92 insertions(+), 42 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 2f5791d219..66f29f5fe6 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -40,8 +40,10 @@ collector ──/run──▶ remote_agent (Gym) ──POST /v1/responses── 1. Implementing `POST {agent_base_url}/v1/responses` and answering every call with a valid Responses API object (required fields and types are enforced, unknown extra fields are tolerated; an invalid object is a terminal, non-retried failure). -2. Deciding what to do next on each call: ask for environment tools, run its own internal - tools, or finish with an assistant message. +2. Deciding what to do next on each call. Within a call your service can do **anything** — + run its own model for any number of turns, execute its own tools, spawn sub-agents. There + are exactly two reasons to return: you need a **Gym-hosted tool** executed (return the ask), + or the rollout is **finished** (return the answer). 3. Being callable N times per rollout. Each call carries the full conversation so far, so a stateless service needs nothing extra; if you want per-rollout state, set a cookie — Gym echoes your cookies back on every subsequent call of the same rollout. @@ -58,7 +60,11 @@ accumulated in `input`: are the only tools you may ask Gym to execute. 3. Your own cookies from earlier calls of this rollout, echoed back. -**Every response you return** is one Responses API object whose `output` decides the next step: +**Every response you return** is one Responses API object whose `output` decides the next step. +Returning does not mean your agent is done thinking — it means one of two things: "I need a Gym +tool" or "I'm finished." Everything your agent did internally since the last call (its own model +turns, its own tool executions) either stays private or rides along as paired call+output +records: | You return | Gym does | |---|---| @@ -81,45 +87,86 @@ accumulated in `input`: ## Quickstart -A complete, runnable example against the in-repo stateful counter environment -(`example_session_state_mgmt`; tasks read "add 1 then add 2 then get the count"). The service -asks Gym for every increment, then for the value, then answers with it — three loop turns: +A complete, realistic service against the in-repo stateful counter environment +(`example_session_state_mgmt`; tasks read "add 1 then add 2 then get the count"). The **agent +brain is the Claude Code CLI**: each call, the service renders the conversation into a prompt, +lets Claude decide, and translates the decision into tool asks or a final answer. Requirements +on the service's machine: `claude` on PATH and authenticated (a logged-in CLI or +`ANTHROPIC_API_KEY` in the environment); pick the model with `CLAUDE_MODEL` (default `haiku`). ```python -# service.py -import json, re -from fastapi import FastAPI, Request +# service.py — the Claude Code CLI as the agent's brain +import json, os, subprocess +from fastapi import FastAPI app = FastAPI() +MODEL = os.environ.get("CLAUDE_MODEL", "haiku") -@app.post("/v1/responses") -async def responses(request: Request): - params = await request.json() - task = next(i["content"] for i in params["input"] if i.get("role") == "user") - # The "brain" of this toy service: reading what the task wants out of the natural-language - # instruction. A real service does this with its model/agent (see the tip below); the tools - # field only carries the callable schemas, never the task-specific values. - increments = [int(n) for n in re.findall(r"add (\d+)", task)] - results = [i for i in params["input"] if i.get("type") == "function_call_output"] - - if not results: # turn 1: ask Gym to run every increment - output = [{"type": "function_call", "id": f"fc_{k}", "call_id": f"c{k}", - "name": "increment_counter", "arguments": json.dumps({"count": n})} - for k, n in enumerate(increments)] - elif len(results) == len(increments): # turn 2: ask for the final value - output = [{"type": "function_call", "id": "fc_get", "call_id": "c_get", - "name": "get_counter_value", "arguments": "{}"}] - else: # turn 3: answer with it - count = json.loads(results[-1]["output"])["count"] - output = [{"type": "message", "role": "assistant", "status": "completed", "id": "msg_1", - "content": [{"type": "output_text", "text": str(count), "annotations": []}]}] +PROMPT = """You are an agent solving a task by calling tools. You do not execute tools \ +yourself; you ask for them and the results come back in the conversation. + +Task conversation so far: +{conversation} + +Tools you may ask for (JSON schemas): +{tools} + +Reply with ONLY one JSON object, no prose, no code fences: +- to call tools: {{"tool_calls": [{{"name": "", "arguments": {{...}}}}]}} +- to finish: {{"final": ""}} +Finish once the conversation contains enough tool results to answer; the final answer must be \ +exactly what the task asks for. +""" + +def render(items: list) -> str: + lines = [] + for item in items: + kind = item.get("type", "message") + if kind == "message": + content = item.get("content") + if isinstance(content, list): + content = " ".join(c.get("text", "") for c in content) + lines.append(f"[{item.get('role', 'user')}] {content}") + elif kind == "function_call": + lines.append(f"[you asked for] {item['name']}({item.get('arguments')})") + elif kind == "function_call_output": + lines.append(f"[tool result] {item.get('output')}") + return "\n".join(lines) + + +@app.post("/v1/responses") +def responses(params: dict): # sync handler: FastAPI runs it in a threadpool, so + # concurrent rollouts don't block each other on the subprocess below. + prompt = PROMPT.format(conversation=render(params["input"]), tools=json.dumps(params.get("tools", []))) + proc = subprocess.run( + ["claude", "-p", prompt, "--output-format", "json", "--model", MODEL], + capture_output=True, timeout=300, check=True, + ) + # The CLI's print mode returns its own JSON envelope (NOT the Anthropic Messages format): + # the final text sits in "result". The Responses object Gym expects is built by hand below — + # translating whatever your brain speaks into Responses format is the service's job. + payload = json.loads(proc.stdout) + decision = json.loads(payload["result"].strip().strip("`").removeprefix("json").strip()) + turn = sum(1 for i in params["input"] if i.get("type") == "function_call_output") + + if "tool_calls" in decision: # unpaired asks: Gym executes these and calls us again + output = [{"type": "function_call", "id": f"fc_{turn}_{k}", "call_id": f"c_{turn}_{k}", + "name": call["name"], "arguments": json.dumps(call.get("arguments", {}))} + for k, call in enumerate(decision["tool_calls"])] + else: # a final assistant message ends the rollout + output = [{"type": "message", "role": "assistant", "status": "completed", "id": f"m_{turn}", + "content": [{"type": "output_text", "text": str(decision["final"]), "annotations": []}]}] + + usage = payload.get("usage") or {} return { - "id": "my-service", "created_at": 0.0, "model": "my-model", "object": "response", + "id": "claude-cli-service", "created_at": 0.0, "model": f"claude-{MODEL}", "object": "response", "output": output, "parallel_tool_calls": False, "tools": [], "tool_choice": "auto", "usage": { - "input_tokens": 0, "output_tokens": 0, "total_tokens": 0, + "input_tokens": usage.get("input_tokens", 0), + "output_tokens": usage.get("output_tokens", 0), + "total_tokens": usage.get("input_tokens", 0) + usage.get("output_tokens", 0), "input_tokens_details": {"cached_tokens": 0}, "output_tokens_details": {"reasoning_tokens": 0}, }, @@ -163,14 +210,15 @@ gym eval run --no-serve +agent_name=remote_agent \ ``` Expected: 5 rollouts, `mean/reward: 1.0`, and each trajectory reads -`function_call → function_call_output → ... → message`. +`function_call → function_call_output → ... → message`. A live end-to-end run of exactly this +setup — the service in a container on its own network, Claude deciding every call — scored 5/5. -A real service replaces the regex with an actual agent: render the incoming conversation into -your agent's context, let it decide, and translate its decision into unpaired `function_call` -items or a final message. A live end-to-end test of this contract ran exactly that — a -containerized service using the Claude CLI as the brain — against the same counter environment, -scoring 5/5. +Nothing here is specific to FastAPI or the Claude CLI: any HTTP stack works (the live run used +stdlib `http.server`), and any brain works — render the conversation into your agent's context, +let it decide, translate the decision into unpaired `function_call` items or a final message. +Errors on your side are safe: a 5xx or timeout becomes a reward-0 row in Gym's failures sidecar, +never a crashed run. ## Configuration knobs diff --git a/responses_api_agents/remote_agent/README.md b/responses_api_agents/remote_agent/README.md index 8d1de57946..78166d6619 100644 --- a/responses_api_agents/remote_agent/README.md +++ b/responses_api_agents/remote_agent/README.md @@ -12,10 +12,12 @@ Gym: seeds a fresh environment session per rollout and holds its cookies (sessio against the resources server, appends the results and calls your service again, converts every failure into a reward-0 sidecar row (never a crashed run), and verifies the finished trajectory. -Your service: answers each call with a valid Responses API object; decides whether to ask for -environment tools, run its own internal tools, or finish; tolerates being called N times per -rollout (each request carries the full conversation; set a cookie if you want per-rollout state — -Gym echoes your cookies back within the rollout); reports full `usage` or omits it. +Your service: answers each call with a valid Responses API object. Within a call it can do +anything — its own model turns, its own tools, sub-agents; there are exactly two reasons to +return: it needs a Gym-hosted tool executed, or the rollout is finished. It tolerates being +called N times per rollout (each request carries the full conversation; set a cookie if you want +per-rollout state — Gym echoes your cookies back within the rollout) and reports full `usage` or +omits it. ## The response contract From 275ea0bd91b8b576ca1c0022e5a2be535cad94fa Mon Sep 17 00:00:00 2001 From: adil-a Date: Thu, 30 Jul 2026 01:06:50 +0000 Subject: [PATCH 20/25] docs: note claude -p tool defaults and internal multi-step in the quickstart Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- fern/versions/latest/pages/agent-server/remote-agent.mdx | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 66f29f5fe6..9786120695 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -139,6 +139,11 @@ def render(items: list) -> str: def responses(params: dict): # sync handler: FastAPI runs it in a threadpool, so # concurrent rollouts don't block each other on the subprocess below. prompt = PROMPT.format(conversation=render(params["input"]), tools=json.dumps(params.get("tools", []))) + # `claude -p` runs Claude's full agentic loop in one invocation: read-only file tools + # (Read/Glob/Grep) work by default; permission-gated tools (WebSearch, Bash, ...) are + # auto-denied headlessly unless pre-approved, e.g. --allowedTools "WebSearch,Bash". + # Either way Claude uses its OWN tools multi-step inside this single call — returning + # to Gym is only for Gym-hosted tools or the final answer. proc = subprocess.run( ["claude", "-p", prompt, "--output-format", "json", "--model", MODEL], capture_output=True, timeout=300, check=True, From 5c9d4464292fbe55c02c9b11be4133766b5606a0 Mon Sep 17 00:00:00 2001 From: adil-a Date: Thu, 30 Jul 2026 01:10:29 +0000 Subject: [PATCH 21/25] docs: split quickstart run commands into per-terminal blocks Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- .../latest/pages/agent-server/remote-agent.mdx | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 9786120695..9da2ce305d 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -204,11 +204,21 @@ remote_agent: max_steps: 8 ``` -Run all three pieces: +Run the three pieces in **separate terminals** — the first two are long-running servers, the +third is the driver: ```bash -uvicorn service:app --host 0.0.0.0 --port 9000 # your service, on ITS machine -gym env start "+config_paths=[counter_remote.yaml]" # Gym: environment + remote_agent +# Terminal 1 — on your service's machine: +uvicorn service:app --host 0.0.0.0 --port 9000 +``` + +```bash +# Terminal 2 — on the Gym machine: environment + remote_agent (leave running): +gym env start "+config_paths=[counter_remote.yaml]" +``` + +```bash +# Terminal 3 — on the Gym machine, once the servers report ready: gym eval run --no-serve +agent_name=remote_agent \ +input_jsonl_fpath=resources_servers/example_session_state_mgmt/data/example.jsonl \ +output_jsonl_fpath=results/rollouts.jsonl From 97280de264380d8c932051c9173b148f0252c606 Mon Sep 17 00:00:00 2001 From: adil-a Date: Thu, 30 Jul 2026 01:13:31 +0000 Subject: [PATCH 22/25] style: strip provenance and narration comments from remote_agent app.py Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- responses_api_agents/remote_agent/app.py | 45 ++++++++++-------------- 1 file changed, 18 insertions(+), 27 deletions(-) diff --git a/responses_api_agents/remote_agent/app.py b/responses_api_agents/remote_agent/app.py index 28432cca3a..27a4d33087 100644 --- a/responses_api_agents/remote_agent/app.py +++ b/responses_api_agents/remote_agent/app.py @@ -21,8 +21,8 @@ environment, it returns a ``function_call`` item WITHOUT a matching ``function_call_output``; tool calls it already answered itself (its own internal tools) ride along as paired call+output items and are passed through untouched. Gym runs the -loop — copied from simple_agent — executing unpaired calls against the resources server -and re-posting until the service returns a final assistant message. +loop: it executes unpaired calls against the resources server and re-posts until the +service returns a final assistant message. The resources server is never exposed to the service: tool execution, session cookies, and ``verifier_metadata`` all stay inside Gym. @@ -130,8 +130,7 @@ class RemoteAgentConfig(BaseResponsesAPIAgentConfig): # the semaphore is acquired so queue wait does not count against it. The collector's # named-agent hop carries no timeout of its own; this is the only whole-rollout bound. run_timeout_secs: float = 2100.0 - # Maximum loop steps (remote calls) per rollout; None leaves run_timeout_secs as the - # only bound, matching simple_agent's default. + # Maximum loop steps (remote calls) per rollout; None leaves run_timeout_secs as the only bound. max_steps: Optional[int] = None @field_validator("agent_base_url") @@ -164,9 +163,6 @@ async def responses( response: Response, body: NeMoGymResponseCreateParamsNonStreaming = Body(), ) -> NeMoGymResponse: - # simple_agent's loop with the model hop swapped for the remote service. The service - # is called like a model: conversation in, Responses object out. Divergences from - # simple_agent are marked; everything else is kept verbatim. body = body.model_copy(deep=True) if isinstance(body.input, str): @@ -176,13 +172,12 @@ async def responses( usage = None step = 0 agent_server_cookies = None # the service's own cookies, round-tripped so it can keep per-rollout state - resources_server_cookies = request.cookies # update the cookies on every resources server response + resources_server_cookies = request.cookies while True: step += 1 new_body = body.model_copy(update={"input": body.input + new_outputs}) - # Divergence: hardened POST to the external service instead of the model server. agent_response, agent_server_cookies = await self._post_agent_responses(new_body, agent_server_cookies) output = agent_response.output @@ -204,9 +199,9 @@ async def responses( if agent_response.incomplete_details: break - # Divergence: execute only UNPAIRED calls. A call the service already answered - # itself (matching function_call_output in the same response) is its own internal - # tool record — it passes through into the trajectory untouched. + # Execute only unpaired calls: a call the service already answered itself (matching + # function_call_output in the same response) is its own internal-tool record and + # passes through into the trajectory untouched. answered_call_ids = {o.call_id for o in output if o.type == "function_call_output"} all_fn_calls: List[NeMoGymResponseFunctionToolCall] = [ o for o in output if o.type == "function_call" and o.call_id not in answered_call_ids @@ -221,15 +216,12 @@ async def responses( try: parsed_arguments = json.loads(output_function_call.arguments) except (json.JSONDecodeError, TypeError) as e: - # The service produced malformed tool-call arguments. Surface the - # error back as a tool response so the rollout can continue - # (or terminate with a low reward) instead of crashing the - # whole batch on json.loads. + # Malformed arguments go back to the service as a tool error output + # instead of crashing the rollout; repr(e) keeps the exception type + # even when str(e) is empty. tool_response = NeMoGymFunctionCallOutput( type="function_call_output", call_id=output_function_call.call_id, - # Use repr(e) so the exception type name is always - # included even when str(e) would be empty. output=json.dumps({"error": f"Invalid tool call arguments: {e!r}"}), ) new_outputs.append(tool_response) @@ -241,7 +233,8 @@ async def responses( json=parsed_arguments, cookies=resources_server_cookies, ) - # We don't raise for status here since it's a valid return for the API to error e.g. if the service asks for an unknown tool or passes an invalid call. + # No raise_for_status: a tool error (unknown tool, invalid call) is a valid + # result the service should see and react to. resources_server_cookies = api_response.cookies tool_response = NeMoGymFunctionCallOutput( @@ -251,12 +244,11 @@ async def responses( ) new_outputs.append(tool_response) - # Check if max steps is not None and if we have exhausted it. if self.config.max_steps and step >= self.config.max_steps: break - # Propagate any extra cookies necessary for downstream verification. The service's - # own cookies are its private session and deliberately stay out of the Gym side. + # Resources-server cookies propagate for downstream verification; the service's own + # cookies are its private session and deliberately stay out. for k, v in resources_server_cookies.items(): response.set_cookie(k, v) @@ -314,8 +306,8 @@ async def _post_agent_responses( f"Is your service running at {self.config.agent_base_url}?" ) - # client.request() returns once headers arrive; the body read can still raise - # (mid-body disconnect, deadline) and must honor the same never-raise contract. + # client.request() returns once headers arrive; the body read can still fail + # (mid-body disconnect, deadline). try: content = await response.read() except Exception as e: @@ -391,7 +383,7 @@ async def _run_once( "remote service; the skills config is ignored.", ) - # 1. Seed the session; the cookies key all per-session state on the resources server. + # Seed the session; the cookies key all per-session state on the resources server. cookies = request.cookies try: seed_response = await self.server_client.post( @@ -407,7 +399,6 @@ async def _run_once( record, f"/seed_session on the resources server failed: {type(e).__name__}: {e}" ) - # 2. Self-post to our own /v1/responses, which drives the agent/tool loop. try: loop_response = await self.server_client.post( server_name=self.config.name, @@ -429,7 +420,7 @@ async def _run_once( self._warn_on_response_quality(response_json) - # 3. Verify on the SAME session; the verify response (reward included) is /run's result. + # Verify on the SAME session; the verify response (reward included) is /run's result. try: verify_response = await self.server_client.post( server_name=self.config.resources_server.name, From f66c1dbab324cddcf5c707b28706d42c1714021f Mon Sep 17 00:00:00 2001 From: adil-a Date: Thu, 30 Jul 2026 01:13:55 +0000 Subject: [PATCH 23/25] docs: show the service's own model and tools in the flow diagram Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- .../latest/pages/agent-server/remote-agent.mdx | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 9da2ce305d..26dbd07f73 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -14,12 +14,12 @@ Your results land in the standard artifacts (`gym eval profile`, aggregation, an pipelines all work unchanged). ``` -collector ──/run──▶ remote_agent (Gym) ──POST /v1/responses──▶ your service - │ ▲ (returns tool asks - │ └── tool results appended, or a final answer) - │ loop repeats - ▼ - resources server (seed / tools / verify — never exposed to your service) +collector ──/run──▶ remote_agent (Gym) ──POST /v1/responses──▶ your service ⟲ your model, + │ ▲ your own tools + │ └── Gym-tool results appended, (any number of internal steps, + │ loop repeats then: Gym-tool asks + ▼ or a final answer) + resources server (seed / Gym tools / verify — never exposed to your service) ``` ## Who does what From c0162c1c0b2db2b8c99e1c3913d15ab7348e07d9 Mon Sep 17 00:00:00 2001 From: adil-a Date: Thu, 30 Jul 2026 02:00:05 +0000 Subject: [PATCH 24/25] docs: name the service contract as OpenAI /v1/responses-compliant in the intro Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- fern/versions/latest/pages/agent-server/remote-agent.mdx | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index 26dbd07f73..bb4745d555 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -7,9 +7,10 @@ position: 3 # Drive a Remote Agent The `remote_agent` server lets an agent that runs as **its own HTTP service** — in your repo, on -your infrastructure — be driven by standard rollout collection. Your service implements one -endpoint and is **called like a model**: it receives the conversation so far and returns what it -wants to do next. Gym runs the loop, executes environment tools, holds the session, and verifies. +your infrastructure — be driven by standard rollout collection. Your service implements an +endpoint **compliant with the OpenAI `/v1/responses` contract**, called like a model: it receives +the conversation so far and returns what it wants to do next. Gym runs the loop, executes +environment tools, holds the session, and verifies. Your results land in the standard artifacts (`gym eval profile`, aggregation, and training pipelines all work unchanged). From 150228168b9199617da39793c149a52032f31020 Mon Sep 17 00:00:00 2001 From: adil-a Date: Thu, 30 Jul 2026 15:44:13 +0000 Subject: [PATCH 25/25] docs: reframe as a composition of Responses-compliant agents per review Co-Authored-By: Claude Fable 5 Signed-off-by: adil-a --- fern/versions/latest/pages/agent-server/remote-agent.mdx | 9 +++++---- responses_api_agents/remote_agent/README.md | 7 ++++--- responses_api_agents/remote_agent/app.py | 5 +++-- 3 files changed, 12 insertions(+), 9 deletions(-) diff --git a/fern/versions/latest/pages/agent-server/remote-agent.mdx b/fern/versions/latest/pages/agent-server/remote-agent.mdx index bb4745d555..71ae4bb803 100644 --- a/fern/versions/latest/pages/agent-server/remote-agent.mdx +++ b/fern/versions/latest/pages/agent-server/remote-agent.mdx @@ -1,6 +1,6 @@ --- title: "Drive a Remote Agent" -description: "Evaluate an agent service you host yourself — Gym drives the loop, your service is called like a model" +description: "Evaluate an agent service you host yourself — a composition of OpenAI Responses-compliant agents, with Gym driving the loop" position: 3 --- @@ -8,9 +8,10 @@ position: 3 The `remote_agent` server lets an agent that runs as **its own HTTP service** — in your repo, on your infrastructure — be driven by standard rollout collection. Your service implements an -endpoint **compliant with the OpenAI `/v1/responses` contract**, called like a model: it receives -the conversation so far and returns what it wants to do next. Gym runs the loop, executes -environment tools, holds the session, and verifies. +endpoint **compliant with the OpenAI `/v1/responses` contract**, and the two servers compose as +Responses-speaking agents: each call your service receives the conversation so far and returns +what it wants to do next. Gym runs the loop, executes environment tools, holds the session, and +verifies. Your results land in the standard artifacts (`gym eval profile`, aggregation, and training pipelines all work unchanged). diff --git a/responses_api_agents/remote_agent/README.md b/responses_api_agents/remote_agent/README.md index 78166d6619..a9cd4bd111 100644 --- a/responses_api_agents/remote_agent/README.md +++ b/responses_api_agents/remote_agent/README.md @@ -1,9 +1,10 @@ # Remote Agent An agent server that drives an agent service you host yourself — in your own repo, on your own -infrastructure. Your service implements one endpoint, `POST /v1/responses`, and is **called like -a model**: each call it receives the conversation so far and returns what it wants to do next. -Gym runs the loop. +infrastructure. Your service implements one endpoint **compliant with the OpenAI +`/v1/responses` contract**, and the two servers compose as Responses-speaking agents: each call +your service receives the conversation so far and returns what it wants to do next. Gym runs +the loop. ## Who does what diff --git a/responses_api_agents/remote_agent/app.py b/responses_api_agents/remote_agent/app.py index 27a4d33087..1e61d09ff2 100644 --- a/responses_api_agents/remote_agent/app.py +++ b/responses_api_agents/remote_agent/app.py @@ -14,8 +14,9 @@ # limitations under the License. """Agent server that drives a user-hosted remote agent service through Gym's tool loop. -The remote service implements ONE endpoint, ``POST {agent_base_url}/v1/responses``, and is -called like a model: each call it receives the conversation so far (the row's +The remote service implements ONE endpoint, ``POST {agent_base_url}/v1/responses``, composing +with this server as OpenAI Responses-compliant agents: each call it receives the conversation +so far (the row's ``responses_create_params`` with the accumulated output and tool results appended to ``input``) and returns a Responses API object. To have Gym execute a tool from the environment, it returns a ``function_call`` item WITHOUT a matching