diff --git a/fern/versions/latest/pages/environment-tutorials/index.mdx b/fern/versions/latest/pages/environment-tutorials/index.mdx index 7353916db1..058fec5830 100644 --- a/fern/versions/latest/pages/environment-tutorials/index.mdx +++ b/fern/versions/latest/pages/environment-tutorials/index.mdx @@ -30,6 +30,12 @@ Start with the single-step tutorial, then progress through increasingly complex + +Compare the native `seed_session()` and `verify()` interface with the Gymnasium `reset()` and `step()` interface. + +start here + + Build a complete environment from scratch: scaffolding, task data, tools, verification, testing, and rollout collection. diff --git a/fern/versions/latest/pages/environment-tutorials/integrate-external-environments.mdx b/fern/versions/latest/pages/environment-tutorials/integrate-external-environments.mdx index 1a8a15a43e..67298c1f4e 100644 --- a/fern/versions/latest/pages/environment-tutorials/integrate-external-environments.mdx +++ b/fern/versions/latest/pages/environment-tutorials/integrate-external-environments.mdx @@ -1,7 +1,7 @@ --- title: "Integrate external libraries" description: "" -position: 7 +position: 8 --- Fundamentally, a training environment in NeMo Gym is some Python logic that performs a sequence or graph of model calls and tool calls. For native NeMo Gym environments, all of the model and tool calls are present within an agent server like [`simple_agent`](https://github.com/NVIDIA-NeMo/Gym/blob/main/responses_api_agents/simple_agent/app.py). For environments that we consider "external", the orchestration of model and tool calls is offloaded to a third-party library rather than implemented within NeMo Gym itself. diff --git a/fern/versions/latest/pages/environment-tutorials/mcp-resources-server.mdx b/fern/versions/latest/pages/environment-tutorials/mcp-resources-server.mdx index 0fbd1938aa..0f75eb0ddf 100644 --- a/fern/versions/latest/pages/environment-tutorials/mcp-resources-server.mdx +++ b/fern/versions/latest/pages/environment-tutorials/mcp-resources-server.mdx @@ -1,7 +1,7 @@ --- title: "MCP Resources Server" description: "Serve a Resources Server's tools over the Model Context Protocol (MCP) with one config flag, and verify their use — with a runnable Claude Code example" -position: 5 +position: 6 --- import { NavButton } from "../../../../components/NavButton"; diff --git a/fern/versions/latest/pages/environment-tutorials/multi-step-environment.mdx b/fern/versions/latest/pages/environment-tutorials/multi-step-environment.mdx index 8026cdfb58..0465940d8a 100644 --- a/fern/versions/latest/pages/environment-tutorials/multi-step-environment.mdx +++ b/fern/versions/latest/pages/environment-tutorials/multi-step-environment.mdx @@ -1,7 +1,7 @@ --- title: "Multi-Step Environment" description: "" -position: 3 +position: 4 --- import { NavButton } from "../../../../components/NavButton"; diff --git a/fern/versions/latest/pages/environment-tutorials/real-world-environment/index.mdx b/fern/versions/latest/pages/environment-tutorials/real-world-environment/index.mdx index aada5ed279..52ae301f88 100644 --- a/fern/versions/latest/pages/environment-tutorials/real-world-environment/index.mdx +++ b/fern/versions/latest/pages/environment-tutorials/real-world-environment/index.mdx @@ -1,7 +1,7 @@ --- title: "Real-World Environment" description: "" -position: 6 +position: 7 --- import { NavButton } from "../../../../../components/NavButton"; diff --git a/fern/versions/latest/pages/environment-tutorials/resources-server-apis.mdx b/fern/versions/latest/pages/environment-tutorials/resources-server-apis.mdx new file mode 100644 index 0000000000..2a6b67a519 --- /dev/null +++ b/fern/versions/latest/pages/environment-tutorials/resources-server-apis.mdx @@ -0,0 +1,452 @@ +--- +title: "Resources Server APIs" +description: "Understand the supported Resources Server protocol APIs and choose the appropriate one to build your Environment" +position: 2 +--- + +# Resources Server APIs + +A Resources Server implements the Environment interface related to: its tools, state, and reward logic. + + +**Agent Server context** + +The Agent is the behavior that decides what to do next, typically with different components like a Harness and a policy-model. +The Agent Server is the HTTP transport for two internal methods: + +- `run()` is exposed as `/run` and implements part of the episode orchestration associated with the Resources Server seed session, delegation of the Agent loop to responses() and call to the verifier. +- `responses()` is exposed as `/v1/responses` and implements Agent behavior. Depending on the Agent Server, this can be a complete agent harness or one policy-model call. + +In `simple_agent`, `run()` delegates the Agent loop to `responses()` before requesting final verification. In `gymnasium_agent`, `run()` owns the model-and-environment loop and does not invoke `responses()` to the tool calling. + + +## Resources Server Protocol Landscape + +NeMo Gym ships two framework-standard base protocols for coordinating an episode between the Agent and the Environment. Agent behavior and rollout orchestration are implemented by `responses()` and `run()` behind the Agent Server's HTTP endpoints; the Environment is implemented by the Resources Server. + +Each protocol has a matching built-in Agent Server: `simple_agent` for the `/seed_session` and `/verify` interface, and `gymnasium_agent` for the `/reset` and `/step` interface. + +| Base protocol | Resources Server API | Matching built-in Agent Server | Reward contract | +|---|---|---|---| +| Seed/verify | `/seed_session` and `/verify` | `simple_agent` | `/verify` scores the completed trajectory once. | +| Gymnasium | `/reset` and `/step` | `gymnasium_agent` | Every `/step` returns a reward; `/run` accumulates them. | + +Pick one base protocol and build the Resources Server and Agent Server to match. `GymnasiumServer` exposes `/reset` and `/step` and does not expose `/verify`. `SimpleResourcesServer` defines `/seed_session` and `/verify` but can be extended with additional environment-specific routes. + + +The two base protocols do not describe every concrete Agent–Environment interface in the repository. Some environments extend the seed/verify protocol with custom transition endpoints. Other environments are embedded inside the Agent and do not use a Resources Server during `/run`. + + +### Seed/Verify Protocol Extensions + +Several environments keep the `/seed_session` and `/verify` lifecycle while adding bespoke transition APIs. These extensions require a matching Agent Server; sharing an endpoint name such as `/step` does not make their request and response contracts interchangeable. + +| Environment | Episode flow | Extension contract | +|---|---|---| +| Aviary | `/seed_session` → repeated `/step` → `/close` → `/verify` | `/step` receives an environment ID and a list of function-call actions, then returns an observation, reward, and `done`. | +| ToolSandbox | `/seed_session` → repeated `/step` → `/close` → `/verify` | `/step` accepts function calls or a natural-language response for the user simulator. `/close` finalizes environment scoring before `/verify`. | +| OpenEnv | `/seed_session` → interaction → `/verify` | Depending on its mode, interaction uses a custom `/step` action or direct per-tool HTTP routes. `/verify` returns the accumulated environment reward. | + +Some Agent Servers also use variants of the seed/verify sequence—for example, skipping a no-op `/seed_session`, treating it as optional, or calling `/verify` without tool execution. These are Agent Server behaviors built around the seed/verify API rather than new Resources Server base classes. + +### Agent-Embedded Environments + +Some Agent Servers embed the environment or an external benchmark harness in their own process. Examples include OSWorld, Tau2, SWE wrappers, Harbor, PinchBench, and Verifiers. Their `/run` implementation computes reward without calling a Resources Server, so they do not implement a Resources Server episode protocol. + +### Action and Tool Transports + +The episode protocol defines initialization, transitions, and scoring. The action transport separately defines how the Agent affects the Environment during the episode. + +| Transport | Interaction | +|---|---| +| Direct HTTP tools | The model emits a function call, and the Agent Server sends `POST /{tool_name}` to the Resources Server. | +| MCP | An MCP-capable harness calls tools through the Resources Server's `/mcp` endpoint. | +| Gymnasium step action | The Agent Server sends the complete model response to `/step`; the Resources Server interprets and executes it. | +| Bespoke transition API | A matching Agent Server and Resources Server exchange environment-specific `/step`, `/close`, or other requests. | +| Agent-embedded tools | The Agent Server or its benchmark harness executes tools internally without a Resources Server. | + +## Seed/Verify Protocol: `seed_session()` and `verify()` + +The seed/verify protocol brackets the Agent harness with environment initialization and final verification: + +```text +Client → Agent Server: POST /run +Agent Server → Resources Server: POST /seed_session +Agent Server run() → Agent Server responses(): POST /v1/responses + responses() ↔ Model Server + responses() → Resources Server tool endpoint + ... repeat the model/tool Agent loop +Agent Server → Resources Server: POST /verify +Agent Server → Client: completed trajectory and reward +``` + +For the built-in `simple_agent`, `run()` performs the lifecycle orchestration and `responses()` implements the Agent loop. The Agent Server exposes those methods as `/run` and `/v1/responses`, respectively. + +### Lifecycle API + +`SimpleResourcesServer` exposes: + +```python +async def seed_session( + self, + body: BaseSeedSessionRequest, +) -> BaseSeedSessionResponse: + ... + +async def verify( + self, + body: BaseVerifyRequest, +) -> BaseVerifyResponse: + ... +``` + +- **`body` in `seed_session()`** contains fields used to initialize an episode. Define a `BaseSeedSessionRequest` subclass for fields such as an initial board, task ID, or sandbox image. The default implementation is a no-op. +- **`body.responses_create_params` in `verify()`** is the original model request. +- **`body.response` in `verify()`** is the completed `NeMoGymResponse`, including the trajectory assembled by the Agent Server. +- **Custom task metadata** requires a `BaseVerifyRequest` subclass. Declare fields such as `answer` or `expected_state`; undeclared fields are not retained by the base Pydantic model. +- **`request: Request`**, when added to either method signature, provides access to the per-rollout session ID. + +### Direct HTTP Tool-Calling Protocol + +The seed/verify episode protocol does not itself define how the Agent acts on the environment. `simple_agent` supplies a direct HTTP function-tool protocol inside `responses()`: + +1. The input row declares function schemas in `responses_create_params.tools`. +2. The Agent Server sends those schemas to the Model Server. +3. The model returns a structured `function_call` with `name`, `arguments`, and `call_id`. +4. `simple_agent` parses `arguments` and sends `POST /{name}` to the Resources Server. +5. The Resources Server's response becomes a `function_call_output` associated with the same `call_id`. +6. `responses()` sends the updated trajectory back to the model and repeats until the model returns a final response or the step limit is reached. + +The tool name in the input schema must match the Resources Server route: + +```python +class SquareRequest(BaseModel): + value: int + + +class SquareResponse(BaseModel): + result: int + + +class CalculatorServer(SimpleResourcesServer): + def setup_webserver(self) -> FastAPI: + app = super().setup_webserver() + app.post("/square")(self.square) + return app + + async def square(self, body: SquareRequest) -> SquareResponse: + return SquareResponse(result=body.value**2) +``` + +For a model response such as: + +```json +{ + "type": "function_call", + "name": "square", + "arguments": "{\"value\": 4}", + "call_id": "call_123" +} +``` + +`simple_agent` calls `POST /square` with `{"value": 4}`. The Agent Server performs this dispatch; the Resources Server only implements the endpoint. + + +Resources Server tools can also be exposed over MCP. MCP changes the tool transport, not the episode protocol: the rollout can still use `/seed_session` and `/verify`. See [MCP Resources Server](/environment-tutorials/mcp-resources-server). + + +### Minimal Seed/Verify Environment + +The complete server below combines the tool route with final trajectory verification: + +```python +from fastapi import FastAPI +from pydantic import BaseModel + +from nemo_gym.base_resources_server import ( + BaseVerifyRequest, + BaseVerifyResponse, + SimpleResourcesServer, +) + + +class SquareRequest(BaseModel): + value: int + + +class SquareResponse(BaseModel): + result: int + + +class CalculatorServer(SimpleResourcesServer): + def setup_webserver(self) -> FastAPI: + app = super().setup_webserver() + app.post("/square")(self.square) + return app + + async def square(self, body: SquareRequest) -> SquareResponse: + return SquareResponse(result=body.value**2) + + async def verify(self, body: BaseVerifyRequest) -> BaseVerifyResponse: + used_square = any( + item.type == "function_call" and item.name == "square" + for item in body.response.output + ) + return BaseVerifyResponse( + responses_create_params=body.responses_create_params, + response=body.response, + reward=float(used_square), + ) + + +if __name__ == "__main__": + CalculatorServer.run_webserver() +``` + +Choose the seed/verify protocol when: + +- The Agent harness should own model calls, tool dispatch, retries, or other orchestration. +- Resources Server routes should be exposed directly as model tools. +- One final reward should be computed from the completed trajectory or final environment state. +- You are pairing the environment with `simple_agent` or a custom Agent Server that implements `/seed_session`, an Agent harness, and `/verify`. + +## Gymnasium Protocol: `reset()` and `step()` + +The Gymnasium protocol models the environment as a transition function. The Agent Server sends every model response to the Resources Server: + +```text +Client → Agent Server: POST /run +Agent Server run() → Resources Server: POST /reset +Agent Server run() → Model Server: POST /v1/responses +Agent Server run() → Resources Server: POST /step +... repeat model → step until terminated or truncated +Agent Server → Client: trajectory and accumulated reward +``` + +`gymnasium_agent` never calls `/verify`. `GymnasiumServer` does not register a `/verify` endpoint; `/step` provides both transitions and rewards. + +### Transition API + +Subclass `GymnasiumServer` and implement `step()`. Override `reset()` when the environment needs initialization or an initial observation. + +```python +async def reset( + self, + metadata: dict, + session_id: str | None = None, +) -> tuple[str | None, dict]: + ... + +async def step( + self, + action: NeMoGymResponse, + metadata: dict, + session_id: str | None = None, +) -> tuple[str | None, float, bool, bool, dict]: + ... +``` + +- **`action`** is the complete structured model response for the current step. It can contain assistant messages, reasoning items, or function calls. +- **`metadata`** contains extra top-level fields from the input row, such as `answer`, `category`, or task settings. It does not include `responses_create_params`. +- **`session_id`** identifies the rollout and keys per-episode state. Stateless environments can ignore it. + +`reset()` returns `(observation, info)`. A non-`None` observation is appended as a user message before the first model call. + +`step()` returns `(observation, reward, terminated, truncated, info)`: + +- **`observation`**: the next user message, or `None` when no message is needed. +- **`reward`**: reward for this step. `gymnasium_agent` sums rewards across the episode. +- **`terminated`**: `True` when the episode reaches a natural terminal state. +- **`truncated`**: `True` when a limit or timeout cuts the episode short. +- **`info`**: diagnostic metadata and optional `tool_outputs` for the Agent Server. + +### Tool Calls as Gymnasium Actions + +`gymnasium_agent` does not dispatch a function call to `POST /{name}`. Instead: + +1. The model returns a `NeMoGymResponse`. +2. The Agent Server sends that response to `/step` as `action`. +3. `step()` interprets text or structured function calls and executes the corresponding environment operation. +4. For function calls, `step()` returns results in `info["tool_outputs"]`. +5. The Agent Server converts those results into `function_call_output` items for the next model call. + +```python +import json + +from resources_servers.gymnasium import GymnasiumServer + + +class CalculatorEnv(GymnasiumServer): + async def step(self, action, metadata, session_id=None): + calls = [item for item in action.output if item.type == "function_call"] + + if calls: + outputs = [] + for call in calls: + args = json.loads(call.arguments) + if call.name == "square": + result = {"result": args["value"] ** 2} + else: + result = {"error": f"Unknown action: {call.name}"} + outputs.append(self.tool_output(call, result)) + + return None, 0.0, False, False, {"tool_outputs": outputs} + + return None, 1.0, True, False, {} +``` + +Tool schemas still come from `responses_create_params.tools`, but they describe the actions the model can emit. The Resources Server does not need matching HTTP tool routes because `step()` performs the dispatch. + +Choose the Gymnasium protocol when: + +- The Resources Server must process every model response before the episode continues. +- The environment decides the next observation and stopping condition. +- Rewards are naturally assigned per transition. +- The model response itself is the environment action. +- You are pairing the environment with `gymnasium_agent` or a custom Agent Server that implements the `/reset` and `/step` loop. + +## State Management + +Both episode protocols use the Resources Server's session middleware. It assigns a unique `session_id` cookie, and the Agent Server preserves that cookie across Resources Server calls in the same rollout. + +Store rollout-specific state under the session ID so concurrent episodes remain isolated. State can range from: + +- **Stateless**: grade an answer directly from the response and task metadata. +- **Lightweight**: store a counter, game board, or conversation state in memory. +- **Heavyweight**: associate each session with a browser, database, or Docker container. + +### Seed/Verify State + +`SimpleResourcesServer` does not define a state container. Add one and read the session ID from the FastAPI request: + +```python +from typing import Any + +from fastapi import Request +from pydantic import Field + +from nemo_gym.server_utils import SESSION_ID_KEY + + +class StatefulServer(SimpleResourcesServer): + session_state: dict[str, Any] = Field(default_factory=dict) + + async def seed_session(self, request: Request, body: SeedRequest): + session_id = request.session[SESSION_ID_KEY] + self.session_state[session_id] = initialize(body) + return BaseSeedSessionResponse() + + async def verify(self, request: Request, body: VerifyRequest): + session_id = request.session[SESSION_ID_KEY] + try: + reward = grade(self.session_state[session_id], body) + return make_verify_response(body, reward) + finally: + self.session_state.pop(session_id, None) +``` + +Tool endpoints can read the same `request.session[SESSION_ID_KEY]`, so `/seed_session`, tool calls, and `/verify` operate on the same environment instance. + +### Gymnasium State + +`GymnasiumServer` provides `self.session_state` and passes the session ID to `reset()` and `step()`: + +```python +class StatefulEnv(GymnasiumServer): + async def reset(self, metadata, session_id=None): + self.session_state[session_id] = initialize(metadata) + return "Environment ready.", {} + + async def step(self, action, metadata, session_id=None): + state = self.session_state[session_id] + reward, done = apply_action_and_score(state, action) + return None, reward, done, False, {} +``` + +Do not put rollout-specific mutable data in an unkeyed server attribute. + +### Session Cleanup + +- **Seed/verify protocol**: remove custom state in `verify()`, preferably in a `finally` block. `SimpleResourcesServer` does not automatically clear custom state. +- **Gymnasium protocol**: when `step()` returns `terminated=True` or `truncated=True`, `GymnasiumServer` calls `close_session()` and removes the entry from `self.session_state`. +- **Heavyweight state**: override `close_session()`, release the external resource, and call `await super().close_session(session_id)`. + +```python +async def close_session(self, session_id): + container = self.session_state.get(session_id, {}).get("container") + if container is not None: + await container.stop() + await super().close_session(session_id) +``` + +Ensure a Gymnasium environment returns a terminal or truncated result before the Agent Server's `max_steps` is exhausted. Also use timeouts or stale-session reclamation so client failures cannot leak external resources indefinitely. + +## YAML Configuration + +The Resources Server definition has the same YAML shape for both episode protocols. The paired Agent Server determines which protocol is used. + + + + +```yaml +calculator: + resources_servers: + calculator: + entrypoint: app.py + domain: knowledge + +calculator_agent: + responses_api_agents: + simple_agent: + entrypoint: app.py + resources_server: + type: resources_servers + name: calculator + model_server: + type: responses_api_models + name: policy_model + datasets: + - name: example + type: example + jsonl_fpath: resources_servers/calculator/data/example.jsonl +``` + + + + +```yaml +calculator: + resources_servers: + calculator: + entrypoint: app.py + domain: knowledge + +calculator_agent: + responses_api_agents: + gymnasium_agent: + entrypoint: app.py + resources_server: + type: resources_servers + name: calculator + model_server: + type: responses_api_models + name: policy_model + max_steps: 10 + datasets: + - name: example + type: example + jsonl_fpath: resources_servers/calculator/data/example.jsonl +``` + + + + +The Agent Server's `resources_server.name` must match the top-level Resources Server instance name. + +## Next Steps + +- Build a seed/verify environment in [Single-Step Environment](/environment-tutorials/single-step-environment). +- Add per-rollout state in [Stateful Environment](/environment-tutorials/stateful-environment). +- Expose Resources Server tools through [MCP Resources Server](/environment-tutorials/mcp-resources-server). +- Review common reward designs in [Verification Patterns](/build-verifiers/verification-patterns). +- See a multi-step Gymnasium implementation in [`resources_servers/blackjack`](https://github.com/NVIDIA-NeMo/Gym/tree/main/resources_servers/blackjack). diff --git a/fern/versions/latest/pages/environment-tutorials/single-step-environment.mdx b/fern/versions/latest/pages/environment-tutorials/single-step-environment.mdx index a921afbd42..173a39e903 100644 --- a/fern/versions/latest/pages/environment-tutorials/single-step-environment.mdx +++ b/fern/versions/latest/pages/environment-tutorials/single-step-environment.mdx @@ -1,7 +1,7 @@ --- title: "Single-Step Environment" description: "" -position: 2 +position: 3 --- import { NavButton } from "../../../../components/NavButton"; diff --git a/fern/versions/latest/pages/environment-tutorials/stateful-environment.mdx b/fern/versions/latest/pages/environment-tutorials/stateful-environment.mdx index fb449eac68..e69100e8a0 100644 --- a/fern/versions/latest/pages/environment-tutorials/stateful-environment.mdx +++ b/fern/versions/latest/pages/environment-tutorials/stateful-environment.mdx @@ -1,7 +1,7 @@ --- title: "Stateful Environment" description: "" -position: 4 +position: 5 --- import { NavButton } from "../../../../components/NavButton";