Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ policy_model_name: gpt-4.1-2025-04-14" > env.yaml
**Terminal 1 (start servers)**:
```bash
# Start servers (this will keep running)
config_paths="resources_servers/example_simple_weather/configs/simple_weather.yaml,\
config_paths="resources_servers/example_single_tool_call/configs/single_tool_call.yaml,\
responses_api_models/openai_model/configs/openai_model.yaml"
ng_run "+config_paths=[${config_paths}]"
```
Expand All @@ -104,7 +104,7 @@ echo '{"responses_create_params":{"input":[{"role":"developer","content":"You ar

# Collect verified rollouts
ng_collect_rollouts \
+agent_name=simple_weather_simple_agent \
+agent_name=single_tool_call_simple_agent \
+input_jsonl_fpath=weather_query.jsonl \
+output_jsonl_fpath=weather_rollouts.jsonl

Expand Down Expand Up @@ -162,11 +162,11 @@ NeMo Gym includes a curated collection of resource servers for training and eval
Purpose: Demonstrate NeMo Gym patterns and concepts.

<!-- START_EXAMPLE_ONLY_SERVERS_TABLE -->
| Name | Demonstrates | Config | README |
| ---------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------- |
| Multi Step | Multi-step tool calling | <a href='resources_servers/example_multi_step/configs/example_multi_step.yaml'>example_multi_step.yaml</a> | <a href='resources_servers/example_multi_step/README.md'>README</a> |
| Simple Weather | Basic single-step tool calling | <a href='resources_servers/example_simple_weather/configs/simple_weather.yaml'>simple_weather.yaml</a> | <a href='resources_servers/example_simple_weather/README.md'>README</a> |
| Stateful Counter | Session state management (in-memory) | <a href='resources_servers/example_stateful_counter/configs/stateful_counter.yaml'>stateful_counter.yaml</a> | <a href='resources_servers/example_stateful_counter/README.md'>README</a> |
| Name | Demonstrates | Config | README |
| ------------------ | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| Multi Step | Multi-step tool calling | <a href='resources_servers/example_multi_step/configs/example_multi_step.yaml'>example_multi_step.yaml</a> | <a href='resources_servers/example_multi_step/README.md'>README</a> |
| Session State Mgmt | Session state management (in-memory) | <a href='resources_servers/example_session_state_mgmt/configs/example_session_state_mgmt.yaml'>example_session_state_mgmt.yaml</a> | <a href='resources_servers/example_session_state_mgmt/README.md'>README</a> |
| Single Tool Call | Basic single-step tool calling | <a href='resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml'>example_single_tool_call.yaml</a> | <a href='resources_servers/example_single_tool_call/README.md'>README</a> |
<!-- END_EXAMPLE_ONLY_SERVERS_TABLE -->

### Table 2: Resource Servers for Training
Expand Down
12 changes: 6 additions & 6 deletions docs/about/concepts/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ responses_api_models/

# Resources Server Config
resources_servers/
└── example_simple_weather/
└── configs/simple_weather.yaml
└── example_single_tool_call/
└── configs/single_tool_call.yaml

# Agent Server Config
responses_api_agents/
Expand All @@ -45,16 +45,16 @@ Different server types have additional required fields (e.g., `domain` for resou
Config files in NeMo Gym often use the same name for both server ID and implementation:

```yaml
example_simple_weather: # ← Server ID
example_single_tool_call: # ← Server ID
resources_servers:
example_simple_weather: # ← Implementation
example_single_tool_call: # ← Implementation
```

These serve different purposes:

- **Server ID** (`example_simple_weather` on line 1): Your chosen identifier for this server instance. Used in API requests and when other servers reference it. You could name it `my_weather` or `weather_prod` instead.
- **Server ID** (`example_single_tool_call` on line 1): Your chosen identifier for this server instance. Used in API requests and when other servers reference it. You could name it `my_weather` or `weather_prod` instead.

- **Implementation** (`example_simple_weather` on line 3): Must match the folder `resources_servers/example_simple_weather/`. This tells NeMo Gym which code to run.
- **Implementation** (`example_single_tool_call` on line 3): Must match the folder `resources_servers/example_single_tool_call/`. This tells NeMo Gym which code to run.

Examples often use matching names for simplicity, but the two values are independent choices.

Expand Down
6 changes: 3 additions & 3 deletions docs/about/concepts/core-components.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Each example shows what **task** the agent solves, what **actions** are availabl
- **Actions**: None (evaluates response format/content)
- **Verification logic**: Checks if response follows all specified instructions

- **[`simple_weather`](https://github.com/NVIDIA-NeMo/Gym/tree/main/resources_servers/example_simple_weather)**: Mock weather API
- **[`example_single_tool_call`](https://github.com/NVIDIA-NeMo/Gym/tree/main/resources_servers/example_single_tool_call)**: Mock weather API
- **Task**: Report weather information
- **Actions**: `get_weather()` returns mock weather data
- **Verification logic**: Checks if weather tool was called correctly
Expand Down Expand Up @@ -104,9 +104,9 @@ your_agent_name: # server ID
your_agent_name: # agent type. name of the folder inside the server type folder
entrypoint: app.py # server entrypoint path, relative to the agent type folder
resources_server: # which resource server to use
name: simple_weather
name: example_single_tool_call
model_server: # which model server to use
name: policy_model
name: policy_model
```

:::
Expand Down
14 changes: 7 additions & 7 deletions docs/get-started/rollout-collection.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Make sure you have:
Look at the example dataset included with the Simple Weather resource server:

```bash
head -1 resources_servers/example_simple_weather/data/example.jsonl | python -m json.tool
head -1 resources_servers/example_single_tool_call/data/example.jsonl | python -m json.tool
```

Each line contains a `responses_create_params` object with:
Expand All @@ -61,21 +61,21 @@ If you still have servers running from the [Setup and Installation](setup-instal
If not, start them again:

```bash
config_paths="resources_servers/example_simple_weather/configs/simple_weather.yaml,\
config_paths="resources_servers/example_single_tool_call/configs/single_tool_call.yaml,\
responses_api_models/openai_model/configs/openai_model.yaml"
ng_run "+config_paths=[${config_paths}]"
```

**✅ Success Check**: You should see 3 Gym servers running including the `simple_weather_simple_agent`, along with the head server.
**✅ Success Check**: You should see 3 Gym servers running including the `single_tool_call_simple_agent`, along with the head server.

## 3. Generate Rollouts

In a separate terminal, run:

```bash
ng_collect_rollouts +agent_name=simple_weather_simple_agent \
+input_jsonl_fpath=resources_servers/example_simple_weather/data/example.jsonl \
+output_jsonl_fpath=results/simple_weather_rollouts.jsonl \
ng_collect_rollouts +agent_name=single_tool_call_simple_agent \
+input_jsonl_fpath=resources_servers/example_single_tool_call/data/example.jsonl \
+output_jsonl_fpath=results/single_tool_call_rollouts.jsonl \
+limit=5 \
+num_repeats=2 \
+num_samples_in_parallel=3
Expand Down Expand Up @@ -119,7 +119,7 @@ Collecting rollouts: 100%|████████████████| 5/5
Launch the rollout viewer:

```bash
ng_viewer +jsonl_fpath=results/simple_weather_rollouts.jsonl
ng_viewer +jsonl_fpath=results/single_tool_call_rollouts.jsonl
```

Then visit <http://127.0.0.1:7860>
Expand Down
2 changes: 1 addition & 1 deletion docs/get-started/setup-installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ Check your `env.yaml` file has the correct API key format.

```bash
# Define which servers to start
config_paths="resources_servers/example_simple_weather/configs/simple_weather.yaml,\
config_paths="resources_servers/example_single_tool_call/configs/single_tool_call.yaml,\
responses_api_models/openai_model/configs/openai_model.yaml"

# Start all servers
Expand Down
4 changes: 2 additions & 2 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ policy_model_name: gpt-4.1-2025-04-14" > env.yaml

```bash
# Start servers (this will keep running)
config_paths="resources_servers/example_simple_weather/configs/simple_weather.yaml,\
config_paths="resources_servers/example_single_tool_call/configs/single_tool_call.yaml,\
responses_api_models/openai_model/configs/openai_model.yaml"
ng_run "+config_paths=[${config_paths}]"
```
Expand All @@ -82,7 +82,7 @@ echo '{"responses_create_params":{"input":[{"role":"developer","content":"You ar

# Collect verified rollouts
ng_collect_rollouts \
+agent_name=simple_weather_simple_agent \
+agent_name=single_tool_call_simple_agent \
+input_jsonl_fpath=weather_query.jsonl \
+output_jsonl_fpath=weather_rollouts.jsonl

Expand Down
6 changes: 3 additions & 3 deletions docs/reference/cli-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ This command reads configuration from YAML files specified via `+config_paths` a

```bash
# Start servers with specific configs
config_paths="resources_servers/example_simple_weather/configs/simple_weather.yaml,\
config_paths="resources_servers/example_single_tool_call/configs/single_tool_call.yaml,\
responses_api_models/openai_model/configs/openai_model.yaml"
ng_run "+config_paths=[${config_paths}]"
```
Expand Down Expand Up @@ -78,7 +78,7 @@ Test a specific server module by running its pytest suite and optionally validat
**Example**

```bash
ng_test +entrypoint=resources_servers/example_simple_weather
ng_test +entrypoint=resources_servers/example_single_tool_call
```

---
Expand Down Expand Up @@ -177,7 +177,7 @@ Perform a batch of rollout collection.

```bash
ng_collect_rollouts \
+agent_name=simple_weather_simple_agent \
+agent_name=single_tool_call_simple_agent \
+input_jsonl_fpath=weather_query.jsonl \
+output_jsonl_fpath=weather_rollouts.jsonl \
+limit=100 \
Expand Down
4 changes: 2 additions & 2 deletions docs/reference/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Keep the server ID as `policy_model` — agent configs reference this name by de
```yaml
my_resource: # Server ID (your choice — agents reference this name)
resources_servers: # Server type (must be "resources_servers" for resources servers)
example_simple_weather: # Implementation (must match a directory in resources_servers/)
example_single_tool_call: # Implementation (must match a directory in resources_servers/)
entrypoint: app.py # Python file to run
domain: agent # Server category (see values below)
verified: false # Passed reward profiling and training checks (default: false)
Expand Down Expand Up @@ -126,7 +126,7 @@ policy_model_name: gpt-4o-2024-11-20
# Optional: store config paths for reuse
my_config_paths:
- responses_api_models/openai_model/configs/openai_model.yaml
- resources_servers/example_simple_weather/configs/simple_weather.yaml
- resources_servers/example_single_tool_call/configs/single_tool_call.yaml

# Optional: validation behavior
error_on_almost_servers: true # Exit on invalid configs (default: true)
Expand Down
6 changes: 3 additions & 3 deletions nemo_gym/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ class RunConfig(BaseNeMoGymCLIConfig):
Examples:

```bash
config_paths="resources_servers/example_simple_weather/configs/simple_weather.yaml,\\
config_paths="resources_servers/example_single_tool_call/configs/single_tool_call.yaml,\\
responses_api_models/openai_model/configs/openai_model.yaml"
ng_run "+config_paths=[${config_paths}]"
```
Expand All @@ -122,7 +122,7 @@ class TestConfig(RunConfig):
Examples:

```bash
ng_test +entrypoint=resources_servers/example_simple_weather
ng_test +entrypoint=resources_servers/example_single_tool_call
```
"""

Expand Down Expand Up @@ -383,7 +383,7 @@ def run(

```bash
# Start servers with specific configs
config_paths="resources_servers/example_simple_weather/configs/simple_weather.yaml,\\
config_paths="resources_servers/example_single_tool_call/configs/single_tool_call.yaml,\\
responses_api_models/openai_model/configs/openai_model.yaml"
ng_run "+config_paths=[${config_paths}]"
```
Expand Down
2 changes: 1 addition & 1 deletion nemo_gym/rollout_collection.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ class RolloutCollectionConfig(BaseNeMoGymCLIConfig):

```bash
ng_collect_rollouts \
+agent_name=simple_weather_simple_agent \
+agent_name=single_tool_call_simple_agent \
+input_jsonl_fpath=weather_query.jsonl \
+output_jsonl_fpath=weather_rollouts.jsonl \
+limit=100 \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@

server_client = ServerClient.load_from_global_config()
task = server_client.post(
server_name="stateful_counter_simple_agent",
server_name="example_session_state_mgmt_simple_agent",
url_path="/run",
json={
"responses_create_params": NeMoGymResponseCreateParamsNonStreaming(
Expand Down
Original file line number Diff line number Diff line change
@@ -1,37 +1,37 @@
example_stateful_counter_resources_server:
example_session_state_mgmt_resources_server:
resources_servers:
example_stateful_counter:
example_session_state_mgmt:
entrypoint: app.py
domain: agent
verified: false
description: Session state management (in-memory)
stateful_counter_simple_agent:
example_session_state_mgmt_simple_agent:
responses_api_agents:
simple_agent:
entrypoint: app.py
resources_server:
type: resources_servers
name: stateful_counter_resources_server
name: example_session_state_mgmt_resources_server
model_server:
type: responses_api_models
name: policy_model
datasets:
- name: train
type: train
jsonl_fpath: resources_servers/example_stateful_counter/data/train.jsonl
jsonl_fpath: resources_servers/example_session_state_mgmt/data/train.jsonl
gitlab_identifier:
dataset_name: stateful_counter
dataset_name: session_state_mgmt
version: 0.0.1
artifact_fpath: train.jsonl
license: Apache 2.0
- name: validation
type: validation
jsonl_fpath: resources_servers/example_stateful_counter/data/validation.jsonl
jsonl_fpath: resources_servers/example_session_state_mgmt/data/validation.jsonl
gitlab_identifier:
dataset_name: stateful_counter
dataset_name: session_state_mgmt
version: 0.0.1
artifact_fpath: validation.jsonl
license: Apache 2.0
- name: example
type: example
jsonl_fpath: resources_servers/example_stateful_counter/data/example.jsonl
jsonl_fpath: resources_servers/example_session_state_mgmt/data/example.jsonl
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

# Run as `python resources_servers/example_stateful_counter/create_examples.py`
# Run as `python resources_servers/example_session_state_mgmt/create_examples.py`
import json
from copy import deepcopy

Expand Down Expand Up @@ -78,5 +78,5 @@

examples.append(json.dumps(example) + "\n")

with open("resources_servers/example_stateful_counter/data/example.jsonl", "w") as f:
with open("resources_servers/example_session_state_mgmt/data/example.jsonl", "w") as f:
f.writelines(examples)
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "example",
"type": "example",
"jsonl_fpath": "resources_servers/example_stateful_counter/data/example.jsonl",
"jsonl_fpath": "resources_servers/example_session_state_mgmt/data/example.jsonl",
"gitlab_identifier": null,
"license": null,
"Number of examples": 5,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
from httpx import Cookies

from nemo_gym.server_utils import ServerClient
from resources_servers.example_stateful_counter.app import (
from resources_servers.example_session_state_mgmt.app import (
StatefulCounterResourcesServer,
StatefulCounterResourcesServerConfig,
)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Description

This is an example environment with simple tool calls. The example data can be found in `example_simple_weather/data/example.jsonl`.
This is an example environment with simple tool calls. The example data can be found in `example_single_tool_call/data/example.jsonl`.


# Licensing information
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@

server_client = ServerClient.load_from_global_config()
task = server_client.post(
server_name="my_simple_weather_server",
server_name="my_single_tool_call_server",
url_path="/get_weather",
json={
"city": "San Francisco, CA",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
example_simple_weather:
example_single_tool_call:
resources_servers:
example_simple_weather:
example_single_tool_call:
entrypoint: app.py
domain: agent
verified: false
description: Basic single-step tool calling
simple_weather_simple_agent:
single_tool_call_simple_agent:
responses_api_agents:
simple_agent:
entrypoint: app.py
resources_server:
type: resources_servers
name: example_simple_weather
name: example_single_tool_call
model_server:
type: responses_api_models
name: policy_model
datasets:
- name: example
type: example
jsonl_fpath: resources_servers/example_simple_weather/data/example.jsonl
jsonl_fpath: resources_servers/example_single_tool_call/data/example.jsonl
Original file line number Diff line number Diff line change
Expand Up @@ -63,5 +63,5 @@
example_strs.append(json.dumps({"responses_create_params": example.model_dump(exclude_unset=True)}) + "\n")


with open("resources_servers/example_simple_weather/data/example.jsonl", "w") as f:
with open("resources_servers/example_single_tool_call/data/example.jsonl", "w") as f:
f.writelines(example_strs)
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "example",
"type": "example",
"jsonl_fpath": "resources_servers/example_simple_weather/data/example.jsonl",
"jsonl_fpath": "resources_servers/example_single_tool_call/data/example.jsonl",
"gitlab_identifier": null,
"license": null,
"Number of examples": 5,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
from unittest.mock import MagicMock

from nemo_gym.server_utils import ServerClient
from resources_servers.example_simple_weather.app import (
from resources_servers.example_single_tool_call.app import (
SimpleWeatherResourcesServer,
SimpleWeatherResourcesServerConfig,
)
Expand Down
Loading