Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions docs/tutorials/nemo-rl-grpo/about-workplace-assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,22 @@

# About the Workplace Assistant Training Environment

The Workplace Assistant is a **{term}`multi-step <Multi-step>` agentic {term}`tool-use <Tool Use / Function Calling>` {term}`training environment <Training environment>`** that tests an model's ability to execute business tasks in a simulated workplace setting.
The Workplace Assistant is a **{term}`multi-step <Multi-step>` agentic {term}`tool-use <Tool Use / Function Calling>` {term}`training environment <Training environment>`** that tests a model's ability to execute business tasks in a simulated workplace setting.

**Generally, the model must:**
1. Understand the user's intent from natural language
2. Determine which tools to call and in what order
3. Infer correct parameters (e.g., look up email addresses, find matching customer records)
3. Infer correct parameters (for example, look up email addresses or find matching customer records)
4. Execute all necessary steps to complete the task


## Task examples
## Task Examples

Each task is a natural language request that the model must complete using the available tools. All tasks share the same set of tools that allow to model to retrieve more information or perform actions. Each {term}`task instance <Task Instance>` uses a separate set of database instances so that information retrieved or actions performed are specific to that task instance only. The model needs to accomplish the task within 6 {term}`steps <Multi-step>`.

- **Databases**: Email, Calendar, Analytics, Project Management, Customer Relationship Manager (CRM)
- **Tools**: distributed across these databases
- **Tasks**: common business activities (e.g., sending emails, scheduling meetings, managing projects)
- **Tasks**: common business activities (such as sending emails, scheduling meetings, and managing projects)

In the examples below, you will see the exact {term}`Responses create params <Responses API>` that will be provided to the model. The model is provided with:
1. Surrounding contextual information in the system prompt (first input with role `system`) like what the current day is.
Expand Down Expand Up @@ -85,7 +85,7 @@ The `email_send_email` tool will be executed with the above parameters. Within t
6. `customer_relationship_manager_update_customer(customer_id="00000035", field="assigned_to_email", new_value="john.smith@atlas.com")`


## Key implementation details
## Key Implementation Details
The environment is implemented as a FastAPI-based resource server that executes tools and verification. Here's how it works:

### Session Management
Expand All @@ -108,7 +108,7 @@ async def seed_session(self, request: Request, body: BaseSeedSessionRequest):

This ensures each task starts with a clean slate and tool calls from different rollouts don't interfere.

### Tool implementations
### Tool Implementations

Tool calls are routed to Python functions:

Expand Down Expand Up @@ -172,7 +172,7 @@ def is_correct(predicted_actions, ground_truth_actions, error):
)
```

**Why State-matching verification?**:
**Why State-Matching Verification?**
- **Flexibility**: Multiple valid solution paths exist for the same task
- **Robustness**: Model can recover from mistakes mid-trajectory
- **Goal-oriented**: Focuses on outcomes, not specific procedures
2 changes: 1 addition & 1 deletion docs/tutorials/nemo-rl-grpo/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ In this tutorial, you will:

:::{tab-item} Hardware Requirements

**Minimum** 1 node of 8× NVIDIA GPUs with 80GB or more memory each (e.g., H100, A100) is required.
**Minimum** 1 node of 8× NVIDIA GPUs with 80GB or more memory each (such as H100 or A100) is required.

NeMo Gym does not require GPUs. GPUs are only necessary for GRPO training with NeMo RL.

Expand Down
6 changes: 2 additions & 4 deletions docs/tutorials/nemo-rl-grpo/multi-node-training.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@

# Multi-Node Training

TODO STILL

:::{important}
**Do the {doc}`Single Node Training <single-node-training>` first. Do not skip it.** The single-node setup validates that your environment is configured correctly before attempting multi-node training.
:::
Expand All @@ -14,7 +12,7 @@ TODO STILL

For production training, scale to multiple nodes by changing `cluster.num_nodes`. This example uses **batch mode** (the `COMMAND` variable specifies what to run automatically when the job starts).

> **Note**: Run this command from the **Slurm login/head node**, not from inside the interactive container from Step 1. This submits a new batch job that will run independently.
> **Note**: Run this command from the **Slurm login/head node**, not from inside the interactive container through Step 1. This submits a new batch job that will run independently.

```bash
# Set experiment name
Expand All @@ -39,7 +37,7 @@ sbatch \
```


## Multi node
## Multi-Node

We will run a multi-node training job on a Slurm cluster. First, we will write our Slurm job launch script and then run it.

Expand Down
2 changes: 1 addition & 1 deletion docs/tutorials/nemo-rl-grpo/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

**Estimated Time:** ~5 minutes

Launch an interactive Slurm session to run training commands. See the [NeMo RL Cluster Setup documentation](https://docs.nvidia.com/nemo/rl/latest/cluster.html#interactive-launching) for more details.
Launch an interactive Slurm session to run training commands. Refer to the [NeMo RL Cluster Setup documentation](https://docs.nvidia.com/nemo/rl/latest/cluster.html#interactive-launching) for more details.

```bash
NUM_ACTOR_NODES=1
Expand Down
14 changes: 5 additions & 9 deletions docs/tutorials/nemo-rl-grpo/single-node-training.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,7 @@

# Single Node Training

TODO STILL

### Single Node Training (interactive mode)
### Single-Node Training (Interactive Mode)

**Estimated Time:** ~2-4 hours

Expand All @@ -27,8 +25,6 @@ CONFIG_PATH=examples/penguin/grpo_workplace_assistant_nemotron_nano_v2_9b.yaml
# HF_TOKEN: Your Hugging Face token for model downloads
# WANDB_API_KEY: Your Weights & Biases API key for logging
# TORCH_CUDA_ARCH_LIST: CUDA architectures compute capability
<!-- @import "[TOC]" {cmd="toc" depthFrom=1 depthTo=6 orderedList=false} -->

# NRL_FORCE_REBUILD_VENVS: Set to true on first run to rebuild venvs
TORCH_CUDA_ARCH_LIST="9.0 10.0" \
HF_HOME=.cache/ \
Expand Down Expand Up @@ -84,9 +80,9 @@ Training is successful when:

### Measuring Real-World Improvement

The Workplace Assistant environment's tool-calling tasks correlate with performance on the [Berkeley Function Calling Leaderboard (BFCL) v3](https://gorilla.cs.berkeley.edu/leaderboard.html) benchmark. To measure improvement, evaluate the Nemotron Nano v2 9B model on BFCL v3 before and after training and compare. You should observe measurable improvement in tool-calling accuracy
The Workplace Assistant environment's tool-calling tasks correlate with performance on the [Berkeley Function Calling Leaderboard (BFCL) v3](https://gorilla.cs.berkeley.edu/leaderboard.html) benchmark. To measure improvement, evaluate the Nemotron Nano v2 9B model on BFCL v3 before and after training, and compare the results. You should observe measurable improvement in tool-calling accuracy.

You can run BFCL v3 evaluations using [NeMo Evaluator](https://github.com/NVIDIA-NeMo/Evaluator), which supports BFCL v3. See the [NeMo Evaluator docs](https://github.com/NVIDIA-NeMo/Evaluator#-supported-benchmarks-and-evaluation-harnesses) for full setup instructions and supported benchmarks.
You can run BFCL v3 evaluations using [NeMo Evaluator](https://github.com/NVIDIA-NeMo/Evaluator), which supports BFCL v3. Refer to the [NeMo Evaluator docs](https://github.com/NVIDIA-NeMo/Evaluator#-supported-benchmarks-and-evaluation-harnesses) for full setup instructions and supported benchmarks.

---

Expand All @@ -112,15 +108,15 @@ results/<EXP_NAME>/ # Checkpoints and metrics



### Single node training
### Single-Node Training

Now you're ready to launch a single-node training run to verify your setup works correctly before scaling to multi-node training.

**Prerequisites for this step**:

1. Completed the `ng_prepare_data` command above (data should be in `data/bytedtsinghua_dapo17k/`)
2. W&B API key (see [Prerequisites](#prerequisites) section above)
3. HuggingFace token configured (see setup steps above)
3. HuggingFace token configured (refer to setup steps above)


```bash
Expand Down
Loading