docs: End-to-end GRPO Training with NeMo RL tutorial [master branch]

docs/about/concepts/key-terminology.md

@@ -68,15 +68,18 @@ Reward / Reward Signal
 SFT (Supervised Fine-Tuning)
     Training approach using examples of good model behavior. Shows successful rollouts as training data.
 
-DPO (Direct Preference Optimization)
-    Training approach using pairs of rollouts where one is preferred over another. Teaches better vs worse responses.
-
 RL (Reinforcement Learning)
     Training approach where models learn through trial-and-error interaction with environments using reward signals.
 
 Online vs Offline Training
-    - **Online**: Model learns while interacting with environment in real-time (RL)
-    - **Offline**: Model learns from pre-collected rollout data (SFT/DPO)
+    - **Online**: Model learns while interacting with environment in real-time
+    - **Offline**: Model learns from pre-collected rollout data
+
+DPO (Direct Preference Optimization)
+    An offline RL training approach using pairs of rollouts where one is preferred over another. Teaches better vs worse responses.
+
+GRPO (Group Relative Policy Optimization)
+    Reinforcement learning algorithm that optimizes policies by comparing groups of rollouts relative to each other. Used for online RL training with language models.
 ```
 
 ## Interaction Patterns

docs/contribute/rl-framework-integration/index.md

@@ -8,7 +8,7 @@ These guides cover how to integrate NeMo Gym into a new RL training framework. U
 - Contributing NeMo Gym integration for a training framework that does not have one yet
 
 :::{tip}
-Just want to train models? Use {ref}`NeMo RL <rl-training-with-nemo-rl>` instead.
+Just want to train models? Use {ref}`NeMo RL <training-nemo-rl-grpo-index>` instead.
 :::
 
 ## Prerequisites

docs/index.md

@@ -108,6 +108,7 @@ Collect and view rollouts
 
 ::::
 
+<!-- This section needs to match the content in docs/tutorials/index.md -->
 ## Tutorials
 
 Hands-on tutorials to build and customize your training environments.
@@ -120,23 +121,23 @@ Hands-on tutorials to build and customize your training environments.
 :link-type: doc
 Implement or integrate existing tools and define task verification logic.
 +++
-{bdg-secondary}`custom-environments` {bdg-secondary}`tools`
+{bdg-primary}`beginner` {bdg-secondary}`30 min` {bdg-secondary}`custom-environments` {bdg-secondary}`tools`
 :::
 
-:::{grid-item-card} {octicon}`database;1.5em;sd-mr-1` Offline Training (SFT, DPO)
-:link: tutorials/offline-training-w-rollouts
-:link-type: doc
-Train with SFT or DPO using collected rollouts.
+:::{grid-item-card} {octicon}`workflow;1.5em;sd-mr-1` Offline Training with Rollouts
+:link: offline-training-w-rollouts
+:link-type: ref
+Transform rollouts into training data for {term}`supervised fine-tuning (SFT) <SFT (Supervised Fine-Tuning)>` and {term}`direct preference optimization (DPO) <DPO (Direct Preference Optimization)>`.
 +++
 {bdg-secondary}`sft` {bdg-secondary}`dpo`
 :::
 
-:::{grid-item-card} {octicon}`zap;1.5em;sd-mr-1` RL Training with NeMo RL
-:link: tutorials/rl-training-with-nemo-rl
-:link-type: doc
-Train with GRPO using NeMo RL and NeMo Gym.
+:::{grid-item-card} {octicon}`workflow;1.5em;sd-mr-1` GRPO with NeMo RL
+:link: training-nemo-rl-grpo-index
+:link-type: ref
+Learn how to set up NeMo Gym and NeMo RL training environments, run tests, prepare data, and launch single-node and multi-node training runs.
 +++
-{bdg-secondary}`grpo` {bdg-secondary}`nemo-rl`
+{bdg-primary}`training` {bdg-secondary}`rl` {bdg-secondary}`grpo`
 :::
 
 ::::
@@ -200,8 +201,8 @@ Rollout Collection <get-started/rollout-collection.md>
 
 tutorials/index.md
 tutorials/creating-resource-server
+tutorials/nemo-rl-grpo/index.md
 tutorials/offline-training-w-rollouts
-tutorials/rl-training-with-nemo-rl
 ```
 
 ```{toctree}

docs/tutorials/creating-resource-server.md

@@ -2,7 +2,7 @@
 
 # Creating a Resource Server
 
-Learn how to create a custom resource server to implement tools, verifiers, and business logic for your AI agents.
+Learn how to create a custom resource server to implement tools, verifiers, and business logic for your training environment.
 
 ::::{grid} 2
 :gutter: 3
@@ -24,19 +24,19 @@ Learn how to create a custom resource server to implement tools, verifiers, and
 
 ## What is a Resource Server?
 
-Resource servers are the backbone of tool-based agent interactions in NeMo Gym. They provide:
+Resource servers are the backbone of tool-based interactions in NeMo Gym. They provide:
 
-- **Tool implementations**: APIs that agents can call to perform actions or retrieve information
-- **Verification logic**: Functions to evaluate agent performance and compute rewards
-- **Business logic abstraction**: Clean separation between agent logic and domain-specific functionality
+- **Tool implementations**: APIs that models can call to perform actions or retrieve information
+- **Verification logic**: Functions to evaluate model performance and compute rewards
+- **Business logic abstraction**: Clean separation between model logic and domain-specific functionality
 
-Each resource server must implement a `verify` function that evaluates the agent's interactions and returns a reward signal for reinforcement learning.
+Each resource server must implement a `verify` function that evaluates the model's interactions and returns a reward signal for reinforcement learning.
 
 ---
 
 ## 1. Initialize the Resource Server
 
-Resource servers live in the `resources_servers/` directory. Create a weather server that provides weather information to agents.
+Resource servers live in the `resources_servers/` directory. Create a weather server that provides weather information to models.
 
 Run the initialization command from the repository root:
 
@@ -153,9 +153,9 @@ class MyWeatherResourcesServer(SimpleResourcesServer):
 
     async def verify(self, body: BaseVerifyRequest) -> BaseVerifyResponse:
         """
-        Verification function: Evaluate agent performance.
+        Verification function: Evaluate rollout performance.
 
-        This function is called after an agent completes an interaction.
+        This function is called after a rollout completes.
         Return a reward between 0.0 and 1.0.
 
         For this simple example, we always return 1.0 (success).
@@ -175,7 +175,7 @@ if __name__ == "__main__":
 3. **Server Class**: Extends `SimpleResourcesServer` and implements tools and verification
 4. **`setup_webserver()`**: Registers FastAPI routes for your tools
 5. **Tool Methods**: Async functions that implement the actual tool logic
-6. **`verify()`**: **Required** method that evaluates agent performance and returns a reward
+6. **`verify()`**: **Required** method that evaluates task performance and returns a reward
 
 ---
 
@@ -293,15 +293,15 @@ policy_base_url: https://api.openai.com/v1
 policy_model_name: gpt-4o-mini
 ```
 
-### Test the Agent
+### Test the resources server
 
-After the servers start, test your agent in a new terminal:
+After the servers start, test your resources server in a new terminal:
 
 ```bash
 python responses_api_agents/simple_agent/client.py
 ```
 
-The agent should be able to use your `get_weather` tool to answer questions about weather!
+The model should be able to use your `get_weather` tool to answer questions about weather!
 
 ---
 
@@ -319,7 +319,7 @@ Your resource server needs example data for testing and validation. Create `reso
 
 ### Generate Example Rollouts
 
-Collect rollouts by running the agent against your example inputs. This generates interaction traces showing how agents use your tools:
+Collect rollouts by running against your example inputs. This generates interaction traces showing how models use your tools:
 
 ```bash
 ng_collect_rollouts +agent_name=my_weather_tool_simple_agent \
@@ -331,7 +331,7 @@ ng_collect_rollouts +agent_name=my_weather_tool_simple_agent \
 ```
 
 :::{note}
-Ensure your servers are running (from step 6) before collecting rollouts. The command processes each input example, runs it through the agent, and saves the complete interaction including tool calls and verification rewards to `example_rollouts.jsonl`.
+Ensure your servers are running (from step 6) before collecting rollouts. The command processes each input example, runs it through the servers, and saves the complete interaction including tool calls and verification rewards to `example_rollouts.jsonl`.
 :::
 
 ---
@@ -386,14 +386,14 @@ Before submitting a PR, ensure you have:
 
 ## Advanced: Custom Verification
 
-For more sophisticated verification, you can implement custom logic in the `verify` function. Here's an example that checks if the agent used the correct tool:
+For more sophisticated verification, you can implement custom logic in the `verify` function. Here's an example that checks if the model used the correct tool:
 
 ```python
 async def verify(self, body: BaseVerifyRequest) -> BaseVerifyResponse:
     """
-    Advanced verification: Check if agent called the get_weather tool.
+    Advanced verification: Check if model called the get_weather tool.
     """
-    # Check if the agent made a function call
+    # Check if the model made a function call
     used_tool = False
     for output in body.response.output:
         if output.type == "function_call" and output.name == "get_weather":
@@ -420,7 +420,7 @@ Now that you have a working resource server:
 1. **Add training data**: Collect rollouts and prepare datasets for RL training
 2. **Add complex verification**: Add reward shaping and detailed performance metrics
 3. **Scale up**: Add more tools and more sophisticated business logic
-4. **Integrate with RL**: Use {doc}`rl-training-with-nemo-rl` to train agents on your tasks
+4. **Integrate with RL**: Use {ref}`RL Training with NeMo RL using GRPO <training-nemo-rl-grpo-index>` to train models on your tasks
 
 ::::{grid} 2
 :gutter: 3
@@ -432,9 +432,9 @@ Learn how to collect and process rollouts for training data.
 :::
 
 :::{grid-item-card} {octicon}`rocket;1.5em;sd-mr-1` Train with NeMo RL
-:link: rl-training-with-nemo-rl
-:link-type: doc
-Train agents using your resource server with NeMo RL.
+:link: training-nemo-rl-grpo-index
+:link-type: ref
+Train models using your resource server with NeMo RL.
 :::
 
 ::::
@@ -480,8 +480,8 @@ You've learned how to:
 ✅ Configure the required `domain` field  
 ✅ Add tools and verification logic  
 ✅ Write and run tests  
-✅ Run your server with an agent  
+✅ Run your server with an model
 ✅ Create required data artifacts  
 
-Resource servers are the foundation for building custom RL environments in NeMo Gym. Experiment with different tool implementations and verification strategies to create engaging tasks for your agents!
+Resource servers are the foundation for building custom RL environments in NeMo Gym. Experiment with different tool implementations and verification strategies to create engaging tasks for your models!
 

docs/tutorials/index.md

@@ -2,7 +2,7 @@
 
 # NeMo Gym Tutorials
 
-Hands-on learning experiences that guide you through building, training, and deploying AI agents with NeMo Gym.
+Hands-on tutorials to build and customize your training environments.
 
 :::{tip}
 **New to NeMo Gym?** Begin with the {doc}`Get Started <../get-started/index>` section for a guided tutorial from installation through your first verified agent. Return here afterward to learn about advanced topics like additional rollout collection methods and training data generation. You can find the project repository on [GitHub](https://github.com/NVIDIA-NeMo/Gym).
@@ -19,9 +19,9 @@ Create custom resource servers and implement tool-based agent interactions.
 :::{grid-item-card} {octicon}`tools;1.5em;sd-mr-1` Creating a Resource Server
 :link: creating-resource-server
 :link-type: doc
-Build custom resource servers with tools, verification logic, and business logic for your AI agents.
+Implement or integrate existing tools and define task verification logic.
 +++
-{bdg-primary}`beginner` {bdg-secondary}`30 min`
+{bdg-primary}`beginner` {bdg-secondary}`30 min` {bdg-secondary}`custom-environments` {bdg-secondary}`tools`
 :::
 
 ::::
@@ -43,12 +43,21 @@ Transform rollouts into training data for {term}`supervised fine-tuning (SFT) <S
 {bdg-secondary}`sft` {bdg-secondary}`dpo`
 :::
 
-:::{grid-item-card} {octicon}`workflow;1.5em;sd-mr-1` RL Training with NeMo RL
-:link: rl-training-with-nemo-rl
-:link-type: doc
-Train a model with NeMo RL. Learn how to set up NeMo Gym and NeMo RL training environments, run tests, prepare data, and launch single-node and multi-node training runs.
+::::
+
+---
+
+## RL Training
+
+::::{grid} 1 1 1 1
+:gutter: 1 1 1 2
+
+:::{grid-item-card} {octicon}`workflow;1.5em;sd-mr-1` GRPO with NeMo RL
+:link: training-nemo-rl-grpo-index
+:link-type: ref
+Learn how to set up NeMo Gym and NeMo RL training environments, run tests, prepare data, and launch single-node and multi-node training runs.
 +++
-{bdg-secondary}`rl` {bdg-secondary}`training`
+{bdg-primary}`training` {bdg-secondary}`rl` {bdg-secondary}`grpo`
 :::
 
 ::::