From dac1f76b2b7808360b2327f6b776845b24262b70 Mon Sep 17 00:00:00 2001 From: sergiopaniego Date: Tue, 5 May 2026 09:38:25 +0200 Subject: [PATCH 1/5] docs(tutorials): rewrite tutorials index with per-tutorial descriptions Replaces the minimal bullet list with a table describing what each tutorial covers and whether a GPU is required. Includes all eight tutorials from the current and in-progress PR set. Co-Authored-By: Claude Sonnet 4.6 --- docs/source/tutorials/index.md | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/source/tutorials/index.md b/docs/source/tutorials/index.md index 0bc7be4d7a..ec72070b70 100644 --- a/docs/source/tutorials/index.md +++ b/docs/source/tutorials/index.md @@ -1,21 +1,25 @@ # Tutorials -Welcome to the OpenEnv tutorials! These guides will help you get started with using and building environments with OpenEnv. - -## Getting Started - -If you're new to OpenEnv, we recommend starting with the [Getting Started](/auto_getting_started/index) series to understand the core concepts and basic usage patterns. - -## Available Tutorials - -- **[OpenEnv Tutorial](openenv-tutorial.md)** - A comprehensive introduction to OpenEnv, covering installation, basic usage, and core concepts. -- **[Wordle GRPO Training](wordle-grpo.md)** - Learn how to train an agent to play Wordle using Group Relative Policy Optimization (GRPO). -- **[RL Training with 2048](rl-training-2048.md)** - Train a language model to play 2048 using GRPO reinforcement learning. *(GPU Required)* +| Tutorial | What it covers | GPU | +|----------|---------------|-----| +| [OpenEnv Tutorial](openenv-tutorial.md) | Full introduction to OpenEnv: install, connect to a hosted environment, step through an episode, define a reward function, and run a basic training loop. Start here if you're new. | No | +| [Wordle GRPO](wordle-grpo.md) | Train an agent to play Wordle using GRPO via TRL's `environment_factory`. Shows the multi-turn tool-calling loop: the model guesses a word each turn and receives letter-position feedback until it wins or the episode ends. | Yes | +| [RL Training with 2048](rl-training-2048.md) | Train a language model to play the 2048 tile-sliding game with GRPO. Covers game-state representation and reward shaping for a puzzle environment. | Yes | +| [End-to-end walkthrough](end-to-end-walkthrough.md) | The full pipeline in one page: connect to `reasoning_gym`, wire it into TRL via `environment_factory`, fine-tune with GRPO on `chain_sum`, read the reward delta from training logs, and push the checkpoint to the Hub. | Yes | +| [SFT warm-up for GRPO](sft-warmup.md) | When a base model's initial reward is near zero, use SFT to bootstrap format compliance before starting RL. Covers: collecting teacher rollouts with `CollectRunner`, filtering by reward, training with `SFTTrainer`, and measuring the before/after delta. | Yes | +| [Rubrics: composable reward computation](rubrics.md) | Build multi-criteria reward functions from reusable primitives (`WeightedSum`, `LLMJudge`, `Gate`). Shows how to compose, weight, and introspect rubrics inside an environment's `step()` method. | No | +| [MCP tools in OpenEnv environments](mcp-environment.md) | How to use the Model Context Protocol (MCP) tool interface across training, offline eval, and external serving. Covers `ListToolsAction`, `CallToolAction`, and the MCP lifecycle from reset to close. | No | +| [Evaluating agents with Inspect AI](evaluation-inspect.md) | Wrap an OpenEnv environment in an Inspect AI `Task` (dataset + solver + scorer), run it via `InspectAIHarness`, and get a structured `EvalResult` with accuracy scores. No training required — useful as a standalone eval pass on any checkpoint. | No | ```{toctree} -:maxdepth: 2 +:maxdepth: 1 :hidden: openenv-tutorial wordle-grpo rl-training-2048 +end-to-end-walkthrough +sft-warmup +rubrics +mcp-environment +evaluation-inspect ``` From 1ce6d1f5457fa480e29a2e97216594065106e44b Mon Sep 17 00:00:00 2001 From: sergiopaniego Date: Tue, 5 May 2026 09:43:10 +0200 Subject: [PATCH 2/5] docs(tutorials): add Open in Colab column to tutorials index table Co-Authored-By: Claude Sonnet 4.6 --- docs/source/tutorials/index.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/source/tutorials/index.md b/docs/source/tutorials/index.md index ec72070b70..d47b2fbe73 100644 --- a/docs/source/tutorials/index.md +++ b/docs/source/tutorials/index.md @@ -1,15 +1,15 @@ # Tutorials -| Tutorial | What it covers | GPU | -|----------|---------------|-----| -| [OpenEnv Tutorial](openenv-tutorial.md) | Full introduction to OpenEnv: install, connect to a hosted environment, step through an episode, define a reward function, and run a basic training loop. Start here if you're new. | No | -| [Wordle GRPO](wordle-grpo.md) | Train an agent to play Wordle using GRPO via TRL's `environment_factory`. Shows the multi-turn tool-calling loop: the model guesses a word each turn and receives letter-position feedback until it wins or the episode ends. | Yes | -| [RL Training with 2048](rl-training-2048.md) | Train a language model to play the 2048 tile-sliding game with GRPO. Covers game-state representation and reward shaping for a puzzle environment. | Yes | -| [End-to-end walkthrough](end-to-end-walkthrough.md) | The full pipeline in one page: connect to `reasoning_gym`, wire it into TRL via `environment_factory`, fine-tune with GRPO on `chain_sum`, read the reward delta from training logs, and push the checkpoint to the Hub. | Yes | -| [SFT warm-up for GRPO](sft-warmup.md) | When a base model's initial reward is near zero, use SFT to bootstrap format compliance before starting RL. Covers: collecting teacher rollouts with `CollectRunner`, filtering by reward, training with `SFTTrainer`, and measuring the before/after delta. | Yes | -| [Rubrics: composable reward computation](rubrics.md) | Build multi-criteria reward functions from reusable primitives (`WeightedSum`, `LLMJudge`, `Gate`). Shows how to compose, weight, and introspect rubrics inside an environment's `step()` method. | No | -| [MCP tools in OpenEnv environments](mcp-environment.md) | How to use the Model Context Protocol (MCP) tool interface across training, offline eval, and external serving. Covers `ListToolsAction`, `CallToolAction`, and the MCP lifecycle from reset to close. | No | -| [Evaluating agents with Inspect AI](evaluation-inspect.md) | Wrap an OpenEnv environment in an Inspect AI `Task` (dataset + solver + scorer), run it via `InspectAIHarness`, and get a structured `EvalResult` with accuracy scores. No training required — useful as a standalone eval pass on any checkpoint. | No | +| Tutorial | What it covers | GPU | Notebook | +|----------|---------------|-----|----------| +| [OpenEnv Tutorial](openenv-tutorial.md) | Full introduction to OpenEnv: install, connect to a hosted environment, step through an episode, define a reward function, and run a basic training loop. Start here if you're new. | No | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/OpenEnv_Tutorial.ipynb) | +| [Wordle GRPO](wordle-grpo.md) | Train an agent to play Wordle using GRPO via TRL's `environment_factory`. Shows the multi-turn tool-calling loop: the model guesses a word each turn and receives letter-position feedback until it wins or the episode ends. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://github.com/huggingface/trl/blob/main/examples/notebooks/openenv_wordle_grpo.ipynb) | +| [RL Training with 2048](rl-training-2048.md) | Train a language model to play the 2048 tile-sliding game with GRPO. Covers game-state representation and reward shaping for a puzzle environment. | Yes | — | +| [End-to-end walkthrough](end-to-end-walkthrough.md) | The full pipeline in one page: connect to `reasoning_gym`, wire it into TRL via `environment_factory`, fine-tune with GRPO on `chain_sum`, read the reward delta from training logs, and push the checkpoint to the Hub. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/end_to_end_walkthrough.ipynb) | +| [SFT warm-up for GRPO](sft-warmup.md) | When a base model's initial reward is near zero, use SFT to bootstrap format compliance before starting RL. Covers: collecting teacher rollouts with `CollectRunner`, filtering by reward, training with `SFTTrainer`, and measuring the before/after delta. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/sft_warmup.ipynb) | +| [Rubrics: composable reward computation](rubrics.md) | Build multi-criteria reward functions from reusable primitives (`WeightedSum`, `LLMJudge`, `Gate`). Shows how to compose, weight, and introspect rubrics inside an environment's `step()` method. | No | — | +| [MCP tools in OpenEnv environments](mcp-environment.md) | How to use the Model Context Protocol (MCP) tool interface across training, offline eval, and external serving. Covers `ListToolsAction`, `CallToolAction`, and the MCP lifecycle from reset to close. | No | — | +| [Evaluating agents with Inspect AI](evaluation-inspect.md) | Wrap an OpenEnv environment in an Inspect AI `Task` (dataset + solver + scorer), run it via `InspectAIHarness`, and get a structured `EvalResult` with accuracy scores. No training required — useful as a standalone eval pass on any checkpoint. | No | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/evaluation_inspect.ipynb) | ```{toctree} :maxdepth: 1 From acb8f7ca69a5d18a5bd352f322cf87f57ccf41b3 Mon Sep 17 00:00:00 2001 From: sergiopaniego Date: Tue, 5 May 2026 11:18:50 +0200 Subject: [PATCH 3/5] docs(tutorials): add Colab badge for rubrics tutorial Co-Authored-By: Claude Sonnet 4.6 --- docs/source/tutorials/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/tutorials/index.md b/docs/source/tutorials/index.md index d47b2fbe73..c196568633 100644 --- a/docs/source/tutorials/index.md +++ b/docs/source/tutorials/index.md @@ -7,7 +7,7 @@ | [RL Training with 2048](rl-training-2048.md) | Train a language model to play the 2048 tile-sliding game with GRPO. Covers game-state representation and reward shaping for a puzzle environment. | Yes | — | | [End-to-end walkthrough](end-to-end-walkthrough.md) | The full pipeline in one page: connect to `reasoning_gym`, wire it into TRL via `environment_factory`, fine-tune with GRPO on `chain_sum`, read the reward delta from training logs, and push the checkpoint to the Hub. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/end_to_end_walkthrough.ipynb) | | [SFT warm-up for GRPO](sft-warmup.md) | When a base model's initial reward is near zero, use SFT to bootstrap format compliance before starting RL. Covers: collecting teacher rollouts with `CollectRunner`, filtering by reward, training with `SFTTrainer`, and measuring the before/after delta. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/sft_warmup.ipynb) | -| [Rubrics: composable reward computation](rubrics.md) | Build multi-criteria reward functions from reusable primitives (`WeightedSum`, `LLMJudge`, `Gate`). Shows how to compose, weight, and introspect rubrics inside an environment's `step()` method. | No | — | +| [Rubrics: composable reward computation](rubrics.md) | Build multi-criteria reward functions from reusable primitives (`WeightedSum`, `LLMJudge`, `Gate`). Shows how to compose, weight, and introspect rubrics inside an environment's `step()` method. | No | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/rubrics.ipynb) | | [MCP tools in OpenEnv environments](mcp-environment.md) | How to use the Model Context Protocol (MCP) tool interface across training, offline eval, and external serving. Covers `ListToolsAction`, `CallToolAction`, and the MCP lifecycle from reset to close. | No | — | | [Evaluating agents with Inspect AI](evaluation-inspect.md) | Wrap an OpenEnv environment in an Inspect AI `Task` (dataset + solver + scorer), run it via `InspectAIHarness`, and get a structured `EvalResult` with accuracy scores. No training required — useful as a standalone eval pass on any checkpoint. | No | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/evaluation_inspect.ipynb) | From 85b680c32b2e7906df1fb751608d039ffb95f708 Mon Sep 17 00:00:00 2001 From: sergiopaniego Date: Tue, 5 May 2026 11:44:53 +0200 Subject: [PATCH 4/5] docs(tutorials): add Colab badge for MCP tutorial Co-Authored-By: Claude Sonnet 4.6 --- docs/source/tutorials/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/tutorials/index.md b/docs/source/tutorials/index.md index c196568633..4db2b8e21e 100644 --- a/docs/source/tutorials/index.md +++ b/docs/source/tutorials/index.md @@ -8,7 +8,7 @@ | [End-to-end walkthrough](end-to-end-walkthrough.md) | The full pipeline in one page: connect to `reasoning_gym`, wire it into TRL via `environment_factory`, fine-tune with GRPO on `chain_sum`, read the reward delta from training logs, and push the checkpoint to the Hub. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/end_to_end_walkthrough.ipynb) | | [SFT warm-up for GRPO](sft-warmup.md) | When a base model's initial reward is near zero, use SFT to bootstrap format compliance before starting RL. Covers: collecting teacher rollouts with `CollectRunner`, filtering by reward, training with `SFTTrainer`, and measuring the before/after delta. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/sft_warmup.ipynb) | | [Rubrics: composable reward computation](rubrics.md) | Build multi-criteria reward functions from reusable primitives (`WeightedSum`, `LLMJudge`, `Gate`). Shows how to compose, weight, and introspect rubrics inside an environment's `step()` method. | No | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/rubrics.ipynb) | -| [MCP tools in OpenEnv environments](mcp-environment.md) | How to use the Model Context Protocol (MCP) tool interface across training, offline eval, and external serving. Covers `ListToolsAction`, `CallToolAction`, and the MCP lifecycle from reset to close. | No | — | +| [MCP tools in OpenEnv environments](mcp-environment.md) | How to use the Model Context Protocol (MCP) tool interface across training, offline eval, and external serving. Covers `ListToolsAction`, `CallToolAction`, and the MCP lifecycle from reset to close. | No | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/mcp_environment.ipynb) | | [Evaluating agents with Inspect AI](evaluation-inspect.md) | Wrap an OpenEnv environment in an Inspect AI `Task` (dataset + solver + scorer), run it via `InspectAIHarness`, and get a structured `EvalResult` with accuracy scores. No training required — useful as a standalone eval pass on any checkpoint. | No | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/evaluation_inspect.ipynb) | ```{toctree} From 8f3913c34ac6081c4a3346ea7157ec35074ca3df Mon Sep 17 00:00:00 2001 From: sergiopaniego Date: Tue, 5 May 2026 11:55:02 +0200 Subject: [PATCH 5/5] fix(tutorials): fix Wordle GRPO Colab badge URL Co-Authored-By: Claude Sonnet 4.6 --- docs/source/tutorials/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/tutorials/index.md b/docs/source/tutorials/index.md index 4db2b8e21e..f7cee1b19d 100644 --- a/docs/source/tutorials/index.md +++ b/docs/source/tutorials/index.md @@ -3,7 +3,7 @@ | Tutorial | What it covers | GPU | Notebook | |----------|---------------|-----|----------| | [OpenEnv Tutorial](openenv-tutorial.md) | Full introduction to OpenEnv: install, connect to a hosted environment, step through an episode, define a reward function, and run a basic training loop. Start here if you're new. | No | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/OpenEnv_Tutorial.ipynb) | -| [Wordle GRPO](wordle-grpo.md) | Train an agent to play Wordle using GRPO via TRL's `environment_factory`. Shows the multi-turn tool-calling loop: the model guesses a word each turn and receives letter-position feedback until it wins or the episode ends. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://github.com/huggingface/trl/blob/main/examples/notebooks/openenv_wordle_grpo.ipynb) | +| [Wordle GRPO](wordle-grpo.md) | Train an agent to play Wordle using GRPO via TRL's `environment_factory`. Shows the multi-turn tool-calling loop: the model guesses a word each turn and receives letter-position feedback until it wins or the episode ends. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/trl/blob/main/examples/notebooks/openenv_wordle_grpo.ipynb) | | [RL Training with 2048](rl-training-2048.md) | Train a language model to play the 2048 tile-sliding game with GRPO. Covers game-state representation and reward shaping for a puzzle environment. | Yes | — | | [End-to-end walkthrough](end-to-end-walkthrough.md) | The full pipeline in one page: connect to `reasoning_gym`, wire it into TRL via `environment_factory`, fine-tune with GRPO on `chain_sum`, read the reward delta from training logs, and push the checkpoint to the Hub. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/end_to_end_walkthrough.ipynb) | | [SFT warm-up for GRPO](sft-warmup.md) | When a base model's initial reward is near zero, use SFT to bootstrap format compliance before starting RL. Covers: collecting teacher rollouts with `CollectRunner`, filtering by reward, training with `SFTTrainer`, and measuring the before/after delta. | Yes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/sft_warmup.ipynb) |