Docs: framework-neutral Training overview and harness pages

docs/source/_toctree.yml

@@ -11,8 +11,6 @@
     title: Auto-Discovery
   - local: guides/catalog-discovery
     title: Catalog Discovery
-  - local: guides/rl-integration
-    title: RL Training
   - local: guides/harbor-provider-qualification
     title: Harbor Spec
   - local: guides/rewards
@@ -51,6 +49,8 @@
       title: Your First Environment
     title: Basics
   - sections:
+    - local: guides/rl-integration
+      title: Overview
     - local: tutorials/end-to-end-walkthrough
       title: Train a Reasoning Model
     - local: tutorials/wordle-grpo
@@ -64,7 +64,7 @@
     - local: tutorials/harnesses
       title: Overview
     - local: tutorials/browsergym-harness
-      title: "White-box: Train with TRL's Loop (BrowserGym)"
+      title: "White-box: Train a Web Agent (BrowserGym)"
     - local: tutorials/harbor-harness
       title: "Black-box: Train Real Agents (Harbor)"
     - local: tutorials/claude-code-harness

docs/source/guides/async-sync.md

@@ -70,5 +70,5 @@ asyncio.run(main())
 
 ## Next Steps
 
-- [RL Framework Integration](rl-integration.md) - Using async with RL training
+- [Training with OpenEnv](rl-integration.md) - Using async with RL training
 - [Auto-Discovery](auto-discovery.md) - Discover and load environments

docs/source/guides/rewards.md

@@ -89,5 +89,5 @@ Per-turn progress helps the agent explore, but the terminal outcome still domina
 
 - [Rubrics tutorial](../tutorials/rubrics.md) — full API walkthrough with composable examples.
 - [RFC 004](https://github.com/huggingface/OpenEnv/blob/main/rfcs/004-rubrics.md) — design rationale.
-- [RL Framework Integration](rl-integration.md) — consume the reward signal in a training loop.
+- [Training with OpenEnv](rl-integration.md) — consume the reward signal in a training loop.
 - [Concepts](concepts.md) — where the rubric plugs in.

docs/source/guides/rl-integration.md

@@ -1,61 +1,50 @@
-# RL Framework Integration
+# Training with OpenEnv
 
-> [!NOTE]
-> This page is still being filled in. TRL integration is covered below; torchforge and SkyRL integrations are planned.
+OpenEnv is a contract between environments and the code that uses them, not a trainer. An environment exposes `reset()`, `step()` and `state()` (and its tools over MCP), runs as a service anywhere, and computes its own reward, so any training framework that can call it can train on it. This page maps the ways to train on OpenEnv environments and the frameworks that support it.
 
-Use OpenEnv with popular RL frameworks like TRL, torchforge, and SkyRL.
+## Ways to Train
 
-## Overview
+| You want to | What the environment gives | Worked examples |
+|---|---|---|
+| Train a model with RL, with the trainer running the episode | Observations, tools and a reward. The trainer generates every turn, so it has the tokens and logprobs. This is the white-box path | [Wordle](../tutorials/wordle-grpo), [a reasoning model](../tutorials/end-to-end-walkthrough), [web tasks in BrowserGym](../tutorials/browsergym-harness) with TRL, [2048](../tutorials/rl-training-2048) with Unsloth |
+| Train the model behind a real agent (OpenCode, Claude Code, Codex, …) | Harbor runs the agent on a task, captures every model call, and returns a `TrainingTrace` with token ids, logprobs and the verifier's reward. This is the black-box path | [Harbor](../tutorials/harbor-harness), trained with TRL's `AsyncGRPOTrainer` |
+| Warm-start a model with supervised data | `openenv collect` runs a teacher in the environment and saves reward-labeled rollouts as a dataset | [Collecting rollouts for SFT](../tutorials/sft-warmup) |
+| Evaluate, not train | Scores from the environment's rubric | [Inspect AI](../tutorials/evaluation-inspect), [an agent inside the environment](../tutorials/claude-code-harness) |
 
-OpenEnv environments are designed to integrate seamlessly with RL training frameworks. The standard `step()`, `reset()`, `state()` API makes it easy to use environments in training loops.
+None of these paths ties the environment to a framework. `openenv.core` does not depend on any trainer, and a `TrainingTrace` or a collected dataset is plain data that any trainer can read. The worked examples use the frameworks named above because that is where the examples exist today. [Harnesses in OpenEnv](../tutorials/harnesses) compares the white-box and black-box paths in more detail.
 
-## TRL Integration
+## Integrations
 
-[TRL (Transformer Reinforcement Learning)](https://huggingface.co/docs/trl) is the recommended framework for training language models with RL.
+These frameworks and platforms train on OpenEnv environments. If your project supports OpenEnv, open a PR to add it here and to the [README](https://github.com/huggingface/OpenEnv#integrations).
 
-```python
-from trl import GRPOTrainer
-from openenv import AutoEnv, AutoAction
-
-env = AutoEnv.from_env("textarena")
-TextAction = AutoAction.from_env("textarena")
-
-# Use with TRL's GRPO trainer
-trainer = GRPOTrainer(
-    model=model,
-    reward_model=reward_model,
-    # ... TRL config
-)
-```
+| Framework | Example |
+|---|---|
+| [TRL](https://huggingface.co/docs/trl) | [OpenEnv integration guide](https://huggingface.co/docs/trl/openenv): `environment_factory` with `GRPOTrainer`, several environments at once, and harness training through Harbor |
+| [torchforge](https://meta-pytorch.org/torchforge/) | [GRPO on BlackJack](https://github.com/huggingface/OpenEnv/tree/main/examples/grpo_blackjack) |
+| [Unsloth](https://unsloth.ai) | [2048 with gpt-oss](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/OpenEnv_gpt_oss_(20B)_Reinforcement_Learning_2048_Game.ipynb) |
+| [SkyRL](https://github.com/NovaSky-AI/SkyRL) | [Training on OpenEnv environments](https://skyrl.readthedocs.io/en/latest/examples/openenv.html) |
+| [ART](https://art.openpipe.ai) | [OpenEnv integration](https://art.openpipe.ai/integrations/openenv-integration) |
+| [Oumi](https://github.com/oumi-ai/oumi) | [OpenEnv GRPO notebook](https://github.com/oumi-ai/oumi/blob/main/notebooks/Oumi%20-%20OpenEnv%20GRPO%20with%20trl.ipynb) |
+| [Lightning AI](https://lightning.ai) | [OpenEnv templates](https://lightning.ai/templates?section=featured&query=openenv) |
+| [Miles](https://github.com/radixark/miles) | [GRPO on Terminal-Bench 2](https://github.com/radixark/miles/tree/main/examples/experimental/openenv) |
 
-See the [Wordle with GRPO](../tutorials/wordle-grpo.md) tutorial for a complete example.
+## Your Own Training Loop
 
-## Generic Training Loop
-
-For custom training setups:
+Every integration above comes down to the same calls. To plug OpenEnv into a framework that has no integration yet, drive the client from its rollout code:
 
 ```python
-from openenv import AutoEnv, AutoAction
+from openenv import AutoAction, AutoEnv
 
 env = AutoEnv.from_env("my-env")
 Action = AutoAction.from_env("my-env")
 
 with env.sync() as client:
     for episode in range(num_episodes):
         result = client.reset()
-
-        while not result.terminated:
-            # Get action from your policy
-            action = policy(result.observation)
-
-            # Take step
+        while not result.done:
+            action = policy(result.observation)  # your model picks the next action
             result = client.step(action)
-
-            # Update policy with reward
-            policy.update(result.reward)
+            # result.reward is the environment's reward for this step
 ```
 
-## Next Steps
-
-- [Reward Design](rewards.md) - Design effective reward functions
-- [Wordle with GRPO](../tutorials/wordle-grpo.md) - Complete TRL example
+The [Task API](task-api) lets a trainer list an environment's tasks and pick which one each episode runs, and [Rewards](rewards) covers how environments compute the reward.

docs/source/guides/task-api.md

@@ -272,6 +272,6 @@ Before shipping a dataset-backed environment:
 ## Related Reading
 
 - [Concepts](concepts.md)
-- [RL Training](rl-integration.md)
+- [Training with OpenEnv](rl-integration.md)
 - [Core API](../reference/core.md)
 - [CLI reference](../reference/cli.md)

docs/source/tutorials/browsergym-harness.md

@@ -1,6 +1,6 @@
 # White-Box: Train a Web Agent on BrowserGym
 
-This tutorial trains a model to complete web tasks in [BrowserGym](../environments/browsergym) with TRL's `GRPOTrainer`. It is the **white-box** harness path: TRL owns the agent loop. It samples every turn, parses the tool calls, runs them against the environment and feeds the results back, so it has the token ids and logprobs of everything the model generated without capturing anything. [Harnesses in OpenEnv](harnesses) compares it to the other paths.
+This tutorial trains a model to complete web tasks in [BrowserGym](../environments/browsergym) with TRL's `GRPOTrainer`. It is the **white-box** harness path: the trainer, here TRL, owns the agent loop. It samples every turn, parses the tool calls, runs them against the environment and feeds the results back, so it has the token ids and logprobs of everything the model generated without capturing anything. [Harnesses in OpenEnv](harnesses) compares it to the other paths, and [Training with OpenEnv](../guides/rl-integration) lists the other frameworks that train on OpenEnv environments.
 
 The full script is TRL's [`examples/grpo_browsergym/grpo_browsergym.py`](https://github.com/huggingface/trl/tree/main/examples/grpo_browsergym). This page walks through its parts.
 

docs/source/tutorials/harbor-harness.md

@@ -1,6 +1,6 @@
 # Black-Box: Train Real Agents with Harbor
 
-Some agents can't be reimplemented as a TRL tool loop. A coding agent such as OpenCode, Claude Code or Codex has its own planner, tools, context management and stop condition, and the point is to train the model that drives *that* agent. This is the **black-box** harness path: the agent owns its loop, and OpenEnv records every model call it makes. Episodes are tasks: an instruction goes in, the agent works on its own, and a verifier checks the result. If the job is a conversation instead, or the agent has to use your environment's own tools, see [Evaluate Claude Code in an Environment](claude-code-harness). [Harnesses in OpenEnv](harnesses) compares the paths.
+Some agents can't be reimplemented as a trainer's tool loop. A coding agent such as OpenCode, Claude Code or Codex has its own planner, tools, context management and stop condition, and the point is to train the model that drives *that* agent. This is the **black-box** harness path: the agent owns its loop, and OpenEnv records every model call it makes. Episodes are tasks: an instruction goes in, the agent works on its own, and a verifier checks the result. If the job is a conversation instead, or the agent has to use your environment's own tools, see [Evaluate Claude Code in an Environment](claude-code-harness). [Harnesses in OpenEnv](harnesses) compares the paths.
 
 OpenEnv does this through [Harbor](../environments/harbor), which supplies the tasks, the sandboxes, the agents and the verifiers. One `harbor_env` server runs any of 16 validated agents on any Harbor dataset, in any of Harbor's sandbox backends, and its capture proxy returns the token ids and logprobs of each model call together with the task's reward.
 
@@ -20,9 +20,9 @@ openenv harbor rollout \
 
 Against a hosted provider you get an evaluation rollout: the reward and the full trace. Against vLLM or SGLang you also get the token ids and logprobs that training needs. `openenv harbor info` checks first which agents, sandboxes and capture level this machine can use.
 
-## Train With TRL
+## Train on the Captures
 
-TRL trains on these rollouts with `AsyncGRPOTrainer` and a `HarnessRolloutWorker`: the agent runs each task to completion, TRL reads back the captured `TrainingTrace`, and the new weights go back into the same vLLM server.
+Each rollout comes back as a `TrainingTrace`: the token ids, logprobs and loss masks of every model call, plus the task's reward. It doesn't depend on a trainer, so any framework can train on it. TRL has a worked example: `AsyncGRPOTrainer` with a `HarnessRolloutWorker` runs each task to completion, reads back the trace, and syncs the new weights into the same vLLM server.
 
 - [TRL's guide to training on harnesses](https://huggingface.co/docs/trl/main/en/openenv#training-on-harnesses-training-real-coding-agents-harbor) explains how it works and how to wire it.
 - [`examples/async_grpo_harbor`](https://github.com/huggingface/trl/tree/main/examples/async_grpo_harbor) is the complete script, with a Hugging Face Jobs launcher.
@@ -32,3 +32,4 @@ TRL trains on these rollouts with `AsyncGRPOTrainer` and a `HarnessRolloutWorker
 
 - [The Harbor environment](../environments/harbor): supported agents and sandboxes, the web UI, deploying to Spaces and troubleshooting.
 - [TRL's Harbor integration](https://huggingface.co/docs/trl/harbor) for the white-box path on Harbor tasks, where TRL's own loop works through a Harbor task suite.
+- [Training with OpenEnv](../guides/rl-integration) for the other ways to train and the frameworks that support them.

docs/source/tutorials/harnesses.md

@@ -4,18 +4,18 @@ A harness is the loop around a model: it sends the conversation to the model, ru
 
 | | White-box | Black-box | Agent inside the environment |
 |---|---|---|---|
-| Who owns the loop | TRL (`environment_factory`) | A real agent (OpenCode, Claude Code, Codex, …) | A real agent (Claude Code in the tutorial) |
+| Who owns the loop | The trainer (for example TRL's `environment_factory`) | A real agent (OpenCode, Claude Code, Codex, …) | A real agent (Claude Code in the tutorial) |
 | What the environment provides | Tools and a reward for the model being trained | Tasks: an instruction, a sandbox, and a verifier | Its own tools and state, injected into the agent over MCP |
 | Who is on the other side | Nobody, the model works through the task | Nobody, the agent runs to completion | A user, simulated or real, over several turns |
-| Trains a model | Yes | Yes, from captured token ids and logprobs (against vLLM or SGLang) | No, nothing is captured |
+| Trains a model | Yes | Yes, from the captured token ids and logprobs in a `TrainingTrace` (against vLLM or SGLang) | No, nothing is captured |
 | Serves real users | No | No | Yes, over `WS /harness` |
 | Example | Web tasks in BrowserGym | "Fix this repository", "analyze these files" | Customer service on τ²-bench |
 | Start here | [BrowserGym](browsergym-harness) | [Harbor](harbor-harness) | [Claude Code on τ²-bench](claude-code-harness) |
 
 ## Which One to Use
 
-- **You want to train a model on your environment and the loop can be simple.** Use the white-box path. TRL's `GRPOTrainer` runs the tool loop, so token ids and logprobs come for free, and the environment only supplies tools and a reward. [Wordle](wordle-grpo) is the same pattern on a text game.
-- **The job is a task that an agent does on its own, and a verifier checks the result.** Use Harbor, to train the model behind the agent or to evaluate it. The agent keeps its own planner, tools and context management in a sandbox, and a proxy records every model call it makes.
+- **You want to train a model on your environment and the loop can be simple.** Use the white-box path. The trainer runs the tool loop, so token ids and logprobs come for free, and the environment only supplies tools and a reward. The BrowserGym tutorial does it with TRL's `GRPOTrainer`, and [Training with OpenEnv](../guides/rl-integration) lists the other frameworks that train on OpenEnv environments.
+- **The job is a task that an agent does on its own, and a verifier checks the result.** Use Harbor, to train the model behind the agent or to evaluate it. The agent keeps its own planner, tools and context management in a sandbox, and a proxy records every model call it makes. The resulting `TrainingTrace` is plain data that doesn't depend on a trainer, and TRL's `AsyncGRPOTrainer` has a worked example.
 - **The job is a conversation, and the agent has to use your environment's tools.** Use `HarnessEnvironment` from [RFC 005](https://github.com/huggingface/OpenEnv/blob/main/rfcs/005-agentic-harnesses.md), to evaluate the agent against a simulated user or to serve it to real ones. The environment injects its tools, answers between turns and scores the conversation with a rubric.
 
 ### Harbor or `HarnessEnvironment`?

docs/source/tutorials/index.md

@@ -3,8 +3,8 @@
 Choose a learning path from the sidebar:
 
 - **Basics:** start with [Hello World](openenv-tutorial) and build [Your First Environment](../guides/first-environment).
-- **Training:** train a [reasoning model](end-to-end-walkthrough), use [TRL](wordle-grpo) or [Unsloth](rl-training-2048), or collect rollouts for [SFT](sft-warmup).
-- **Harnesses:** [pick a path](harnesses): train with TRL's own loop on [BrowserGym](browsergym-harness) (white-box), train real agents through [Harbor](harbor-harness) (black-box), or [evaluate Claude Code inside an environment](claude-code-harness). The [OpenCode](opencode-agent-grpo) and [Pi](pi-agent-grpo) tutorials are deprecated.
+- **Training:** [the ways to train and the supported frameworks](../guides/rl-integration), then train a [reasoning model](end-to-end-walkthrough), use [TRL](wordle-grpo) or [Unsloth](rl-training-2048), or collect rollouts for [SFT](sft-warmup).
+- **Harnesses:** [pick a path](harnesses): train with the trainer's own loop on [BrowserGym](browsergym-harness) (white-box), train real agents through [Harbor](harbor-harness) (black-box), or [evaluate Claude Code inside an environment](claude-code-harness). The [OpenCode](opencode-agent-grpo) and [Pi](pi-agent-grpo) tutorials are deprecated.
 - **Evals:** follow [Evaluating with Environments](evaluation-inspect).
 
 For [MCP environments](mcp-environment) and [rubrics](rubrics), see Concepts in the sidebar.

docs/source/tutorials/rubrics.md

@@ -379,4 +379,4 @@ The same rubric object used to compute training rewards doubles as the eval metr
 - **Real-world trajectory example** — walk through `envs/chess_env/server/rubrics.py` and `chess_environment.py` to see `ExponentialDiscountingTrajectoryRubric` wired into a game environment.
 - **Design details** — [RFC 004](https://github.com/huggingface/OpenEnv/blob/main/rfcs/004-rubrics.md) covers the rationale for the composable API and the "rewards inside the environment" invariant.
 - **Reward design basics** — the [Reward Design](../guides/rewards.md) guide covers sparse-vs-dense signals and common pitfalls that still apply on top of any rubric composition.
-- **Training loop integration** — see the [RL Framework Integration](../guides/rl-integration.md) guide and the [TRL OpenEnv integration guide](https://huggingface.co/docs/trl/main/en/openenv) for the recommended `environment_factory` pattern.
+- **Training loop integration** — see [Training with OpenEnv](../guides/rl-integration.md) and the [TRL OpenEnv integration guide](https://huggingface.co/docs/trl/main/en/openenv) for the recommended `environment_factory` pattern.