diff --git a/docs/docs.json b/docs/docs.json index 553af7b1f80..1a45487479e 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -173,6 +173,14 @@ "user-guide/agentic-chat-template", "user-guide/cli-reference" ] + }, + { + "group": "Environments", + "root": "user-guide/environments", + "pages": [ + "user-guide/harbor", + "user-guide/openenv" + ] } ] }, diff --git a/docs/user-guide/environments.md b/docs/user-guide/environments.md new file mode 100644 index 00000000000..15bf96c5f25 --- /dev/null +++ b/docs/user-guide/environments.md @@ -0,0 +1,59 @@ +--- +title: Environments +description: How Miles trains on RL environments — datasets with rewards, self-wired environments, and optional external ecosystems. +--- + +Miles owns the training side of environment RL — batch orchestration, lossless +token-in/token-out recording, reward hooks, filtering — and is agnostic about +where the environment itself comes from: + +- **No environment** — single-turn RLVR: a prompt dataset scored by the + built-in rule-based rewards (math, ifbench, ...) or a custom reward function. +- **Your own environment** — plug your code into one of the three rollout + layers described in [Integration shapes](#integration-shapes); most + environments sit in the agent function, with the session server recording + tokens (see [Rollout Endpoints](/user-guide/rollout-endpoints)). +- **An external ecosystem** — adopt a prebuilt connector from the table below; + connectors occupy the same three layers. + +| Integration | Plugs in at | +|---|---| +| [Harbor](/user-guide/harbor) | agent function | +| [OpenEnv](/user-guide/openenv) | agent function | +| [Strands Agents](https://github.com/radixark/miles/tree/main/examples/strands_sglang) | generate function | +| [τ-bench](https://github.com/radixark/miles/tree/main/examples/tau-bench) | generate function | + +Sandbox providers are a different axis: they provision the task containers +*inside* a connector rather than occupying a rollout layer. + +| Sandbox provider | Used within | +|---|---| +| [Daytona](https://www.daytona.io/) | OpenEnv, Harbor | + +All external ecosystem support is experimental. + +## Integration shapes + +The rollout stack is three nested plug-in layers (see +[Customization](/user-guide/customization)): each column in the table below +wraps the one to its left, so replacing an outer layer also takes over +everything an inner one would. A connector replaces exactly one layer. + +✓ = the external framework takes it over; ○ = stays in Miles. + +| | Agent function (innermost) | Generate function | Rollout function (outermost) | +|---|:---:|:---:|:---:| +| Plug-point flag | `--custom-agent-function-path` | `--custom-generate-function-path` | `--rollout-function-path` | +| Agent–environment loop | ✓ | ✓ | ✓ | +| Trajectory & token recording | ○ | ✓¹ | ✓¹ | +| Reward pathway (RM hooks, group rewards) | ○² | ○² | ✓ | +| Data source (prompts / taskset) | ○ | ○ | ✓ | +| Batch orchestration (grouping, filtering) | ○ | ○ | ✓ | +| Model, engines & weight updates, advantages, optimizer | ○ | ○ | ○ | + +¹ Typically by speaking SGLang's native `/generate` (token IDs in and out) +rather than the session-server chat endpoint Miles' own recording uses. + +² The environment may grade an episode itself (Harbor and τ-bench do); the +score still enters training through Miles' `Sample.reward` / RM hooks, and +group-level reward handling stays in Miles. diff --git a/docs/user-guide/harbor.md b/docs/user-guide/harbor.md new file mode 100644 index 00000000000..36a6cffd95a --- /dev/null +++ b/docs/user-guide/harbor.md @@ -0,0 +1,25 @@ +--- +title: Harbor +description: Train agents on mixed task suites (SWE-bench, Terminal-Bench, custom) through the Harbor framework. +--- + +[Harbor](https://github.com/harbor-framework/harbor) is an agent-environment +framework from the Laude Institute: agent orchestration and grading are unified +in a single `Trial.run()` call, and a task is fully described by four files +(`instruction.md`, `Dockerfile`, `test.sh`, `task.toml`), so mixed task suites — +SWE-bench, Terminal-Bench, custom tasks — train through one endpoint. + +Miles integrates Harbor as an +[agent-function integration](/user-guide/environments): the agent function +hands each session's OpenAI-compatible URL to a Harbor server, which runs the +per-task container, installs and runs the agent against that URL, and grades +the result; the grade becomes the sample's reward through a custom reward +hook. + +## Try it + +The maintained recipe lives in +[`examples/experimental/swe-agent-v2`](https://github.com/radixark/miles/tree/main/examples/experimental/swe-agent-v2), +with synchronous and fully-async launchers. Follow the +[recipe README](https://github.com/radixark/miles/blob/main/examples/experimental/swe-agent-v2/README.md) +for the architecture, Harbor server setup, task format, and launch scripts. diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index 033b0c306ea..ff45507e646 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -14,6 +14,7 @@ description: Concepts, launch script walkthrough, customization hooks, and a com | [Fully Async Rollout](/user-guide/fully-async) | Queue-backed rollout production, tuning knobs, and when to use `train_async.py`. | | [Agentic Chat Templates](/user-guide/agentic-chat-template) | Turning on and verifying TITO so multi-turn agentic rollout stays append-only. | | [CLI Reference](/user-guide/cli-reference) | Every flag Miles accepts, grouped by subsystem. | +| [Environments](/user-guide/environments) | Supplying an environment: dataset + reward, your own env via the plug points, or an external ecosystem. | ## Which pages do I actually need? diff --git a/docs/user-guide/openenv.md b/docs/user-guide/openenv.md new file mode 100644 index 00000000000..0d7c7aec80f --- /dev/null +++ b/docs/user-guide/openenv.md @@ -0,0 +1,27 @@ +--- +title: OpenEnv +description: Train on Hugging Face OpenEnv environments through the agent-function extension point. +--- + +[OpenEnv](https://github.com/huggingface/openenv) is Hugging Face's open +protocol for RL environments: an environment is an HTTP service exposing +`reset` / `step` (and optionally `evaluate`), so any environment speaking the +protocol can serve any trainer. + +Miles integrates OpenEnv as an +[agent-function integration](/user-guide/environments): a Miles-side agent +function drives the agentic loop — `reset(task_id)`, repeated `step`s, then +scoring the episode with the task's own tests — against an unmodified OpenEnv +server, and the score becomes the sample's reward through a custom reward +hook. + +## Try it + +The maintained end-to-end recipe is **Terminal-Bench-2 GRPO** in +[`examples/experimental/openenv`](https://github.com/radixark/miles/tree/main/examples/experimental/openenv). +It runs against a shared Docker env server (full per-task image fidelity) or +per-episode [Daytona](https://www.daytona.io/) cloud sandboxes built from each +task's official image (no resident infrastructure). Follow the +[recipe README](https://github.com/radixark/miles/blob/main/examples/experimental/openenv/README.md) +for prompt-data preparation, env-server modes, launcher flags, and operational +notes.