Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,14 @@
"user-guide/agentic-chat-template",
"user-guide/cli-reference"
]
},
{
"group": "Environments",
"root": "user-guide/environments",
"pages": [
"user-guide/harbor",
"user-guide/openenv"
]
}
]
},
Expand Down
59 changes: 59 additions & 0 deletions docs/user-guide/environments.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
title: Environments
description: How Miles trains on RL environments — datasets with rewards, self-wired environments, and optional external ecosystems.
---

Miles owns the training side of environment RL — batch orchestration, lossless
token-in/token-out recording, reward hooks, filtering — and is agnostic about
where the environment itself comes from:

- **No environment** — single-turn RLVR: a prompt dataset scored by the
built-in rule-based rewards (math, ifbench, ...) or a custom reward function.
- **Your own environment** — plug your code into one of the three rollout
layers described in [Integration shapes](#integration-shapes); most
environments sit in the agent function, with the session server recording
tokens (see [Rollout Endpoints](/user-guide/rollout-endpoints)).
- **An external ecosystem** — adopt a prebuilt connector from the table below;
connectors occupy the same three layers.

| Integration | Plugs in at |
|---|---|
| [Harbor](/user-guide/harbor) | agent function |
| [OpenEnv](/user-guide/openenv) | agent function |
| [Strands Agents](https://github.com/radixark/miles/tree/main/examples/strands_sglang) | generate function |
| [τ-bench](https://github.com/radixark/miles/tree/main/examples/tau-bench) | generate function |

Sandbox providers are a different axis: they provision the task containers
*inside* a connector rather than occupying a rollout layer.

| Sandbox provider | Used within |
|---|---|
| [Daytona](https://www.daytona.io/) | OpenEnv, Harbor |
Comment on lines +29 to +31

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would replace this table with text

Users can choose their own sandbox provide, for example, Daytona, ...

Having a table here with a single row "Daytona" might give people the false impression that we have a special relationship with them lol.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a placeholder. I'll very soon add more items here like agentENV and modal


All external ecosystem support is experimental.

## Integration shapes

The rollout stack is three nested plug-in layers (see
[Customization](/user-guide/customization)): each column in the table below
wraps the one to its left, so replacing an outer layer also takes over
everything an inner one would. A connector replaces exactly one layer.

✓ = the external framework takes it over; ○ = stays in Miles.

| | Agent function (innermost) | Generate function | Rollout function (outermost) |
|---|:---:|:---:|:---:|
| Plug-point flag | `--custom-agent-function-path` | `--custom-generate-function-path` | `--rollout-function-path` |
| Agent–environment loop | ✓ | ✓ | ✓ |
| Trajectory & token recording | ○ | ✓¹ | ✓¹ |
| Reward pathway (RM hooks, group rewards) | ○² | ○² | ✓ |
| Data source (prompts / taskset) | ○ | ○ | ✓ |
| Batch orchestration (grouping, filtering) | ○ | ○ | ✓ |
| Model, engines & weight updates, advantages, optimizer | ○ | ○ | ○ |

¹ Typically by speaking SGLang's native `/generate` (token IDs in and out)
rather than the session-server chat endpoint Miles' own recording uses.

² The environment may grade an episode itself (Harbor and τ-bench do); the
score still enters training through Miles' `Sample.reward` / RM hooks, and
group-level reward handling stays in Miles.
25 changes: 25 additions & 0 deletions docs/user-guide/harbor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
title: Harbor
description: Train agents on mixed task suites (SWE-bench, Terminal-Bench, custom) through the Harbor framework.
---

[Harbor](https://github.com/harbor-framework/harbor) is an agent-environment
framework from the Laude Institute: agent orchestration and grading are unified
in a single `Trial.run()` call, and a task is fully described by four files
(`instruction.md`, `Dockerfile`, `test.sh`, `task.toml`), so mixed task suites —
SWE-bench, Terminal-Bench, custom tasks — train through one endpoint.

Miles integrates Harbor as an
[agent-function integration](/user-guide/environments): the agent function
hands each session's OpenAI-compatible URL to a Harbor server, which runs the
per-task container, installs and runs the agent against that URL, and grades
the result; the grade becomes the sample's reward through a custom reward
hook.

## Try it

The maintained recipe lives in
[`examples/experimental/swe-agent-v2`](https://github.com/radixark/miles/tree/main/examples/experimental/swe-agent-v2),
with synchronous and fully-async launchers. Follow the
[recipe README](https://github.com/radixark/miles/blob/main/examples/experimental/swe-agent-v2/README.md)
for the architecture, Harbor server setup, task format, and launch scripts.
Comment on lines +21 to +25

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example is stale and I have a new one introduced in PR 1741. But let's merge your PR first, and I'll update this section in my PR.

1 change: 1 addition & 0 deletions docs/user-guide/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ description: Concepts, launch script walkthrough, customization hooks, and a com
| [Fully Async Rollout](/user-guide/fully-async) | Queue-backed rollout production, tuning knobs, and when to use `train_async.py`. |
| [Agentic Chat Templates](/user-guide/agentic-chat-template) | Turning on and verifying TITO so multi-turn agentic rollout stays append-only. |
| [CLI Reference](/user-guide/cli-reference) | Every flag Miles accepts, grouped by subsystem. |
| [Environments](/user-guide/environments) | Supplying an environment: dataset + reward, your own env via the plug points, or an external ecosystem. |

## Which pages do I actually need?

Expand Down
27 changes: 27 additions & 0 deletions docs/user-guide/openenv.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
title: OpenEnv
description: Train on Hugging Face OpenEnv environments through the agent-function extension point.
---

[OpenEnv](https://github.com/huggingface/openenv) is Hugging Face's open
protocol for RL environments: an environment is an HTTP service exposing
`reset` / `step` (and optionally `evaluate`), so any environment speaking the
protocol can serve any trainer.

Miles integrates OpenEnv as an
[agent-function integration](/user-guide/environments): a Miles-side agent
function drives the agentic loop — `reset(task_id)`, repeated `step`s, then
scoring the episode with the task's own tests — against an unmodified OpenEnv
server, and the score becomes the sample's reward through a custom reward
hook.

## Try it

The maintained end-to-end recipe is **Terminal-Bench-2 GRPO** in
[`examples/experimental/openenv`](https://github.com/radixark/miles/tree/main/examples/experimental/openenv).
It runs against a shared Docker env server (full per-task image fidelity) or
per-episode [Daytona](https://www.daytona.io/) cloud sandboxes built from each
task's official image (no resident infrastructure). Follow the
[recipe README](https://github.com/radixark/miles/blob/main/examples/experimental/openenv/README.md)
for prompt-data preparation, env-server modes, launcher flags, and operational
notes.
Loading