Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/en/get_started/agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Most agentic RL tasks should start with `--custom-generate-function-path`. This

The agent workflow itself may speak in strings, chat messages, tool calls, environment observations, or framework-specific events. The training target, however, should stay token based. Preserve the model-sampled token ids and use `loss_mask` to separate trainable model output from prompt, template, tool-observation, or environment text.

If one prompt rollout corresponds to one training sample, return a single `Sample`. If one rollout splits into multiple trainable segments, such as subagent trajectories, main-agent continuations, or pre/post-compaction segments, return `list[Sample]` and set the same `group_id` on all sibling samples. slime then keeps those samples together for train-step splitting and loss aggregation instead of counting them as independent groups. `Sample.rollout_id` remains as a deprecated write-only alias for older code that only assigns it.
If one prompt rollout corresponds to one training sample, return a single `Sample`. If one rollout splits into multiple trainable segments, such as subagent trajectories, main-agent continuations, or pre/post-compaction segments, return `list[Sample]` and set the same `rollout_id` on all sibling samples. slime then keeps those samples together for train-step splitting and loss aggregation instead of counting them as independent rollouts.

Reach for `--rollout-function-path` only when you need to replace the whole rollout orchestration. Common reasons include custom data-source scheduling, cross-rollout background queues, fully asynchronous generation, or workflows that cannot fit the default `sglang_rollout` prompt-by-sample structure.

Expand Down Expand Up @@ -68,6 +68,6 @@ Agentic rollouts tend to depend more heavily on serving configuration than ordin

The full coding-agent example is [`examples/coding_agent_rl`](../_examples_synced/coding_agent_rl/README.md). It shows an end-to-end agent RL setup that is close to a real software-engineering workflow: each sample boots an isolated sandbox, the agent uses tools to edit code, the rollout captures a `git diff`, and a clean sandbox runs the tests to produce the reward.

This example also demonstrates agent fan-out training. Its middleware splits one trajectory into `subagent`, `wipe` (the chain frozen before compaction), and `final` segments. `generate()` returns `list[Sample]`, and all segments share the same `group_id`.
This example also demonstrates agent fan-out training. Its middleware splits one trajectory into `subagent`, `wipe` (the chain frozen before compaction), and `final` segments. `generate()` returns `list[Sample]`, and all segments share the same `rollout_id`.

For smaller starting points, see [`examples/search-r1`](../_examples_synced/search-r1/README.md) for multi-turn tool use, [`examples/retool`](../_examples_synced/retool/README.md) for tool-augmented generation, and [`examples/multi_agent`](../_examples_synced/multi_agent/README.md) for the multi-agent pattern.
6 changes: 3 additions & 3 deletions docs/en/get_started/customization.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ async def custom_generate(args, sample: Sample, sampling_params: dict) -> Sample

In agentic settings such as subagents, multi-agent execution, or context compaction, one prompt rollout can naturally split into multiple trainable segments. For example, a subagent trajectory and the main-agent continuation may both need to be trained, or the context before and after compaction may be represented as separate segments.

You do not need to replace the whole rollout function for this. A `custom_generate` function may return `list[Sample]`. The key contract is that sibling samples produced by the same rollout must share the same `group_id`, so slime keeps them together for train-step splitting and loss aggregation instead of counting them as independent groups. `Sample.rollout_id` remains as a deprecated write-only alias for older code that only assigns it.
You do not need to replace the whole rollout function for this. A `custom_generate` function may return `list[Sample]`. The key contract is that sibling samples produced by the same rollout must share the same `rollout_id`, so slime keeps them together for train-step splitting and loss aggregation instead of counting them as independent rollouts.

```python
import copy
Expand All @@ -98,7 +98,7 @@ from slime.utils.types import Sample

async def custom_generate(args, sample: Sample, sampling_params: dict) -> list[Sample]:
segments = await run_agent_and_split_segments(args, sample, sampling_params)
group_id = sample.group_id if sample.group_id is not None else sample.index
rollout_id = sample.rollout_id if sample.rollout_id is not None else sample.index

samples: list[Sample] = []
for segment in segments:
Expand All @@ -109,7 +109,7 @@ async def custom_generate(args, sample: Sample, sampling_params: dict) -> list[S
s.loss_mask = segment.loss_mask
s.reward = segment.reward
s.status = Sample.Status.COMPLETED
s.group_id = group_id
s.rollout_id = rollout_id
samples.append(s)
return samples
```
Expand Down
4 changes: 2 additions & 2 deletions docs/zh/get_started/agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ slime 的核心定位并不只是跑单轮 RL,而是把高性能训练、SGLan

agent workflow 本身可以使用字符串、chat messages、tool calls、环境 observation 或框架自己的事件格式。但训练目标仍然应该是 token based:尽量保留模型实际采样得到的 token ids,并用 `loss_mask` 区分可训练的模型输出和 prompt、template、tool observation、环境文本。

如果一次 prompt rollout 只对应一个训练样本,返回一个 `Sample` 即可。如果一次 rollout 会拆成多个训练片段,例如 subagent 轨迹、main-agent 轨迹、compact 前后的片段,则返回 `list[Sample]`,并给这些 sibling samples 设置相同的 `group_id`。这样 slime 会在训练 step 切分和 loss 聚合时把它们视作同一个训练分组,而不是重复计数。`Sample.rollout_id` 仍然作为 deprecated write-only alias 保留给只赋值它的旧代码
如果一次 prompt rollout 只对应一个训练样本,返回一个 `Sample` 即可。如果一次 rollout 会拆成多个训练片段,例如 subagent 轨迹、main-agent 轨迹、compact 前后的片段,则返回 `list[Sample]`,并给这些 sibling samples 设置相同的 `rollout_id`。这样 slime 会在训练 step 切分和 loss 聚合时把它们视作同一次 rollout,而不是重复计数。

只有当你需要替换整个 rollout 编排时,才优先考虑 `--rollout-function-path`。典型场景包括:自定义数据源调度、跨 rollout 的后台队列、完全异步生成,或者默认 `sglang_rollout` 的 prompt × sample 结构已经无法表达你的 workflow。

Expand Down Expand Up @@ -68,6 +68,6 @@ agentic rollout 往往比普通单轮 generation 更依赖 serving 配置:上

完整的 coding-agent 样例见 [`examples/coding_agent_rl`](../_examples_synced/coding_agent_rl/README.md)。它展示了一个比较接近真实 agent RL 的端到端形态:每条 sample 启动独立 sandbox,agent 使用工具修改代码,生成 `git diff`,再在干净 sandbox 里跑测试得到 reward。

这个样例也演示了 agent fan-out 的训练方式:middleware 会把 trajectory 切成 `subagent`、`wipe`(compact 前被冻结的链)和 `final` 等片段,`generate()` 返回 `list[Sample]`,并让这些片段共享同一个 `group_id`。
这个样例也演示了 agent fan-out 的训练方式:middleware 会把 trajectory 切成 `subagent`、`wipe`(compact 前被冻结的链)和 `final` 等片段,`generate()` 返回 `list[Sample]`,并让这些片段共享同一个 `rollout_id`。

如果你只需要更轻量的入门例子,可以先看 [`examples/search-r1`](../_examples_synced/search-r1/README.md) 的多轮工具调用、[`examples/retool`](../_examples_synced/retool/README.md) 的工具增强生成、以及 [`examples/multi_agent`](../_examples_synced/multi_agent/README.md) 的多 agent 模式。
6 changes: 3 additions & 3 deletions docs/zh/get_started/customization.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ async def custom_generate(args, sample: Sample, sampling_params: dict) -> Sample

在 subagent、multi-agent、context compact 等 agentic 场景中,一次 prompt rollout 可能会自然拆成多个可训练片段。例如:主 agent 调用 subagent 后,subagent 的轨迹和主 agent 的后续轨迹都需要参与训练;或者发生 compact 后,compact 前后的上下文被切成多个 segment。

这种情况下不需要重写整个 rollout 函数,`custom_generate` 可以直接返回 `list[Sample]`。关键是:这些由同一次 rollout 拆出来的 sibling samples 必须设置相同的 `group_id`,这样 slime 会在训练切分和 loss 聚合时把它们视作同一个训练分组,而不是重复计数为多个独立分组。`Sample.rollout_id` 仍然作为 deprecated write-only alias 保留给只赋值它的旧代码
这种情况下不需要重写整个 rollout 函数,`custom_generate` 可以直接返回 `list[Sample]`。关键是:这些由同一次 rollout 拆出来的 sibling samples 必须设置相同的 `rollout_id`,这样 slime 会在训练切分和 loss 聚合时把它们视作同一次 rollout,而不是重复计数为多次独立 rollout

```python
import copy
Expand All @@ -98,7 +98,7 @@ from slime.utils.types import Sample

async def custom_generate(args, sample: Sample, sampling_params: dict) -> list[Sample]:
segments = await run_agent_and_split_segments(args, sample, sampling_params)
group_id = sample.group_id if sample.group_id is not None else sample.index
rollout_id = sample.rollout_id if sample.rollout_id is not None else sample.index

samples: list[Sample] = []
for segment in segments:
Expand All @@ -109,7 +109,7 @@ async def custom_generate(args, sample: Sample, sampling_params: dict) -> list[S
s.loss_mask = segment.loss_mask
s.reward = segment.reward
s.status = Sample.Status.COMPLETED
s.group_id = group_id
s.rollout_id = rollout_id
samples.append(s)
return samples
```
Expand Down
2 changes: 1 addition & 1 deletion examples/coding_agent_rl/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ restarts.
## Fan-out Semantics

- `generate()` returns `list[Sample]` — one Sample per trajectory **segment** (`subagent` / `wipe` / `final`).
- Per-trajectory reward is split as `reward / K` across segments; `group_id` is shared so the per-rollout-mean loss reducer still counts the trajectory once.
- Per-trajectory reward is split as `reward / K` across segments; `rollout_id` is shared so the per-rollout-mean loss reducer still counts the trajectory once.
- Sub-agent dispatch increases `K` (each completed `Agent` turn block becomes its own segment), so the effective batch after flatten can be much larger than `rollout_batch_size * n_samples_per_prompt`.

## Porting to a New Sandbox Backend
Expand Down
3 changes: 2 additions & 1 deletion examples/coding_agent_rl/generate.py
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,8 @@ def _merge_samples(
"elapsed_sec": elapsed_sec,
}

# All K samples share group_id so the loss reducer counts this trajectory once.
# All K samples share rollout_id so the loss reducer counts this
# trajectory once.
fanned = fan_out_sample_segments(
sample,
segments,
Expand Down
10 changes: 5 additions & 5 deletions examples/multi_agent/agent_system.py
Original file line number Diff line number Diff line change
Expand Up @@ -203,17 +203,17 @@ async def run_agent_system(args, sample):
args.sample = sample
args.results_dict = {"solver": [], "rewriter": [], "selector": []}
# Every sample emitted below is a training sample split out of this one
# rollout execution (the input ``sample``). Stamp the shared group id on
# every collected sample at each return point so the per-group loss
# rollout execution (the input ``sample``). Stamp the shared rollout id on
# every collected sample at each return point so the per-rollout loss
# reducer aggregates the solver / rewriter / selector siblings as one
# group instead of N, and the by-group step splitter keeps them in
# rollout instead of N, and the by-rollout step splitter keeps them in
# the same step. Captured here because ``sample`` gets shadowed by zip-
# loop variables further down.
input_group_id = sample.index
input_rollout_id = sample.index

def _emit(samples_list):
for s in samples_list:
s.group_id = input_group_id
s.rollout_id = input_rollout_id
return samples_list

problem_statement = sample.prompt
Expand Down
7 changes: 4 additions & 3 deletions slime/agent/trajectory.py
Original file line number Diff line number Diff line change
Expand Up @@ -181,22 +181,23 @@ def fan_out_sample_segments(
tokenizer,
*,
metadata: dict[str, Any] | None = None,
rollout_id: int | None = None,
) -> list[Sample]:
"""Emit one Sample per segment, splitting reward uniformly across them.

Sibling samples share ``group_id`` so reducers that average by group do
Sibling samples share ``rollout_id`` so reducers that average by rollout do
not over-count trajectories split by compaction or sub-agent dispatch.
"""
k = len(segments)
per_segment_reward = float(reward) / max(1, k)
shared_group_id = sample.group_id if sample.group_id is not None else sample.index
shared_rollout_id = getattr(sample, "index", None) if rollout_id is None else rollout_id
base_metadata = {**(sample.metadata or {}), **(metadata or {})}

out: list[Sample] = []
for i, segment in enumerate(segments):
sub = sample if i == 0 else copy.copy(sample)
write_segment_to_sample(sub, segment, per_segment_reward, tokenizer)
sub.group_id = shared_group_id
sub.rollout_id = shared_rollout_id
sub.metadata = {
**base_metadata,
**(segment.metadata or {}),
Expand Down
8 changes: 4 additions & 4 deletions slime/backends/megatron_utils/actor.py
Original file line number Diff line number Diff line change
Expand Up @@ -222,11 +222,11 @@ def _get_rollout_data(self, rollout_data_ref: Box) -> RolloutBatch:
rollout_data["loss_masks"] = [
torch.tensor(t, dtype=torch.int, device=torch.cuda.current_device()) for t in rollout_data["loss_masks"]
]
if "group_mask_sums" in rollout_data:
# Promote precomputed per-group mask totals to GPU tensors here
if "rollout_mask_sums" in rollout_data:
# Promote precomputed per-rollout mask totals to GPU tensors here
# (matching loss_masks) so the loss reducer can just divide.
rollout_data["group_mask_sums"] = torch.tensor(
rollout_data["group_mask_sums"], dtype=torch.float32, device=torch.cuda.current_device()
rollout_data["rollout_mask_sums"] = torch.tensor(
rollout_data["rollout_mask_sums"], dtype=torch.float32, device=torch.cuda.current_device()
)
if "multimodal_train_inputs" in rollout_data:
# Move multimodal training tensors to GPU in advance
Expand Down
22 changes: 11 additions & 11 deletions slime/backends/megatron_utils/data.py
Original file line number Diff line number Diff line change
Expand Up @@ -279,25 +279,25 @@ def log_rollout_data(
loss_masks = rollout_data["loss_masks"]
total_lengths = rollout_data["total_lengths"]
max_seq_lens = rollout_data.get("max_seq_lens", None)
# Same per-group denominators the training loss uses, so reported
# log_probs / returns / advantages / etc. live in the same per-group
# Same per-rollout denominators the training loss uses, so reported
# log_probs / returns / advantages / etc. live in the same per-rollout
# mean space (rather than per-sample) as the gradient signal.
group_mask_sums = rollout_data.get("group_mask_sums", None)
rollout_mask_sums = rollout_data.get("rollout_mask_sums", None)
# For per-rollout-mean metrics: ``rollout_log_metric_contribution``
# produces the ``(sum, count)`` tuple so gather_log_data's
# ``sum / count`` lands on ``sum_DP_full / num_groups`` — the
# ``Σsum / Σcount`` lands on ``sum_DP_full / num_rollouts`` — the
# same number train_one_step reports for the same samples.
dp_world = mpu.get_data_parallel_world_size(with_context_parallel=False)
num_groups_in_rollout = sum(rollout_data["global_batch_sizes"])
num_rollouts_in_rollout = sum(rollout_data["global_batch_sizes"])

for key, val in rollout_data.items():
if key in [
"tokens",
"multimodal_train_inputs",
"loss_masks",
"sample_indices",
"group_ids",
"group_mask_sums",
"rollout_ids",
"rollout_mask_sums",
"rollout_routed_experts",
"max_seq_lens",
"global_batch_sizes",
Expand Down Expand Up @@ -328,7 +328,7 @@ def log_rollout_data(
total_lengths,
response_lengths,
loss_masks,
group_mask_sums,
rollout_mask_sums,
qkv_format=args.qkv_format,
max_seq_lens=max_seq_lens,
)
Expand All @@ -337,7 +337,7 @@ def log_rollout_data(
sum_value, count = rollout_log_metric_contribution(
sum_of_sample_mean(tensor).item(),
cp_size=cp_size,
num_rollouts_in_rollout=num_groups_in_rollout,
num_rollouts_in_rollout=num_rollouts_in_rollout,
dp_size=dp_world,
)
log_dict[key] = (sum_value, count)
Expand Down Expand Up @@ -433,8 +433,8 @@ def quantile(total_value, n_quantiles, data) -> dict:
rollout_data[f"correct_length/{p}"] = [val] * num_correct_responses
if len(correct_entropy) > 0:
# NOTE: per-sample-mean over the correct subset, not per-rollout.
# A group's siblings may not all be correct, and slicing
# ``group_mask_sums`` here would leave a denom that still
# A rollout's siblings may not all be correct, and slicing
# ``rollout_mask_sums`` here would leave a denom that still
# includes incorrect siblings — meaningless for a "correct-only"
# entropy report. Per-sample-mean over the filtered subset is
# the cleanest semantic.
Expand Down
8 changes: 4 additions & 4 deletions slime/backends/megatron_utils/loss.py
Original file line number Diff line number Diff line change
Expand Up @@ -933,15 +933,15 @@ def policy_loss_function(

# [decouple IS and rejection] Rebuild sum_of_sample_mean with
# modified_response_masks for numerator correction (rejected tokens
# zeroed in pg_loss). Denominators stay the precomputed per-group
# totals from ``group_mask_sums`` (based on original loss_masks) —
# zeroed in pg_loss). Denominators stay the precomputed per-rollout
# totals from ``rollout_mask_sums`` (based on original loss_masks) —
# same normalizer as the outer reducer, so pg_loss and the rest of the
# reported metrics live in the same per-rollout-mean space.
sum_of_sample_mean = get_sum_of_sample_mean(
total_lengths,
response_lengths,
modified_response_masks,
batch["group_mask_sums"],
batch["rollout_mask_sums"],
args.calculate_per_token_loss,
args.qkv_format,
max_seq_lens,
Expand Down Expand Up @@ -1177,7 +1177,7 @@ def loss_function(
batch["total_lengths"],
batch["response_lengths"],
batch["loss_masks"],
batch["group_mask_sums"],
batch["rollout_mask_sums"],
args.calculate_per_token_loss,
args.qkv_format,
batch.get("max_seq_lens", None),
Expand Down
2 changes: 1 addition & 1 deletion slime/backends/megatron_utils/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -501,7 +501,7 @@ def forward_step(data_iterator: DataIterator, model: GPTModel, return_schedule_p
"rollout_log_probs",
"max_seq_lens",
"teacher_log_probs",
"group_mask_sums",
"rollout_mask_sums",
],
args.data_pad_size_multiplier,
args.qkv_format,
Expand Down
Loading
Loading