Skip to content
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 33 additions & 1 deletion docs/algorithms.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ This page covers the math and the configurable algorithmic components: the algor
- [Custom Loss](#custom-loss)
- [Advantage](#advantage)
- [Default Advantage](#default-advantage)
- [Hierarchical Advantage (Proposer–Solver)](#hierarchical-advantage-proposersolver)
- [Self-Play Advantage (RAE)](#self-play-advantage-rae)
- [Authoring an Algorithm](#authoring-an-algorithm)
- [Reference Scoring](#reference-scoring)
Expand Down Expand Up @@ -69,6 +70,7 @@ type = "grpo" # the default
| `grpo` | policy | `rl` on actions | Standard group-relative RL. |
| `max_rl` | policy | `rl` on actions | MaxRL ([arXiv:2602.02710](https://arxiv.org/abs/2602.02710)): GRPO's centered reward normalized by the group **mean** instead of the standard deviation — the gradient is unbiased for the order-`group_size` truncation of the maximum-likelihood objective, upweighting hard examples like `1/p`. |
| `rae` | policy | `rl` on actions | RAE (SPIRAL, [arXiv:2506.24119](https://arxiv.org/abs/2506.24119)): reward minus a per-agent EMA baseline of that agent's own rewards — the estimator for multi-agent self-play envs, where the group mean would mix the agents' opposite reward scales. See [Self-Play Advantage](#self-play-advantage-rae). |
| `hierarchical_grpo` | policy | `rl` on actions | For Proposer-Solver type envs. Needs `episode_agents`. See [Hierarchical Advantage](#hierarchical-advantage-proposersolver). |
| `opd` | policy | `ref_kl` on actions | On-policy distillation ([Thinking Machines](https://thinkingmachines.ai/blog/on-policy-distillation/)): the policy samples, per-token reverse KL against a reference model as the gradient signal. Needs a `teacher`. |
| `sft` | *(the teacher)* | `ce` on actions | Hard distillation: a frozen model generates rollouts, the policy trains with CE on its tokens. Needs a frozen `sampling.source` (the teacher it samples from). |
| `opsd` | policy | `ref_kl` on actions | SDFT ([arXiv:2601.19897](https://arxiv.org/abs/2601.19897)): the model is its own reference, conditioned on an expert demonstration. The teacher *is* the live policy (the paper's setting, no extra deployment) — no model to configure. |
Expand Down Expand Up @@ -143,6 +145,7 @@ At runtime, each env's resolved config builds two objects: a `Sampler` (`prime_r
| `echo` | `EchoAlgorithm` | `score_rollout`: weighted ce on observation tokens; `score_group`: group-norm credit (inherited) |
| `max_rl` | `MaxRLAlgorithm` | `score_group`: mean-normalized group credit |
| `rae` | `RAEAlgorithm` | `score_group`: per-agent EMA-baseline credit |
| `hierarchical_grpo` | `HierarchicalGRPOAlgorithm` | `score_group`: GRPO baseline per episode for solvers, per group for the proposer |
| `opd` | `OPDAlgorithm` | `score_rollout`: own-context prefill under the teacher |
| `opsd` | `OPSDAlgorithm` | `score_rollout`: demo-conditioned prefill under the live policy |
| `sft` | `SFTDistillAlgorithm` | `score_group`: group-norm credit (feeds filters) |
Expand Down Expand Up @@ -179,7 +182,7 @@ $$
\mathcal{L} = \frac{\sum \mathcal{L}_{rl}}{N_{rl}} + \frac{\sum \mathcal{L}_{ce}}{N_{ce}} + \frac{\sum \mathcal{L}_{ref\_kl}}{N_{ref\_kl}}
$$

- `rl` — the configured RL loss (`[trainer.loss]`): DPPO + KL by default, or a [custom loss](#custom-loss). Fed by the advantage-assigning algorithms (`grpo`, `max_rl`, `rae`, and `echo`'s action tokens).
- `rl` — the configured RL loss (`[trainer.loss]`): DPPO + KL by default, or a [custom loss](#custom-loss). Fed by the advantage-assigning algorithms (`grpo`, `max_rl`, `rae`, `hierarchical_grpo`, and `echo`'s action tokens).
- `ce` — masked NLL. Used for frozen-model tokens (`sft`) and env-observation tokens (`echo`).
- `ref_kl` — the per-token reverse KL to a reference model ($\log \pi_{\text{ref}} - \log \pi$) as the policy-gradient signal, importance-ratio corrected with a one-sided trust region (`opd`, `opsd`). Requires `ref_logprobs` from a [reference scoring](#reference-scoring); the scoring model must be a vLLM server (it's the only one that exposes `prompt_logprobs`).

Expand Down Expand Up @@ -282,6 +285,7 @@ The per-token training signal is set by `algo.type` and the [algorithm](#the-alg
| `grpo` | `rl` | Group-norm: reward minus per-group baseline, optional length penalty. |
| `max_rl` | `rl` | Mean-normalized group credit (maximum-likelihood RL). |
| `rae` | `rl` | Reward minus a per-agent EMA baseline (SPIRAL's role-conditioned advantage estimation) — for multi-agent self-play envs. |
| `hierarchical_grpo` | `rl` | Reward minus a GRPO baseline taken at two levels of the episode tree: each solver against the siblings attempting its own minted task, the proposer across the group's proposals — proposer-solver envs only. |
| `echo` | `rl` + `ce` | Group-norm on action tokens, plus weighted CE on env-provided tokens selected by message role (each role's `alpha` is its ECHO λ), optionally narrowed by a user filter. |
| `opd` | `ref_kl` | On-policy distillation: per-token reverse KL to a reference model (`teacher`, an inline frozen hosted model), evaluated in the trainer from shipped reference logprobs. No credit — rollouts keep `advantages = None` (advantage-based filters never fire) and ship no advantage stream; `group_size` only fans out sampling. |
| `opsd` | `ref_kl` | SDFT: per-token reverse KL to a demo-conditioned reference. No credit — rollouts keep `advantages = None` (advantage-based filters never fire) and ship no advantage stream. |
Expand All @@ -303,6 +307,34 @@ type = "grpo"
type = "linear"
```

### Hierarchical Advantage (Proposer–Solver)

Task-generating envs break the flat-group assumption twice over. In `proposer-solver-v1`, one episode is a proposer trace plus the n solver traces its minted problem fanned out to, and a group is `group_size` episodes of the same source task — the proposer asked `group_size` times to invent a problem. Pooling solver rewards across episodes baselines them against attempts at *different* problems; pooling proposer and solver traces mixes reward scales across agents (the proposer earns learnability, the solvers correctness).

`hierarchical_grpo` computes GRPO baselines at two levels of that episode tree. Every trainable trace is mean-centered against its *peer set*:

- agents listed in `episode_agents` (the solvers) against the same-agent traces of **their own episode** — sibling attempts at the same minted task;
- every other agent (the proposer) against its same-agent traces **across the group** — parallel attempts at the same source task, each rewarded by what its minted task did to the solvers (`proposer-solver-v1`'s learnability, `4p(1−p)`, peaks when half the solvers crack the problem).

Rewards never mix across agents or across minted tasks. A peer set of one centers to zero advantage (GRPO's singleton convention; the zero-advantage filter drops it). Config validation rejects any env that isn't a proposer-solver env — the two-level peer sets *are* that episode tree, and on a flat env they collapse to one trace each, so every group would train on nothing but zeros. `episode_agents` is required — which agents are episode-scoped is env truth the algorithm must not guess:

```toml
[orchestrator.algo]
type = "hierarchical_grpo"
episode_agents = ["solver"]

[[orchestrator.train.env]]
name = "proposer-solver"
env.taskset = { id = "proposer-solver-v1" }
env.n = 4 # solvers per proposed problem
env.proposer.harness = { id = "null" }
env.proposer.runtime = { type = "subprocess" }
env.solver.harness = { id = "null" }
env.solver.runtime = { type = "subprocess" }
```

Here `group_size` is the number of proposals per source task and `env.n` the solvers per proposal, so one group carries `group_size × (1 + n)` traces. To train only one side, flip the env's own switches (`env.train_proposer` / `env.train_solver`) — the untrainable side's traces never reach the advantage computation.

### Self-Play Advantage (RAE)

Group-relative baselines assume the group is exchangeable attempts by one agent. A multi-agent self-play env breaks that: one episode yields one trace per agent, all trainable, and in a zero-sum game the rewards sum to ~0 whatever the policy does — the group mean carries no information, and centering against it converts any structural asymmetry (a first-mover edge) into permanent credit for one agent.
Expand Down
3 changes: 2 additions & 1 deletion docs/training.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ A condensed view of the knobs you'll most often tune. For trainer-side paralleli
| `orchestrator.batch_size` | Tasks per trainer step. |
| `orchestrator.group_size` | Rollouts generated per task. |
| `orchestrator.max_off_policy_steps` | How many distinct policies may have contributed to one rollout before it's discarded (default 8). The main off-policy dial on long agentic rollouts — bump for throughput, lower for tighter on-policyness. Watch `errored_rollouts` and `mismatch_kl/all/mean` when tuning. |
| `[orchestrator.algo]` | Training algorithm — its `type` names it (`grpo` default, `max_rl`, `rae`, `opd`, `opsd`, `sft`, `echo`). See [Algorithms](#algorithms). |
| `[orchestrator.algo]` | Training algorithm — its `type` names it (`grpo` default, `max_rl`, `rae`, `hierarchical_grpo`, `opd`, `opsd`, `sft`, `echo`). See [Algorithms](#algorithms). |
| `[[orchestrator.train.env]]` | Training environments. List multiple tables for multi-env training; weight them via `ratio`. See [Configuration § Environments](configuration.md#environments-orchestratortrainenv). |
| `[[orchestrator.eval.env]]` + `orchestrator.eval.interval` | Eval environments and cadence (default every 100 steps). |

Expand Down Expand Up @@ -90,6 +90,7 @@ The RL entrypoint supports several training algorithms, switched via `[orchestra
| `grpo` (default) | None | Standard group-relative RL |
| `max_rl` | None | [MaxRL](https://arxiv.org/abs/2602.02710): GRPO with mean-normalized advantages (maximum-likelihood RL) |
| `rae` | None | [SPIRAL](https://arxiv.org/abs/2506.24119)'s role-conditioned advantage estimation: reward minus a per-agent EMA baseline, for multi-agent self-play envs (e.g. `kuhn-poker-v1`) |
| `hierarchical_grpo` | None | Two-level GRPO for proposer-solver envs: each solver is baselined against the siblings attempting its own minted task, the proposer across the group's proposals |
| `opd` | Required, must be vLLM (needs `prompt_logprobs`) | [On-policy distillation](https://thinkingmachines.ai/blog/on-policy-distillation/): the policy generates rollouts, the trainer minimizes per-token reverse KL to a reference model |
| `sft` | Required, any OpenAI-compatible endpoint | Hard-distill: a frozen model generates rollouts, the policy trains on its tokens |
| `opsd` | None — the live policy is its own reference (no deployment) | [SDFT](https://arxiv.org/abs/2601.19897): the model is its own reference conditioned on expert demonstrations |
Expand Down
64 changes: 63 additions & 1 deletion packages/prime-rl-configs/src/prime_rl/configs/algorithm.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@

from typing import Annotated, Any, ClassVar, Literal, TypeAlias

import verifiers.v1 as vf
from pydantic import Field, model_validator
from renderers import AutoRendererConfig, RendererConfig

Expand Down Expand Up @@ -188,6 +189,13 @@ def validate_sampling_source(self):
)
return self

def validate_env(self, env_config: vf.EnvConfig) -> None:
"""Raise if this algorithm cannot run on the env it is configured for,
given that env's resolved config. Most algorithms read only the rollouts
and their rewards, so they run on any env and the base does nothing;
override where the credit assignment encodes an env's episode structure.
Called once per train env after algorithm inheritance resolves."""


class GRPOAlgoConfig(BaseAlgoConfig):
type: Literal["grpo"] = "grpo"
Expand Down Expand Up @@ -253,6 +261,52 @@ class RAEAlgoConfig(BaseAlgoConfig):
memory — a restart re-warms them over ~1/(1 − decay) traces per agent."""


class HierarchicalGRPOAlgoConfig(BaseAlgoConfig):
type: Literal["hierarchical_grpo"] = "hierarchical_grpo"
"""Hierarchical GRPO for task-generating envs (``proposer-solver-v1`` and
friends): GRPO baselines computed at two levels of the episode tree. A
group is ``group_size`` episodes of the same source task, and one episode
holds a proposer trace plus the n solver traces its minted task fanned out
to. Every trainable trace is mean-centered against its *peer set*: agents
listed in ``episode_agents`` (the solvers) against the same-agent traces of
their own episode — sibling attempts at the same minted task — and every
other agent (the proposer) against its same-agent traces across the group —
parallel attempts at the same source task. Rewards never mix across agents
or across minted tasks. Only a proposer-solver env may train under it — the
peer sets are that env's episode tree (see ``validate_env``)."""

action_loss_type: ClassVar[ActionLossType] = "rl"

episode_agents: list[str] = Field(min_length=1)
"""Agents whose peer set is their own episode (``["solver"]`` for
``proposer-solver-v1``): their reward compares only against sibling traces
of the same agent in the same episode, because each episode mints its own
task and rewards are not exchangeable across episodes. Required — which
agents are episode-scoped is env truth the algorithm must not guess."""

def validate_env(self, env_config: vf.EnvConfig) -> None:
"""Only a proposer-solver env may train under hierarchical GRPO. The
two-level peer sets *are* that env's episode tree — one proposer trace
per episode plus the solver traces its minted task fanned out to — so on
a flat env the levels collapse and every trace silently lands in a peer
set of one (zero advantage, whole group filtered). Nothing in a rollout
reveals the tree, so the env's own config class is the check."""
try:
from proposer_solver_v1 import ProposerSolverEnvConfig
except ImportError as e:
raise ValueError(
"algorithm 'hierarchical_grpo' requires a proposer-solver env, but the "
"proposer-solver-v1 package is not installed (`uv sync --all-packages`)."
) from e
if not isinstance(env_config, ProposerSolverEnvConfig):
raise ValueError(
f"algorithm 'hierarchical_grpo' needs a proposer-solver env, but this env resolved to "
f"{type(env_config).__name__} — its two-level baselines read an episode tree "
"(a proposer trace plus the solver traces its minted task fanned out to) that a flat "
"env does not have. Use 'grpo' (or 'rae' for multi-agent self-play)."
)


class OPDAlgoConfig(BaseAlgoConfig):
type: Literal["opd"] = "opd"
"""On-policy distillation: the per-token signal is the reverse KL to
Expand Down Expand Up @@ -326,7 +380,14 @@ def require_frozen_source(self):


AlgoConfig: TypeAlias = Annotated[
GRPOAlgoConfig | EchoAlgoConfig | MaxRLAlgoConfig | RAEAlgoConfig | OPDAlgoConfig | OPSDAlgoConfig | SFTAlgoConfig,
GRPOAlgoConfig
| EchoAlgoConfig
| MaxRLAlgoConfig
| RAEAlgoConfig
| HierarchicalGRPOAlgoConfig
| OPDAlgoConfig
| OPSDAlgoConfig
| SFTAlgoConfig,
Field(discriminator="type"),
]
"""The training algorithm: sampling plus the per-token training signal (credit
Expand All @@ -336,6 +397,7 @@ def require_frozen_source(self):
- ``grpo`` — policy group sampling, group-relative advantage, RL loss (the default).
- ``max_rl`` — GRPO with mean-normalized advantages (maximum-likelihood RL).
- ``rae`` — reward minus a per-agent EMA baseline (SPIRAL), for multi-agent self-play envs.
- ``hierarchical_grpo`` — two-level GRPO for proposer-solver envs: each solver is baselined against the siblings attempting its own minted task, the proposer across the group's proposals. Needs ``episode_agents``.
- ``opd`` — on-policy distillation: policy samples, per-token reverse KL against a reference model. Needs ``teacher``.
- ``opsd`` — SDFT: policy samples, demo-conditioned reverse KL against the live policy (the teacher is the policy itself).
- ``sft`` — a frozen model samples, the policy trains with CE on its tokens. Needs a frozen ``sampling.source``.
Expand Down
11 changes: 11 additions & 0 deletions packages/prime-rl-configs/src/prime_rl/configs/orchestrator.py
Original file line number Diff line number Diff line change
Expand Up @@ -622,6 +622,17 @@ def inherit_env_algorithms(self):
env_cfg.algo = self.algo.model_copy(deep=True)
return self

@model_validator(mode="after")
def validate_env_algorithms(self):
"""Let each env's resolved algorithm reject the env it was pointed at —
the algorithms whose credit assignment encodes an env's episode structure
(hierarchical_grpo) only make sense on that env, and a flat env would
silently train on collapsed peer sets."""
for env_cfg in self.train.env:
assert env_cfg.algo is not None # resolved by inherit_env_algorithms
env_cfg.algo.validate_env(env_cfg.env)
return self

@property
def any_policy_sourced(self) -> bool:
"""True when at least one train env samples rollouts from the live policy."""
Expand Down
2 changes: 1 addition & 1 deletion skills/configs/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ CLI: `--orchestrator.train.env.0.env.taskset.id reverse-text-v1`.

**Discriminated unions** — set the `type` field to pick the variant (`[orchestrator.algo] type = "max_rl"`). Omit `type` to keep the default variant.

**Algorithms** — `[orchestrator.algo] type = "grpo" | "max_rl" | "rae" | "opd" | "opsd" | "sft" | "echo"` — the type names the algorithm (credit assignment + loss routing, fused), and each type's class defaults are its vetted setting; any other key you set is your own assembly (e.g. `[orchestrator.algo.roles.user] alpha = 0.1` for echo — setting any echo role replaces the whole role table). There is no preset layer, and no config hook that points at user code — a new algorithm is a named class in the repo (subclass `Algorithm`, register it). Per-env override: `[orchestrator.train.env.algo] type = "opd"` (the env assembles its own algorithm). prime-rl only hosts the trainable policy; frozen models are inline external endpoints on the algorithm, named where the model is used — `[orchestrator.algo.teacher]` for opd (the frozen model scored against), `[orchestrator.algo.sampling.source]` for sft (the model it samples from), each with `name` + `base_url`. There is no shared `teacher` slot. opsd declares no model — it self-distills against the live policy. See `docs/algorithms.md`.
**Algorithms** — `[orchestrator.algo] type = "grpo" | "max_rl" | "rae" | "hierarchical_grpo" | "opd" | "opsd" | "sft" | "echo"` — the type names the algorithm (credit assignment + loss routing, fused), and each type's class defaults are its vetted setting; any other key you set is your own assembly (e.g. `[orchestrator.algo.roles.user] alpha = 0.1` for echo — setting any echo role replaces the whole role table). An algorithm may constrain its env: `hierarchical_grpo` is rejected at config validation on anything but a proposer-solver env, since its two-level baselines read that episode tree. There is no preset layer, and no config hook that points at user code — a new algorithm is a named class in the repo (subclass `Algorithm`, register it). Per-env override: `[orchestrator.train.env.algo] type = "opd"` (the env assembles its own algorithm). prime-rl only hosts the trainable policy; frozen models are inline external endpoints on the algorithm, named where the model is used — `[orchestrator.algo.teacher]` for opd (the frozen model scored against), `[orchestrator.algo.sampling.source]` for sft (the model it samples from), each with `name` + `base_url`. There is no shared `teacher` slot. opsd declares no model — it self-distills against the live policy. See `docs/algorithms.md`.

**`BaseModel | None` fields** — bare flag enables defaults; nested override enables and sets:

Expand Down
Loading