diff --git a/docs/algorithms.md b/docs/algorithms.md index 2c8846e09ff..fb6874a58b6 100644 --- a/docs/algorithms.md +++ b/docs/algorithms.md @@ -17,6 +17,7 @@ This page covers the math and the configurable algorithmic components: the algor - [Custom Loss](#custom-loss) - [Advantage](#advantage) - [Default Advantage](#default-advantage) + - [Hierarchical GRPO](#hierarchical-grpo) - [Self-Play Advantage (RAE)](#self-play-advantage-rae) - [Authoring an Algorithm](#authoring-an-algorithm) - [Reference Scoring](#reference-scoring) @@ -69,6 +70,7 @@ type = "grpo" # the default | `grpo` | policy | `rl` on actions | Standard group-relative RL. | | `max_rl` | policy | `rl` on actions | MaxRL ([arXiv:2602.02710](https://arxiv.org/abs/2602.02710)): GRPO's centered reward normalized by the group **mean** instead of the standard deviation — the gradient is unbiased for the order-`group_size` truncation of the maximum-likelihood objective, upweighting hard examples like `1/p`. | | `rae` | policy | `rl` on actions | RAE (SPIRAL, [arXiv:2506.24119](https://arxiv.org/abs/2506.24119)): reward minus a per-agent EMA baseline of that agent's own rewards — the estimator for multi-agent self-play envs, where the group mean would mix the agents' opposite reward scales. See [Self-Play Advantage](#self-play-advantage-rae). | +| `hierarchical_grpo` | policy | `rl` on actions | GRPO for proposer-solver envs. Solvers are compared only with attempts on the same proposed problem; proposers are compared with the other proposals in the group. See [Hierarchical GRPO](#hierarchical-grpo). | | `opd` | policy | `ref_kl` on actions | On-policy distillation ([Thinking Machines](https://thinkingmachines.ai/blog/on-policy-distillation/)): the policy samples, per-token reverse KL against a reference model as the gradient signal. Needs a `teacher`. | | `sft` | *(the teacher)* | `ce` on actions | Hard distillation: a frozen model generates rollouts, the policy trains with CE on its tokens. Needs a frozen `sampling.source` (the teacher it samples from). | | `opsd` | policy | `ref_kl` on actions | SDFT ([arXiv:2601.19897](https://arxiv.org/abs/2601.19897)): the model is its own reference, conditioned on an expert demonstration. The teacher *is* the live policy (the paper's setting, no extra deployment) — no model to configure. | @@ -143,6 +145,7 @@ At runtime, each env's resolved config builds two objects: a `Sampler` (`prime_r | `echo` | `EchoAlgorithm` | `score_rollout`: weighted ce on observation tokens; `score_group`: group-norm credit (inherited) | | `max_rl` | `MaxRLAlgorithm` | `score_group`: mean-normalized group credit | | `rae` | `RAEAlgorithm` | `score_group`: per-agent EMA-baseline credit | +| `hierarchical_grpo` | `HierarchicalGRPOAlgorithm` | `score_group`: GRPO baseline per episode for solvers, per group for the proposer | | `opd` | `OPDAlgorithm` | `score_rollout`: own-context prefill under the teacher | | `opsd` | `OPSDAlgorithm` | `score_rollout`: demo-conditioned prefill under the live policy | | `sft` | `SFTDistillAlgorithm` | `score_group`: group-norm credit (feeds filters) | @@ -179,7 +182,7 @@ $$ \mathcal{L} = \frac{\sum \mathcal{L}_{rl}}{N_{rl}} + \frac{\sum \mathcal{L}_{ce}}{N_{ce}} + \frac{\sum \mathcal{L}_{ref\_kl}}{N_{ref\_kl}} $$ -- `rl` — the configured RL loss (`[trainer.loss]`): DPPO + KL by default, or a [custom loss](#custom-loss). Fed by the advantage-assigning algorithms (`grpo`, `max_rl`, `rae`, and `echo`'s action tokens). +- `rl` — the configured RL loss (`[trainer.loss]`): DPPO + KL by default, or a [custom loss](#custom-loss). Fed by the advantage-assigning algorithms (`grpo`, `max_rl`, `rae`, `hierarchical_grpo`, and `echo`'s action tokens). - `ce` — masked NLL. Used for frozen-model tokens (`sft`) and env-observation tokens (`echo`). - `ref_kl` — the per-token reverse KL to a reference model ($\log \pi_{\text{ref}} - \log \pi$) as the policy-gradient signal, importance-ratio corrected with a one-sided trust region (`opd`, `opsd`). Requires `ref_logprobs` from a [reference scoring](#reference-scoring); the scoring model must be a vLLM server (it's the only one that exposes `prompt_logprobs`). @@ -282,6 +285,7 @@ The per-token training signal is set by `algo.type` and the [algorithm](#the-alg | `grpo` | `rl` | Group-norm: reward minus per-group baseline, optional length penalty. | | `max_rl` | `rl` | Mean-normalized group credit (maximum-likelihood RL). | | `rae` | `rl` | Reward minus a per-agent EMA baseline (SPIRAL's role-conditioned advantage estimation) — for multi-agent self-play envs. | +| `hierarchical_grpo` | `rl` | GRPO for proposer-solver envs: solvers are compared within one proposed problem, while proposers are compared across proposals. | | `echo` | `rl` + `ce` | Group-norm on action tokens, plus weighted CE on env-provided tokens selected by message role (each role's `alpha` is its ECHO λ), optionally narrowed by a user filter. | | `opd` | `ref_kl` | On-policy distillation: per-token reverse KL to a reference model (`teacher`, an inline frozen hosted model), evaluated in the trainer from shipped reference logprobs. No credit — rollouts keep `advantages = None` (advantage-based filters never fire) and ship no advantage stream; `group_size` only fans out sampling. | | `opsd` | `ref_kl` | SDFT: per-token reverse KL to a demo-conditioned reference. No credit — rollouts keep `advantages = None` (advantage-based filters never fire) and ship no advantage stream. | @@ -303,6 +307,57 @@ type = "grpo" type = "linear" ``` +### Hierarchical GRPO + +GRPO gives each rollout its reward minus the average reward of comparable rollouts. In an ordinary single-agent group, every rollout answers the same task, so one group average is enough. + +A proposer-solver env is different. Starting from one source task, it produces several proposed problems, then runs several solver attempts on each problem: + +```text +one source task +├── proposed problem A +│ ├── proposer trace +│ ├── solver attempt 1 +│ └── solver attempt 2 +└── proposed problem B + ├── proposer trace + ├── solver attempt 1 + └── solver attempt 2 +``` + +The solver attempts for A should not be compared with the solver attempts for B: the two problems may have very different difficulty. Proposer and solver rewards should not be compared either: they measure different jobs. + +`hierarchical_grpo` therefore chooses the average separately for each role: + +| Trace | Compared with | Why | +|---|---|---| +| Solver | Other solver attempts on the same proposed problem | They attempted the same problem. | +| Proposer | Other proposer traces in the group | They started from the same source task and proposed alternatives. | + +For example, if three solvers receive rewards `[1, 1, 0]` on one proposed problem, their average is `2/3` and their advantages are `[1/3, 1/3, -2/3]`. Solver rewards from other proposed problems do not affect those values. The proposers are scored separately according to how useful their problems were for the solvers, then compared with the other proposers in the group. + +Configure which roles are compared within a single proposed problem with `episode_agents`. For `proposer-solver-v1`, that role is `solver`: + +```toml +[orchestrator.algo] +type = "hierarchical_grpo" +episode_agents = ["solver"] + +[[orchestrator.train.source]] +name = "proposer-solver" +group_size = 4 # proposed problems per source task +env.taskset = { id = "proposer-solver-v1" } +env.n = 4 # solver attempts per proposed problem +env.proposer.harness = { id = "null" } +env.proposer.runtime = { type = "subprocess" } +env.solver.harness = { id = "null" } +env.solver.runtime = { type = "subprocess" } +``` + +`group_size` controls how many problems are proposed from each source task. `env.n` controls how many solvers attempt each proposed problem. If a comparison contains only one trace—for example, a solver when `env.n = 1`—its advantage is zero and the zero-advantage filter removes it. + +This algorithm is accepted only for proposer-solver envs. Use the env's `train_proposer` and `train_solver` settings if you want to train only one role. + ### Self-Play Advantage (RAE) Group-relative baselines assume the group is exchangeable attempts by one agent. A multi-agent self-play env breaks that: one episode yields one trace per agent, all trainable, and in a zero-sum game the rewards sum to ~0 whatever the policy does — the group mean carries no information, and centering against it converts any structural asymmetry (a first-mover edge) into permanent credit for one agent. diff --git a/docs/training.md b/docs/training.md index 1fbf0b82e12..da60d5b4fb7 100644 --- a/docs/training.md +++ b/docs/training.md @@ -59,7 +59,7 @@ A condensed view of the knobs you'll most often tune. For trainer-side paralleli | `orchestrator.batch_size` | Tasks per trainer step. | | `orchestrator.group_size` | Rollouts generated per task. | | `orchestrator.max_off_policy_steps` | How many distinct policies may have contributed to one rollout before it's discarded (default 8). The main off-policy dial on long agentic rollouts — bump for throughput, lower for tighter on-policyness. Watch `errored_rollouts` and `mismatch_kl/all/mean` when tuning. | -| `[orchestrator.algo]` | Training algorithm — its `type` names it (`grpo` default, `max_rl`, `rae`, `opd`, `opsd`, `sft`, `echo`). See [Algorithms](#algorithms). | +| `[orchestrator.algo]` | Training algorithm — its `type` names it (`grpo` default, `max_rl`, `rae`, `hierarchical_grpo`, `opd`, `opsd`, `sft`, `echo`). See [Algorithms](#algorithms). | | `[[orchestrator.train.source]]` | Training sources. List multiple tables for multi-env training; weight them via `ratio`. See [Configuration § Training sources](configuration.md#training-sources-orchestratortrainsource). | | `[[orchestrator.eval.source]]` + `orchestrator.eval.interval` | Eval environments and cadence (default every 100 steps). | @@ -90,6 +90,7 @@ The RL entrypoint supports several training algorithms, switched via `[orchestra | `grpo` (default) | None | Standard group-relative RL | | `max_rl` | None | [MaxRL](https://arxiv.org/abs/2602.02710): GRPO with mean-normalized advantages (maximum-likelihood RL) | | `rae` | None | [SPIRAL](https://arxiv.org/abs/2506.24119)'s role-conditioned advantage estimation: reward minus a per-agent EMA baseline, for multi-agent self-play envs (e.g. `kuhn-poker-v1`) | +| `hierarchical_grpo` | None | GRPO for proposer-solver envs: compare solvers only with attempts on the same proposed problem, and compare proposers with the other proposals in the group | | `opd` | Required, must be vLLM (needs `prompt_logprobs`) | [On-policy distillation](https://thinkingmachines.ai/blog/on-policy-distillation/): the policy generates rollouts, the trainer minimizes per-token reverse KL to a reference model | | `sft` | Required, any OpenAI-compatible endpoint | Hard-distill: a frozen model generates rollouts, the policy trains on its tokens | | `opsd` | None — the live policy is its own reference (no deployment) | [SDFT](https://arxiv.org/abs/2601.19897): the model is its own reference conditioned on expert demonstrations | diff --git a/packages/prime-rl-configs/src/prime_rl/configs/algorithm.py b/packages/prime-rl-configs/src/prime_rl/configs/algorithm.py index 1549c2db43e..a4d212d9c81 100644 --- a/packages/prime-rl-configs/src/prime_rl/configs/algorithm.py +++ b/packages/prime-rl-configs/src/prime_rl/configs/algorithm.py @@ -35,6 +35,7 @@ from typing import Annotated, Any, ClassVar, Literal, TypeAlias +import verifiers.v1 as vf from pydantic import Field, model_validator from renderers import AutoRendererConfig, RendererConfig @@ -188,6 +189,13 @@ def validate_sampling_source(self): ) return self + def validate_env(self, env_config: vf.EnvConfig) -> None: + """Raise if this algorithm cannot run on the env it is configured for, + given that env's resolved config. Most algorithms read only the rollouts + and their rewards, so they run on any env and the base does nothing; + override where the credit assignment encodes an env's episode structure. + Called once per train env after algorithm inheritance resolves.""" + class GRPOAlgoConfig(BaseAlgoConfig): type: Literal["grpo"] = "grpo" @@ -253,6 +261,39 @@ class RAEAlgoConfig(BaseAlgoConfig): memory — a restart re-warms them over ~1/(1 − decay) traces per agent.""" +class HierarchicalGRPOAlgoConfig(BaseAlgoConfig): + type: Literal["hierarchical_grpo"] = "hierarchical_grpo" + """GRPO for proposer-solver envs. + + Agents in ``episode_agents`` are compared with same-role attempts on the + same proposed problem. Other agents are compared with the same role across + proposals generated from one source task. This keeps rewards from different + problems and different roles out of the same average.""" + + action_loss_type: ClassVar[ActionLossType] = "rl" + + episode_agents: list[str] = Field(min_length=1) + """Roles compared within one proposed problem. Use ``["solver"]`` for + ``proposer-solver-v1``.""" + + def validate_env(self, env_config: vf.EnvConfig) -> None: + """Require the proposer-solver structure used by the comparisons.""" + try: + from proposer_solver_v1 import ProposerSolverEnvConfig + except ImportError as e: + raise ValueError( + "algorithm 'hierarchical_grpo' requires a proposer-solver env, but the " + "proposer-solver-v1 package is not installed (`uv sync --all-packages`)." + ) from e + if not isinstance(env_config, ProposerSolverEnvConfig): + raise ValueError( + f"algorithm 'hierarchical_grpo' needs a proposer-solver env, but this env resolved to " + f"{type(env_config).__name__}. It compares solver attempts on the same proposed " + "problem and proposers across proposals. Use 'grpo' for a flat env or 'rae' for " + "multi-agent self-play." + ) + + class OPDAlgoConfig(BaseAlgoConfig): type: Literal["opd"] = "opd" """On-policy distillation: the per-token signal is the reverse KL to @@ -326,7 +367,14 @@ def require_frozen_source(self): AlgoConfig: TypeAlias = Annotated[ - GRPOAlgoConfig | EchoAlgoConfig | MaxRLAlgoConfig | RAEAlgoConfig | OPDAlgoConfig | OPSDAlgoConfig | SFTAlgoConfig, + GRPOAlgoConfig + | EchoAlgoConfig + | MaxRLAlgoConfig + | RAEAlgoConfig + | HierarchicalGRPOAlgoConfig + | OPDAlgoConfig + | OPSDAlgoConfig + | SFTAlgoConfig, Field(discriminator="type"), ] """The training algorithm: sampling plus the per-token training signal (credit @@ -336,6 +384,7 @@ def require_frozen_source(self): - ``grpo`` — policy group sampling, group-relative advantage, RL loss (the default). - ``max_rl`` — GRPO with mean-normalized advantages (maximum-likelihood RL). - ``rae`` — reward minus a per-agent EMA baseline (SPIRAL), for multi-agent self-play envs. +- ``hierarchical_grpo`` — GRPO for proposer-solver envs: solvers are compared within one proposed problem and proposers across proposals. Needs ``episode_agents``. - ``opd`` — on-policy distillation: policy samples, per-token reverse KL against a reference model. Needs ``teacher``. - ``opsd`` — SDFT: policy samples, demo-conditioned reverse KL against the live policy (the teacher is the policy itself). - ``sft`` — a frozen model samples, the policy trains with CE on its tokens. Needs a frozen ``sampling.source``. diff --git a/packages/prime-rl-configs/src/prime_rl/configs/orchestrator.py b/packages/prime-rl-configs/src/prime_rl/configs/orchestrator.py index 86657ba0750..434a361f28b 100644 --- a/packages/prime-rl-configs/src/prime_rl/configs/orchestrator.py +++ b/packages/prime-rl-configs/src/prime_rl/configs/orchestrator.py @@ -606,6 +606,14 @@ def inherit_env_algorithms(self): env_cfg.algo = self.algo.model_copy(deep=True) return self + @model_validator(mode="after") + def validate_env_algorithms(self): + """Let each algorithm reject environments it cannot score correctly.""" + for env_cfg in self.train.source: + assert env_cfg.algo is not None # resolved by inherit_env_algorithms + env_cfg.algo.validate_env(env_cfg.env) + return self + @property def any_policy_sourced(self) -> bool: """True when at least one train env samples rollouts from the live policy.""" diff --git a/skills/configs/SKILL.md b/skills/configs/SKILL.md index 9c5d47f6cc6..2bede0dc866 100644 --- a/skills/configs/SKILL.md +++ b/skills/configs/SKILL.md @@ -60,7 +60,7 @@ CLI: `--orchestrator.train.source.0.env.taskset.id reverse-text-v1` or `--orches **Discriminated unions** — set the `type` field to pick the variant (`[orchestrator.algo] type = "max_rl"`). Omit `type` to keep the default variant. -**Algorithms** — `[orchestrator.algo] type = "grpo" | "max_rl" | "rae" | "opd" | "opsd" | "sft" | "echo"` — the type names the algorithm (credit assignment + loss routing, fused), and each type's class defaults are its vetted setting; any other key you set is your own assembly (e.g. `[orchestrator.algo.roles.user] alpha = 0.1` for echo — setting any echo role replaces the whole role table). There is no preset layer, and no config hook that points at user code — a new algorithm is a named class in the repo (subclass `Algorithm`, register it). Per-source override: `[orchestrator.train.source.algo] type = "opd"` (the source assembles its own algorithm). prime-rl only hosts the trainable policy; frozen models are inline external endpoints on the algorithm, named where the model is used — `[orchestrator.algo.teacher]` for opd (the frozen model scored against), `[orchestrator.algo.sampling.source]` for sft (the model it samples from), each with `name` + `base_url`. There is no shared `teacher` slot. opsd declares no model — it self-distills against the live policy. See `docs/algorithms.md`. +**Algorithms** — `[orchestrator.algo] type = "grpo" | "max_rl" | "rae" | "hierarchical_grpo" | "opd" | "opsd" | "sft" | "echo"` — the type names the algorithm (credit assignment + loss routing, fused), and each type's class defaults are its vetted setting; any other key you set is your own assembly (e.g. `[orchestrator.algo.roles.user] alpha = 0.1` for echo — setting any echo role replaces the whole role table). `hierarchical_grpo` is only valid with a proposer-solver env: it compares solvers with attempts on the same proposed problem and proposers with other proposals in the group. There is no preset layer, and no config hook that points at user code — a new algorithm is a named class in the repo (subclass `Algorithm`, register it). Per-source override: `[orchestrator.train.source.algo] type = "opd"` (the source assembles its own algorithm). prime-rl only hosts the trainable policy; frozen models are inline external endpoints on the algorithm, named where the model is used — `[orchestrator.algo.teacher]` for opd (the frozen model scored against), `[orchestrator.algo.sampling.source]` for sft (the model it samples from), each with `name` + `base_url`. There is no shared `teacher` slot. opsd declares no model — it self-distills against the live policy. See `docs/algorithms.md`. **`BaseModel | None` fields** — bare flag enables defaults; nested override enables and sets: diff --git a/src/prime_rl/orchestrator/algo/__init__.py b/src/prime_rl/orchestrator/algo/__init__.py index cc5ddd60edf..745a8ebc799 100644 --- a/src/prime_rl/orchestrator/algo/__init__.py +++ b/src/prime_rl/orchestrator/algo/__init__.py @@ -5,8 +5,9 @@ turns the signal half into runtime objects (the sampling half is the env's :class:`~prime_rl.orchestrator.sampler.Sampler`): -- one module per algorithm (``grpo``, ``echo``, ``max_rl``, ``rae``, ``opd``, - ``opsd``, ``sft``) — each named class owns its scoring hooks +- one module per algorithm (``grpo``, ``echo``, ``max_rl``, ``rae``, + ``hierarchical_grpo``, ``opd``, ``opsd``, ``sft``) — each named class owns + its scoring hooks (``score_rollout`` / ``score_group``) and declares what it needs (loss component, a "teacher", ...). One instance per env, built by :func:`build_algorithm`. A new credit-assignment scheme is a new named class: @@ -31,6 +32,7 @@ from prime_rl.orchestrator.algo.base import Algorithm, connect_frozen_pool from prime_rl.orchestrator.algo.echo import EchoAlgorithm from prime_rl.orchestrator.algo.grpo import GRPOAlgorithm +from prime_rl.orchestrator.algo.hierarchical_grpo import HierarchicalGRPOAlgorithm from prime_rl.orchestrator.algo.max_rl import MaxRLAlgorithm from prime_rl.orchestrator.algo.opd import OPDAlgorithm from prime_rl.orchestrator.algo.opsd import OPSDAlgorithm @@ -50,6 +52,7 @@ "echo": EchoAlgorithm, "max_rl": MaxRLAlgorithm, "rae": RAEAlgorithm, + "hierarchical_grpo": HierarchicalGRPOAlgorithm, "opd": OPDAlgorithm, "opsd": OPSDAlgorithm, "sft": SFTDistillAlgorithm, @@ -71,6 +74,7 @@ def build_algorithm(config: AlgoConfig, policy_pool: InferencePool) -> Algorithm "Algorithm", "EchoAlgorithm", "GRPOAlgorithm", + "HierarchicalGRPOAlgorithm", "MaxRLAlgorithm", "OPDAlgorithm", "OPSDAlgorithm", diff --git a/src/prime_rl/orchestrator/algo/hierarchical_grpo.py b/src/prime_rl/orchestrator/algo/hierarchical_grpo.py new file mode 100644 index 00000000000..70171ae9831 --- /dev/null +++ b/src/prime_rl/orchestrator/algo/hierarchical_grpo.py @@ -0,0 +1,40 @@ +from __future__ import annotations + +from collections import defaultdict +from typing import TYPE_CHECKING + +from prime_rl.configs.algorithm import HierarchicalGRPOAlgoConfig +from prime_rl.orchestrator.algo.base import Algorithm + +if TYPE_CHECKING: + from prime_rl.orchestrator.types import Rollout + from prime_rl.utils.client import InferencePool + + +class HierarchicalGRPOAlgorithm(Algorithm): + """GRPO for proposer-solver envs. + + Solver rewards are compared only with other attempts on the same proposed + problem. Proposer rewards are compared with the other proposals generated + from the same source task. Keeping those comparisons separate avoids + treating different problem difficulties or different agent roles as + interchangeable. + + ``episode_agents`` lists the roles, normally ``solver``, that are compared + within one episode. Other roles are compared across the full rollout group. + A comparison group with one trace produces zero advantage.""" + + def __init__(self, config: HierarchicalGRPOAlgoConfig, policy_pool: InferencePool): + super().__init__(config, policy_pool) + self.episode_agents = set(config.episode_agents) + + async def score_group(self, group: list[Rollout]) -> None: + peers: dict[tuple[str | None, str | None], list[Rollout]] = defaultdict(list) + for rollout in group: + episode_scoped = rollout.agent_name in self.episode_agents + key = (rollout.agent_name, rollout.episode_id if episode_scoped else None) + peers[key].append(rollout) + for members in peers.values(): + baseline = sum(rollout.reward for rollout in members) / len(members) + for rollout in members: + rollout.assign_advantages(rollout.reward - baseline)