[faresobeid]

sft's teacher replaces algo.sampling.source, and opsd renders hints with the policy's renderer.

Changes

• algo.sampling is removed (SamplingConfig, BaseAlgoConfig.sampling and its source/loss validator). sampling.source could only be "policy" for 8 of the 9 algorithms and was only meaningful for sft. Its name also collided with train.sampling (temperature, top_p, …) on the same parent.
• SFTAlgoConfig.teacher: FrozenModelConfig is required, the same type as opd.teacher, so both distillation teachers are configured as [orchestrator.train.algo.teacher].
• algo.rollout_model returns "policy", or the teacher for sft. It replaces every algo.sampling.source read in the orchestrator, the config validators (including the replay prl_replay setup) and env setup. The rl entrypoint lists the frozen teacher endpoints of opd and sft once.
• "policy" cannot be a teacher, so "sft needs a frozen source" and "rl/ref_kl must sample from the policy" are now enforced by the types instead of validators.
• opsd.renderer is removed. Its docstring asked users to set it by hand to match the policy renderer. Hints are now rendered with the policy client's renderer config (orchestrator.renderer) and the same load_tokenizer(model.name) tokenizer as before.

Breaking: [orchestrator.train.algo.sampling.source] → [orchestrator.train.algo.teacher]; opsd.renderer is gone. The three shipped sft configs, docs/algorithms.md, docs/training.md, configs/debug/algo/README.md and skills/configs/SKILL.md are updated.

Validation

• After merging main (1810d5b): pytest tests/unit/orchestrator tests/unit/test_configs.py tests/unit/inference/test_dynamo.py -m "not gpu": 401 passed. test_load_configs loads every TOML in configs/ and examples/, so the sft configs are covered.
• A renderer-type policy InferenceClient exposes train_client.renderer (the config opsd now reads).
• ruff check, ruff format --check: clean. No dependency changes.

Overlaps

Open PRs #3699 (PPO/SAO), #3685 and #3640 (CriPO-S) build on BaseAlgoConfig. After this lands they should drop sampling (and the CriPO-S per-algorithm renderer) and use rollout_model and the policy renderer.

---

Note

Medium Risk
Breaking TOML/API change for SFT configs and OPSD renderer overrides; behavior is intended to stay equivalent but mis-migrated configs will fail at validation or change hint tokenization if users relied on a custom opsd renderer.

Overview
Breaking config refactor for how algorithms declare which model generates train rollouts and how SFT/OPSD wire frozen models and renderers.

algo.sampling is removed (SamplingConfig, nested sampling.source, and cross-algorithm validators). Rollout sourcing is expressed via a rollout_model property on each algo config: "policy" for all algorithms except sft, which returns its required teacher. Orchestrator validators, env setup, sampling replay (prl_replay), and the RL entrypoint’s frozen-endpoint warnings now key off rollout_model instead of sampling.source.

SFT now requires [orchestrator.train.algo.teacher] with the same FrozenModelConfig type as OPD (replacing [orchestrator.train.algo.sampling.source]). Type requirements replace runtime checks that forbade policy-as-teacher or frozen sampling for RL/ref_kl algos.

OPSD drops opsd.renderer; demonstration hints use the policy inference client’s renderer config (orchestrator.renderer) and tokenizer in setup().

Shipped SFT TOMLs, algorithm/training docs, debug README, and config skill text are updated to the new teacher path.

^{Reviewed by Cursor Bugbot for commit b7256ff. Bugbot is set up for automated code reviews on this repo. Configure here.}