feat: first-class sft+opd training modes

CHANGELOG.md

@@ -2,6 +2,13 @@
 
 Documenting **breaking** configuration changes — renamed, removed, or moved fields that require users to update existing configs.
 
+- **First-class `training_mode` + batch-driven loss dispatch** (collection of removals/renames). Loss selection is now driven by `TrainingSample.training_mode` (`rl` / `opd` / `sft`), set under `[orchestrator]`. The trainer is mode-agnostic and dispatches per batch.
+  - **`orchestrator.use_sft_loss` removed**: replaced by `[orchestrator] training_mode = "sft"`. The orchestrator stamps each sample's `training_mode` and the trainer dispatches to `sft_loss_fn` per batch.
+  - **`orchestrator.teacher_rollout_model` removed (no alias)**: configs must rename to `[orchestrator.teacher]` (`teacher_model` is also accepted as a back-compat alias for `teacher`). Used in both opd and sft modes; the role is determined by `training_mode`.
+  - **`orchestrator.model` → `orchestrator.student`**: renamed for symmetry with `orchestrator.teacher`. Legacy `[orchestrator.model]` and a flat `[orchestrator.model.lora]`-style layout are aliased via a back-compat shim, so most existing configs still parse. New code should use `[orchestrator.student.*]`.
+  - **`trainer.loss.type = "sft"` removed (`SFTLossConfig` deleted)**: zero-field discriminator that is now implied by `training_mode = "sft"`. `trainer.loss` collapsed to a `DefaultLossConfig | CustomLossConfig` union. Existing configs setting `[trainer.loss] type = "sft"` must remove that line; the orchestrator-level `training_mode = "sft"` is enough.
+  - **`trainer.loss.teacher_tau` removed from `DefaultLossConfig`**: the teacher-KL term is now exclusive to `opd_loss_fn` (which inlines `teacher_tau = 1.0`). `default_loss_fn` only handles reward-driven RL and no longer accepts a `teacher_tau`. Configs that set `teacher_tau` on `[trainer.loss]` will fail validation — switch to `[orchestrator] training_mode = "opd"` (which uses `opd_loss_fn` with the value baked in).
+  - **`trainer.loss` only applies to rl-mode batches**: opd and sft don't read `trainer.loss`. `opd_loss_fn`'s knobs (dppo_mask_*, kl_tau) are inlined as literals; tweak them by editing the function. `CustomLossConfig` overrides the rl-mode loss only — opd and sft always use their dedicated fns. (2026-05-19)
 - **`orchestrator.use_token_client` removed**: The server-tokenized TITO path has been deprecated end-to-end. The orchestrator-side config flag (`use_token_client`), the verifiers client_type (`openai_chat_completions_token`), and the inference server's `/v1/chat/completions/tokens` route (along with the `OpenAIServingChatWithTokens` wrapper) are all gone. The orchestrator now picks between renderer-backed TITO (`use_renderer = true`, default) and MITO (`use_renderer = false`, fallback). Existing configs with `use_token_client = true` must migrate to `use_renderer = true` (or `use_renderer = false` for MITO); configs with `use_token_client = false` can simply drop the field. (2026-05-19)
 - **`orchestrator.advantage.length_penalty` → discriminated sub-config**: The scalar `length_penalty: Literal["tokens","turns"] | None` is replaced by a `LengthPenaltyConfig | None` discriminated on `type`. Token shaping now takes weighted completion + tool-response token costs. Migration: `length_penalty = "tokens"` becomes `[orchestrator.advantage.length_penalty]\ntype = "tokens"` (default weights `completion_weight = 1.0`, `tool_response_weight = 1.0` — total context). `length_penalty = "turns"` becomes `[orchestrator.advantage.length_penalty]\ntype = "turns"`. (2026-05-06)
 - **`orchestrator.advantage.length_shaping` → `orchestrator.advantage.length_penalty`**: The boolean `length_shaping` flag has been replaced by `length_penalty: Literal["tokens", "turns"] | None` (default: `None`). `length_shaping = true` becomes `length_penalty = "tokens"`; `length_shaping = false` becomes `length_penalty = None`. The new `"turns"` option applies the same correctness-gated efficiency shaping using trajectory turn count instead of completion-token count. (2026-05-01)

configs/debug/training_modes/README.md

@@ -0,0 +1,47 @@
+# Training Mode — Debug Configs
+
+Minimal end-to-end configs for the three training modes (`rl` / `opd` / `sft`) against the `reverse-text` env, using `PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT` as the student.
+
+| Config | Mode | Teacher | Notes |
+|---|---|---|---|
+| `rl.toml` | `rl` | none | |
+| `opd.toml` | `opd` | local vLLM (`Qwen3-0.6B-Reverse-Text-RL`) | |
+| `opd_lora.toml` | `opd` | local vLLM (`Qwen3-0.6B-Reverse-Text-RL`) | trains a LoRA adapter (rank 8) |
+| `sft.toml` | `sft` | local vLLM (`Qwen3-0.6B-Reverse-Text-RL`) | |
+| `sft_lora.toml` | `sft` | local vLLM (`Qwen3-0.6B-Reverse-Text-RL`) | trains a LoRA adapter (rank 8) |
+| `sft_external.toml` | `sft` | PI inference (`openai/gpt-5-mini`) | external OAI endpoint; no local teacher |
+
+The student inference server is auto-launched on GPU 0 at `http://localhost:8000/v1` with `gpu_memory_utilization=0.5`. The local teacher (used by everything except `rl.toml` and `sft_external.toml`) is **not** auto-launched — start it manually on GPU 1.
+
+## Start the local teacher
+
+Needed for `opd*.toml` and `sft.toml` / `sft_lora.toml`:
+
+```bash
+CUDA_VISIBLE_DEVICES=1 uv run inference \
+  --model.name PrimeIntellect/Qwen3-0.6B-Reverse-Text-RL \
+  --server.port 8001 \
+  --gpu-memory-utilization 0.5 \
+  --model.enforce-eager
+```
+
+## Run the debug configs
+
+```bash
+# RL (no teacher)
+uv run rl @ configs/debug/training_modes/rl.toml
+
+# OPD (needs teacher on port 8001)
+uv run rl @ configs/debug/training_modes/opd.toml
+uv run rl @ configs/debug/training_modes/opd_lora.toml
+
+# SFT hard distill (needs teacher on port 8001)
+uv run rl @ configs/debug/training_modes/sft.toml
+uv run rl @ configs/debug/training_modes/sft_lora.toml
+
+# SFT hard distill from openai/gpt-5-mini via PI inference
+# (requires PRIME_API_KEY + PRIME_TEAM_ID in env; no local teacher needed)
+uv run rl @ configs/debug/training_modes/sft_external.toml
+```
+
+See [docs/training_modes.md](../../docs/training_modes.md) for what each mode does.

configs/debug/training_modes/opd.toml

@@ -0,0 +1,54 @@
+# Start the teacher inference server first (on a separate GPU):
+#   CUDA_VISIBLE_DEVICES=1 uv run inference \
+#     --model.name PrimeIntellect/Qwen3-0.6B-Reverse-Text-RL \
+#     --server.port 8001 --gpu-memory-utilization 0.5 --model.enforce-eager
+# Then:
+#   uv run rl @ configs/debug/training_modes/opd.toml
+
+max_steps = 20
+seq_len = 2048
+
+[model]
+name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"
+
+[wandb]
+project = "reverse-text-debug"
+name = "debug-opd"
+
+[orchestrator]
+training_mode = "opd"
+batch_size = 128
+rollouts_per_example = 16
+
+[orchestrator.renderer]
+name = "qwen3"
+
+[orchestrator.train.sampling]
+max_completion_tokens = 128
+
+[[orchestrator.train.env]]
+id = "reverse-text"
+
+[orchestrator.eval]
+interval = 1
+num_examples = 128
+
+[orchestrator.eval.sampling]
+max_completion_tokens = 128
+
+[[orchestrator.eval.env]]
+id = "reverse-text"
+
+[orchestrator.teacher.model]
+name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-RL"
+
+[orchestrator.teacher.client]
+base_url = ["http://localhost:8001/v1"]
+
+[trainer.optim]
+lr = 3e-6
+
+[ckpt]
+
+[inference]
+gpu_memory_utilization = 0.5

configs/debug/training_modes/opd_lora.toml

@@ -0,0 +1,60 @@
+# Start the teacher inference server first (on a separate GPU):
+#   CUDA_VISIBLE_DEVICES=1 uv run inference \
+#     --model.name PrimeIntellect/Qwen3-0.6B-Reverse-Text-RL \
+#     --server.port 8001 --gpu-memory-utilization 0.5 --model.enforce-eager
+# Then:
+#   uv run rl @ configs/debug/training_modes/opd_lora.toml
+
+max_steps = 20
+seq_len = 2048
+
+[model]
+name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"
+
+[wandb]
+project = "reverse-text-debug"
+name = "debug-opd-lora"
+
+[orchestrator]
+training_mode = "opd"
+batch_size = 128
+rollouts_per_example = 16
+
+[orchestrator.renderer]
+name = "qwen3"
+
+[orchestrator.train.sampling]
+max_completion_tokens = 128
+
+[[orchestrator.train.env]]
+id = "reverse-text"
+
+[orchestrator.eval]
+interval = 1
+num_examples = 128
+
+[orchestrator.eval.sampling]
+max_completion_tokens = 128
+
+[[orchestrator.eval.env]]
+id = "reverse-text"
+
+[orchestrator.teacher.model]
+name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-RL"
+
+[orchestrator.teacher.client]
+base_url = ["http://localhost:8001/v1"]
+
+[trainer.optim]
+lr = 1e-4
+
+[trainer.model.lora]
+rank = 8
+
+[trainer.ckpt.weights]
+save_adapter_separately = true
+
+[ckpt]
+
+[inference]
+gpu_memory_utilization = 0.5

configs/debug/training_modes/rl.toml

@@ -0,0 +1,41 @@
+max_steps = 20
+seq_len = 2048
+
+[model]
+name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"
+
+[wandb]
+project = "reverse-text-debug"
+name = "debug-rl"
+
+[orchestrator]
+training_mode = "rl"
+batch_size = 128
+rollouts_per_example = 16
+
+[orchestrator.renderer]
+name = "qwen3"
+
+[orchestrator.train.sampling]
+max_completion_tokens = 128
+
+[[orchestrator.train.env]]
+id = "reverse-text"
+
+[orchestrator.eval]
+interval = 1
+num_examples = 128
+
+[orchestrator.eval.sampling]
+max_completion_tokens = 128
+
+[[orchestrator.eval.env]]
+id = "reverse-text"
+
+[trainer.optim]
+lr = 3e-6
+
+[ckpt]
+
+[inference]
+gpu_memory_utilization = 0.5

configs/debug/training_modes/sft.toml

@@ -0,0 +1,51 @@
+# Start the teacher inference server first (on a separate GPU):
+#   CUDA_VISIBLE_DEVICES=1 uv run inference \
+#     --model.name PrimeIntellect/Qwen3-0.6B-Reverse-Text-RL \
+#     --server.port 8001 --gpu-memory-utilization 0.5 --model.enforce-eager
+# Then:
+#   uv run rl @ configs/debug/training_modes/sft.toml
+
+max_steps = 20
+seq_len = 2048
+
+[model]
+name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"
+
+[wandb]
+project = "reverse-text-debug"
+name = "debug-sft"
+
+[orchestrator]
+training_mode = "sft"
+batch_size = 128
+rollouts_per_example = 4
+
+[orchestrator.train.sampling]
+max_completion_tokens = 128
+
+[[orchestrator.train.env]]
+id = "reverse-text"
+
+[orchestrator.eval]
+interval = 1
+num_examples = 128
+
+[orchestrator.eval.sampling]
+max_completion_tokens = 128
+
+[[orchestrator.eval.env]]
+id = "reverse-text"
+
+[orchestrator.teacher.model]
+name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-RL"
+
+[orchestrator.teacher.client]
+base_url = ["http://localhost:8001/v1"]
+
+[trainer.optim]
+lr = 3e-6
+
+[ckpt]
+
+[inference]
+gpu_memory_utilization = 0.5

configs/debug/training_modes/sft_external.toml

@@ -0,0 +1,55 @@
+# SFT from openai/gpt-5-mini via PI inference.
+# Requires PRIME_API_KEY + PRIME_TEAM_ID in the environment.
+#
+# Run with:
+#   uv run rl @ configs/debug/training_modes/sft_external.toml
+
+max_steps = 20
+seq_len = 2048
+
+[model]
+name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"
+
+[wandb]
+project = "reverse-text-debug"
+name = "debug-sft-external"
+
+[orchestrator]
+training_mode = "sft"
+batch_size = 128
+rollouts_per_example = 4
+
+[orchestrator.train.sampling]
+max_completion_tokens = 2048
+extra_body = { reasoning_effort = "minimal" }
+
+[[orchestrator.train.env]]
+id = "reverse-text"
+
+[orchestrator.eval]
+interval = 1
+num_examples = 128
+
+[orchestrator.eval.sampling]
+max_completion_tokens = 128
+
+[[orchestrator.eval.env]]
+id = "reverse-text"
+
+[orchestrator.teacher.model]
+name = "openai/gpt-5-mini"
+
+[orchestrator.teacher.client]
+base_url = ["https://api.pinference.ai/api/v1"]
+api_key_var = "PRIME_API_KEY"
+
+[orchestrator.teacher.client.headers_from_env]
+X-Prime-Team-ID = "PRIME_TEAM_ID"
+
+[trainer.optim]
+lr = 3e-6
+
+[ckpt]
+
+[inference]
+gpu_memory_utilization = 0.5

configs/debug/training_modes/sft_lora.toml

@@ -0,0 +1,57 @@
+# Start the teacher inference server first (on a separate GPU):
+#   CUDA_VISIBLE_DEVICES=1 uv run inference \
+#     --model.name PrimeIntellect/Qwen3-0.6B-Reverse-Text-RL \
+#     --server.port 8001 --gpu-memory-utilization 0.5 --model.enforce-eager
+# Then:
+#   uv run rl @ configs/debug/training_modes/sft_lora.toml
+
+max_steps = 20
+seq_len = 2048
+
+[model]
+name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"
+
+[wandb]
+project = "reverse-text-debug"
+name = "debug-sft-lora"
+
+[orchestrator]
+training_mode = "sft"
+batch_size = 128
+rollouts_per_example = 4
+
+[orchestrator.train.sampling]
+max_completion_tokens = 128
+
+[[orchestrator.train.env]]
+id = "reverse-text"
+
+[orchestrator.eval]
+interval = 1
+num_examples = 128
+
+[orchestrator.eval.sampling]
+max_completion_tokens = 128
+
+[[orchestrator.eval.env]]
+id = "reverse-text"
+
+[orchestrator.teacher.model]
+name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-RL"
+
+[orchestrator.teacher.client]
+base_url = ["http://localhost:8001/v1"]
+
+[trainer.optim]
+lr = 1e-4
+
+[trainer.model.lora]
+rank = 8
+
+[trainer.ckpt.weights]
+save_adapter_separately = true
+
+[ckpt]
+
+[inference]
+gpu_memory_utilization = 0.5

docs/mint.json

@@ -7,6 +7,7 @@
                 "index",
                 "entrypoints",
                 "configs",
+                "training_modes",
                 "environments",
                 "async",
                 "logging",