feat: vf v1 <> nano bridge

.github/workflows/cpu_tests.yaml

@@ -77,8 +77,11 @@ jobs:
           from prime_rl.utils.config import BaseConfig, find_package_resource, rgetattr, rsetattr
           from prime_rl.utils.validation import validate_shared_ckpt_config
 
+          # `verifiers` (+ its `datasets` dep) is a declared slim dep: the v1 config types
+          # (EnvConfig, Task, ...) extend verifiers.v1, which is pure-pydantic and pulls no
+          # GPU/ML deps. We still forbid the actual heavy training deps below.
           forbidden = ["torch", "transformers", "vllm", "wandb", "ring_flash_attn",
-                       "verifiers", "prime", "datasets", "liger_kernel", "loguru"]
+                       "prime", "liger_kernel", "loguru"]
           leaked = [m for m in forbidden if m in sys.modules]
           if leaked:
               raise SystemExit(f"slim install leaked heavy deps into sys.modules: {leaked}")

CHANGELOG.md

@@ -2,6 +2,12 @@
 
 Documenting **breaking** configuration changes — renamed, removed, or moved fields that require users to update existing configs.
 
+- **verifiers v1 training integration** (collection of removals/renames). The orchestrator now trains against a v1 env server returning `vf.Trace` rollouts, renderer-only, with a unified token schema.
+  - **`[[orchestrator.train.env]]` is now a v1 `vf.EnvServerConfig`**: a v1 env is `taskset = { id = "<id>" }` + `harness = { id = "default", runtime = { type = "subprocess" } }`. A bare `id = "<id>"` (optionally with `args`) runs the v0 env through the verifiers legacy bridge — old flat `id`/`args`-only configs keep working as legacy envs; v1 envs must use the `taskset`/`harness` shape.
+  - **`orchestrator.train.env` no longer defaults to a `reverse-text` env**: the default is now empty (`[]`), so configs must declare `[[orchestrator.train.env]]` explicitly.
+  - **`...env[].num_workers` → `pool`**: per-env worker sizing moves to a `pool` config (`static`/`elastic`). Back-compat: `num_workers = N` maps to `pool = { type = "static", num_workers = N }` and `"auto"` to the default elastic pool; the group-level `orchestrator.train.num_workers` / `orchestrator.eval.num_workers` are removed.
+  - **`orchestrator.renderer` is now required (MITO removed)**: it was `RendererConfig | None`, where `renderer = None` selected the MITO path; training is now renderer-only, so `renderer` is non-optional. Drop `renderer = None` / `renderer = "None"`.
+  - **`...client.timeout` / `...client.connect_timeout` removed**: the v1 rollout client manages its own timeouts. Drop these from `[orchestrator.student.client]` and `[orchestrator.teacher.client]`. (2026-06-23)
 - **`orchestrator.advantage` length-penalty overhaul + length-weighted baseline**: `length_penalty` is now a single shape — a linear penalty that subtracts `coef * pass_rate * (model completion tokens / orchestrator.seq_len)` from each reward before the baseline subtraction (`pass_rate` = group mean reward, so solved-often problems get the strongest concision pressure and never-solved groups get none), with an optional `gate_by_correctness` flag (default `false`) that restricts the penalty to correct rollouts (`reward == 1`). The previous discriminated `type = "tokens" | "turns"` efficiency-shaping variants and the per-penalty `max_length` field have been removed — configure `[orchestrator.advantage.length_penalty]` with `coef` (and optionally `gate_by_correctness`) instead; the denominator is always `orchestrator.seq_len`. Also adds `orchestrator.advantage.length_weighted_baseline` (default `false`): when enabled, the GRPO baseline is the token-length-weighted mean reward (`sum(len_i * reward_i) / sum(len_i)`) instead of the plain mean. (2026-06-03)
 - **`trainer.experimental.token_export` removed → `trainer.enable_token_export`**: `TrainerExperimentalConfig` and the `[trainer.experimental] token_export = true` setting have been removed with no migration alias. Existing trainer configs must delete the `[trainer.experimental]` block and set the new top-level trainer flag instead: `enable_token_export = true` in a trainer config file, or `trainer.enable_token_export = true` when nested under `[trainer]` in an RL config. The trainer entrypoint CLI flag is `--enable-token-export`. (2026-06-10)
 - **`inference.kv_cache_offload.cpu_bytes` removed → discriminated `type` config**: The flat `[inference.kv_cache_offload]` block with a single `cpu_bytes` field is replaced by a backend-discriminated union with composable `cpu`/`disk` tiers. Migrate native CPU offload from `[inference.kv_cache_offload]\ncpu_bytes = N` to `[inference.kv_cache_offload]\ntype = "native"` plus `[inference.kv_cache_offload.cpu]\nnum_bytes = N`. A `type = "mooncake"` backend (per-node distributed store; multi-node/SLURM only) and an optional `[inference.kv_cache_offload.disk]\npath = "..."` tier (layered behind cpu) are also available. `extra="forbid"` rejects the old `cpu_bytes` key, so existing configs must migrate. (2026-06-02)

configs/ci/integration/reverse_text_rl_sft/start.toml

@@ -19,6 +19,10 @@ training_mode = "sft"
 batch_size = 128
 group_size = 16
 
+# Teacher rolls out over plain chat-completions (no tokens); the renderer backfills them.
+[orchestrator.renderer]
+name = "qwen3"
+
 [orchestrator.train.sampling]
 max_completion_tokens = 128
 

configs/debug/training_modes/README.md

@@ -9,9 +9,8 @@ Minimal end-to-end configs for the three training modes (`rl` / `opd` / `sft`) a
 | `opd_lora.toml` | `opd` | local vLLM (`Qwen3-0.6B-Reverse-Text-RL`) | trains a LoRA adapter (rank 8) |
 | `sft.toml` | `sft` | local vLLM (`Qwen3-0.6B-Reverse-Text-RL`) | |
 | `sft_lora.toml` | `sft` | local vLLM (`Qwen3-0.6B-Reverse-Text-RL`) | trains a LoRA adapter (rank 8) |
-| `sft_external.toml` | `sft` | PI inference (`openai/gpt-5-mini`) | external OAI endpoint; no local teacher |
 
-The student inference server is auto-launched on GPU 0 at `http://localhost:8000/v1` with `gpu_memory_utilization=0.5`. The local teacher (used by everything except `rl.toml` and `sft_external.toml`) is **not** auto-launched — start it manually on GPU 1.
+The student inference server is auto-launched on GPU 0 at `http://localhost:8000/v1` with `gpu_memory_utilization=0.5`. The local teacher (used by everything except `rl.toml`) is **not** auto-launched — start it manually on GPU 1.
 
 ## Start the local teacher
 
@@ -38,10 +37,6 @@ uv run rl @ configs/debug/training_modes/opd_lora.toml
 # SFT hard distill (needs teacher on port 8001)
 uv run rl @ configs/debug/training_modes/sft.toml
 uv run rl @ configs/debug/training_modes/sft_lora.toml
-
-# SFT hard distill from openai/gpt-5-mini via PI inference
-# (requires PRIME_API_KEY + PRIME_TEAM_ID in env; no local teacher needed)
-uv run rl @ configs/debug/training_modes/sft_external.toml
 ```
 
 See [docs/training.md](../../docs/training.md#training-modes-rl--opd--sft-via-orchestrator) for what each mode does.

configs/debug/training_modes/sft.toml

@@ -20,6 +20,10 @@ training_mode = "sft"
 batch_size = 128
 group_size = 4
 
+# Teacher rolls out over plain chat-completions (no tokens); the renderer backfills them.
+[orchestrator.renderer]
+name = "qwen3"
+
 [orchestrator.train.sampling]
 max_completion_tokens = 128
 

configs/debug/training_modes/sft_lora.toml

@@ -20,6 +20,10 @@ training_mode = "sft"
 batch_size = 128
 group_size = 4
 
+# Teacher rolls out over plain chat-completions (no tokens); the renderer backfills them.
+[orchestrator.renderer]
+name = "qwen3"
+
 [orchestrator.train.sampling]
 max_completion_tokens = 128
 

configs/debug/v1/alphabet_sort.toml

@@ -0,0 +1,42 @@
+# v1 port of examples/alphabet_sort/rl.toml — identical except the env block, which loads
+# the v1 `alphabet-sort-v1` taskset (multi-turn via a colocated vf.User) instead of the v0
+# `primeintellect/alphabet-sort` env. Harness runs on the subprocess runtime.
+
+max_steps = 200
+seq_len = 2048
+
+[ckpt] # Checkpoint at the end of training
+
+[model]
+name = "Qwen/Qwen3-4B-Instruct-2507"
+
+[wandb]
+project = "alphabet-sort"
+name = "alphabet-sort"
+
+[trainer.model]
+impl = "auto"
+
+[trainer.model.ac]
+freq = 1
+
+[trainer.model.lora]
+rank = 32
+alpha = 64
+
+[trainer.optim]
+lr = 1e-5
+
+[orchestrator]
+batch_size = 512
+group_size = 8
+
+[orchestrator.train.sampling]
+max_completion_tokens = 768
+
+[[orchestrator.train.env]]
+name = "alphabet-sort"
+taskset = { id = "alphabet-sort-v1", min_turns = 3, max_turns = 5, power_per_turn = false }
+harness = { id = "default", runtime = { type = "subprocess" } }
+
+[inference] # Default inference config

configs/debug/v1/general_agent.toml

@@ -0,0 +1,55 @@
+# Debug v1 RL run: the v1 `general-agent-v1` taskset (GeneralAgentSolverTaskset) driven by the
+# default harness, with the agent and its tool server colocated in a modal sandbox. Copied from
+# configs/general_agent/rl_qwen3_4b.toml and rewired for v1 (taskset + harness + renderer; the
+# v0 `id = "general-agent-solver-rlm"` env becomes the v1 taskset + a separate harness).
+
+max_steps = 200
+seq_len = 32768
+
+[deployment]
+num_train_gpus = 4
+num_infer_gpus = 4
+
+[wandb]
+project = "general-agent-debug"
+name = "qwen3-4b-v1"
+
+[ckpt]
+interval = 100
+keep_last = 1
+
+[model]
+name = "Qwen/Qwen3-4B-Instruct-2507"
+
+[trainer]
+
+[trainer.model]
+cp = 2
+
+[trainer.model.ac]
+freq = 1
+
+[trainer.model.compile]
+
+[orchestrator]
+training_mode = "rl"
+batch_size = 256
+rollouts_per_example = 8
+max_off_policy_steps = 32
+
+[orchestrator.renderer]
+name = "qwen3"
+
+[[orchestrator.train.env]]
+# v1 general-agent solver taskset; the tool server is colocated in the harness's modal sandbox.
+taskset = { id = "general-agent-v1", tools = { colocated = true } }
+harness = { id = "default", runtime = { type = "modal" } }
+
+[inference]
+gpu_memory_utilization = 0.85
+
+[inference.model]
+max_model_len = 32768
+
+[inference.parallel]
+dp = 4

configs/debug/v1/gsm8k.toml

@@ -0,0 +1,42 @@
+# GSM8K (v1) — the v1 port of configs/gsm8k/rl.toml. Trains Qwen3-0.6B on GSM8K through the
+# v1 env server: the orchestrator spawns a v1 EnvServer for the gsm8k-v1 taskset and trains on
+# the returned Traces. Runs on 2 GPUs (1 inference, 1 training):
+#   uv run inference --model.name PrimeIntellect/Qwen3-0.6B   # Inference pane (GPU 0)
+#   uv run rl @ configs/debug/v1/gsm8k.toml --ckpt            # Trainer pane (GPU 1)
+
+max_steps = 100
+seq_len = 2048
+
+[wandb]
+project = "gsm8k-debug"
+name = "gsm8k-v1"
+
+[model]
+name = "Qwen/Qwen3-0.6B"
+
+[orchestrator]
+training_mode = "rl"
+batch_size = 512
+group_size = 16
+
+[orchestrator.train.sampling]
+max_completion_tokens = 2048
+
+[[orchestrator.train.env]]
+taskset = { id = "gsm8k-v1", split = "train" }
+harness = { id = "default", runtime = { type = "subprocess" } }
+
+[orchestrator.eval]
+interval = 20
+num_examples = 256
+
+[orchestrator.eval.sampling]
+max_completion_tokens = 2048
+
+[[orchestrator.eval.env]]
+taskset = { id = "gsm8k-v1", split = "test" }
+harness = { id = "default", runtime = { type = "subprocess" } }
+
+[trainer] # Default trainer config
+
+[inference] # Default inference config

configs/debug/v1/hendrycks_sanity.toml

@@ -0,0 +1,55 @@
+# v1 analog of examples/hendrycks_sanity/rl.toml — identical config with only
+# the env sections swapped to v1 taskset/harness syntax (math-env taskset,
+# default harness, modal runtime). Submits to slurm by default; pass --no-slurm
+# to run locally.
+
+output_dir = "/beegfs/mika/hendrycks-sanity-v1"
+max_steps = 200
+
+[wandb]
+project = "hendrycks-sanity"
+name = "v1"
+
+[deployment]
+num_train_gpus = 4
+num_infer_gpus = 4
+
+[slurm]
+job_name = "hendrycks-modal"
+partition = "all"
+
+[model]
+name = "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B"
+
+[orchestrator]
+batch_size = 512
+group_size = 8
+seq_len = 8192
+max_inflight_rollouts = 256
+
+[[orchestrator.train.env]]
+name = "hendrycks-math"
+taskset = { id = "math-env-v1", dataset_name = "mikasenghaas/Sanity-Test-R1D-1.5B", dataset_subset = "default" }
+harness = { id = "default", runtime = { type = "modal" } }
+timeout = { scoring = 300 }
+
+[orchestrator.eval]
+interval = 50
+
+[[orchestrator.eval.env]]
+name = "aime2024"
+taskset = { id = "aime24-v1" }
+harness = { id = "default", runtime = { type = "modal" } }
+timeout = { scoring = 300 }
+group_size = 32
+
+[trainer.model]
+seq_len = 16384
+
+[inference.model]
+max_model_len = 8192
+
+# Model not in MODEL_RENDERER_MAP — opt into DefaultRenderer (apply_chat_template).
+[orchestrator.renderer]
+name = "default"
+reasoning_parser = "think"