feat: vf v1 <> nano bridge

[mikasenghaas]

Companion PR to PrimeIntellect-ai/verifiers#1576 for verifiers v1 training integration.

Summary

Integrates verifiers v1 into prime-rl. Training rolls out against a v1 env server that returns vf.Trace rollouts, with renderer-only (token-in/out) training and a unified token_ids sample schema.

• v1 env server. The orchestrator spawns a vf EnvServer per environment (subprocess runtime, ZMQ) and drives it purely by task index — it never loads the env. v0 envs run through the verifiers legacy bridge; native v1 tasksets serve directly; both return vf.Trace over one protocol.
• vf.Trace rollouts. Rollout is a vf.Trace[TaskT]; prime-rl's orchestration metadata lives on it as exclude=True fields, so a dumped rollout is a plain trace. The wire trace is re-typed into Rollout via model_construct (no serialize round-trip of the node tensors).
• Unified token_ids schema. Rollouts are tokenized renderer-side into one TrainingSample / MicroBatch representation (replaces the prompt/completion split); main's bin_cost packing and MicroBatch.sequence_lengths are grafted onto it.
• Renderer-only training. RL/OPD student and SFT teacher all roll out through the renderer (token) client; SFT re-renders to backfill tokens.
• Configs consume vf v1. EnvConfig extends vf.EnvServerConfig (taskset / harness / runtime / pool); EvalClientConfig / TrainClientConfig replace the prior client config.
• verifiers submodule bumped to the v1 line; main (v0.6.0) merged in.

Verification

Scaleswe v1 (W&B)

Hendrycks v0/v1 (W&B)

General Agent v1 on Modal (W&B)

Wiki Search v0 (W&B)

Alphabet Sort v0 (W&B)

Breaking

• ClientConfig.timeout / ClientConfig.connect_timeout removed. They were no longer wired into rollout clients (the v1 client manages its own timeouts). Remove them from configs.
• Custom advantage functions receive a vf.Trace, not a raw dict. CustomAdvantageConfig functions must use attribute access (r.reward, r.completion_len, r.nodes) instead of dict keys (r["reward"]).
• Renderer-only training. SFT distillation from an external chat-completions-only teacher (a provider endpoint that returns no token ids, e.g. openai/gpt-5-mini) is no longer supported — SFT teachers must be a local-vLLM or v0-bridge env that returns tokens. Teacher-less SFT now errors; use the standalone sft entrypoint.
• Env config is v1. [[orchestrator.train.env]] takes taskset = { id = … } + harness = { id = …, runtime = { type = … } } (extends vf.EnvServerConfig); a bare v0 env id runs through the legacy bridge.
• Orchestrator-side token-export W&B aggregation dropped. trainer/mismatch_kl and trainer/entropy are no longer folded into W&B (the trainer still writes the token-export JSONL when enable_token_export = True).
• Namespaced legacy env auto-install dropped. Startup no longer runs prime env install for namespaced legacy env IDs (e.g. primeintellect/wordle); the v1 env server (serve_env) loads envs directly. Runs that relied on automatic install — including the Wordle legacy-bridge config — fail unless the env is installed manually (uv run prime env install <id>) beforehand. The env_install_prerelease config flags and the install_env helper are removed.

---

Note

High Risk
Large orchestrator rewrite (env serving, rollout schema, training sample construction) touches the core RL loop and breaks multiple config contracts; misconfiguration or bridge edge cases could silently drop or mis-tokenize rollouts.

Overview
Integrates verifiers v1 so the orchestrator trains on vf.Trace rollouts from a spawned or external v1 env server, driven by task index instead of loading envs in-process. v0 envs still work via the legacy bridge; native configs use taskset + harness + pool on EnvConfig (extends vf.EnvServerConfig).

Rollout and training path: Rollout is now a typed vf.Trace with orchestration metadata on exclude=True fields (replacing TrainRollout/EvalRollout + raw RolloutOutput dicts). Dispatcher, sinks, filters, and advantages use trace attributes (reward, nodes, completion_len, has_error). trajectories.py is rewritten: trace_to_samples builds one TrainingSample per branch from flat token_ids/mask/logprobs (no MITO backfill or trajectory interleaving). Renderer is required for all modes; SFT teachers use the renderer client for tokens. Startup prime env install and orchestrator token-export W&B drain are removed.

Config and deps: Train/eval env lists default empty (no implicit reverse-text); num_workers migrates to pool; client timeout/connect_timeout dropped; slim prime-rl-configs may depend on verifiers v1 types. Adds v1 debug configs and bumps the verifiers submodule; removes external SFT example config.

^{Reviewed by Cursor Bugbot for commit 5056e27. Bugbot is set up for automated code reviews on this repo. Configure here.}

[hallerite]

lgtm!

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 19c88f5. Configure here.}