[mikasenghaas]

Summary

• Give every train and eval source a select block (EnvConfig.select, a vf.SelectConfig): include/exclude by task idx, ids, keys or names, then shuffle, skip and limit. The orchestrator loads each source's tasks with vf.load_taskset(...).select(source.select). prime-rl no longer shuffles or slices tasksets itself, so it picks the same tasks as vf-eval for the same selection.

  [[orchestrator.train.source]]
  env.taskset.id = "gsm8k"
  select.exclude.idx = [":100"]   # hold out the first 100 tasks
  
  [[orchestrator.eval.source]]
  env.taskset.id = "gsm8k"
  select.include.idx = [":100"]

• Every source group ([orchestrator.train], [orchestrator.eval], SFT [eval], the top level of uv run eval) takes a shared select block. The same before-validator that layers the group env block layers it under each source, field by field, and a field that the source sets wins. A source's include or exclude replaces the group's whole block. A source select that does not validate keeps its own block, so its error surfaces. For example, this evaluates 128 tasks of each taskset:

  [orchestrator.eval.select]
  limit = 128

• Only uv run eval has the -n/-s short flags (select.limit/select.shuffle), through vf.SelectCLIConfig. The rl, orchestrator and sft CLIs have several select blocks, so they get no short flags.
• The dashboard derives the expected episode count from select.limit, per source and for vf-shaped configs.
• Migrate the configs, examples and docs from num_examples to select.limit. Group-level blocks use [... .select] tables.
• Update the eval docs and skill: resume no longer checks the config (feat(eval): allow any config override on resume #3690), and a resume matches landed episodes to the new select by task.key.
• Update the configs skill ([select], -s). Update the eval skill: the select shorthands, and common patterns (a random subset, tasks by position or name, dropping broken tasks, contiguous and random train/test splits via include/exclude and skip).
• Pin deps/verifiers to its main, which merged feat: add taskset selection views and select config verifiers#2695 (Taskset.select, vf.SelectConfig). Raise the verifiers minimum to 0.3.2.dev152, its first release, in both pyproject.toml files, so the slim prime-rl-configs install gets it.

Companion PRs: PrimeIntellect-ai/verifiers#2695 (merged), PrimeIntellect-ai/prime-envs#852 (independent; compile_statements stops calling head()).

Breaking

• EnvConfig.shuffle is removed. Use select.shuffle.
• EvalSourceConfig.num_examples and the group-level num_examples are removed. Use select.limit on the source or on the eval block.
• The shuffle seed changes from 42 to verifiers' default 0 (select.seed). Existing shuffle = true configs pick other tasks, so eval numbers before and after this change are not comparable.
• Default task keys change for tasksets that never set idx (see verifiers#2695). An interrupted eval of such a taskset cannot resume across this change.

Verification

• The slim-install CI check, run locally: the prime-rl-configs wheel installed outside the workspace resolves verifiers 0.3.2.dev152, and the workflow's import script passes.
• uv run pytest tests/unit/test_configs.py tests/unit/eval tests/unit/orchestrator tests/unit/test_dashboard_timeline.py: 296 passed (2 vLLM-only tests deselected; vllm is not installed on the test box). test_load_configs loads every migrated TOML.
• Real configs and CLI calls against a local mock OpenAI server (no inference spend). The mock counted the model calls, and the selected task idx matched an independent oracle:
  • uv run eval @ configs/debug/eval/single-turn.toml: tasks 0–7 × 2, 16 calls.
  • uv run eval @ configs/debug/eval/resume.toml: first run 16 calls. --resume --select.limit 10 restored 16 episodes and ran only tasks 8–9 (4 calls). --resume --select.include.idx 0:5 kept the landed episodes of tasks 0–4 and made no model calls.
  • uv run eval gsm8k -n 4 -s --select.exclude.idx '[0]': the oracle's 4 tasks, 4 calls.
  • A two-source TOML that splits gsm8k into [:50] and the rest under a group select default: both sources match the oracle, 5 calls.
• Config checks: train and eval group select defaults reach every source, a source's own fields win, and the resolved OrchestratorConfig re-validates to the same values. uv run rl -h, orchestrator -h and sft -h list no -n/-s; uv run eval -h does.
• uv run rl --dry-run on configs/debug/algo/grpo.toml, examples/basic/reverse-text/rl.toml and configs/debug/multi-env/rl.toml, and uv run sft --dry-run on examples/basic/reverse-text/sft.toml: group eval limits (128, 64, 32) resolve onto each source, a per-source select.limit = 256 is kept, and train sources default to all tasks.

🤖 Generated with Claude Code

---

Note

Medium Risk
Breaking config renames and verifiers-driven selection/shuffle/task-key semantics affect every train/eval run and eval resume comparability across the upgrade, though runtime logic is largely delegated to verifiers.

Overview
Replaces num_examples and per-source shuffle with verifiers' select block on every train/eval source (and group-level [select] / [orchestrator.eval.select] / [eval.select]). select.limit caps how many tasks run; include/exclude, shuffle, skip, and seed follow verifiers' ordering so prime-rl matches vf-eval for the same selection.

The orchestrator now loads tasks with vf.load_taskset(...).select(source.select) instead of client-side shuffle (seed 42) and islice. Group config merging layers shared select onto each source like env. uv run eval maps -n/-s to select.limit/select.shuffle; eval resume no longer validates config against the interrupted run—landed episodes are matched by task.key to the new select.

All example/debug/CI TOMLs, docs, and skills are migrated. verifiers minimum is raised to 0.3.2.dev152. The dashboard expects select.limit when estimating episode counts.

Breaking: rename configs from num_examples → select.limit, shuffle → select.shuffle; default shuffle seed and task keys may differ from pre-change runs (resume/comparability implications).

^{Reviewed by Cursor Bugbot for commit f30798a. Bugbot is set up for automated code reviews on this repo. Configure here.}