[mikasenghaas]

Summary

• Add lazy, chainable selection views to Taskset: include and exclude (by idx, ids, keys or names), shuffle(seed), skip(n) and take(n). Each view is a shallow copy, so the base taskset does not change.

  taskset.include(idx=["0:100"]).exclude(names=["broken"]).shuffle(seed=0).take(5)

• Add SelectConfig and TaskMatchConfig (verifiers/v1/configs/select.py). Taskset.select(config) chains the views in a fixed order: include, exclude, shuffle, skip, limit. idx takes ints and Python slices start:stop:step, each part optional ("100:", ":50", "::2", "0:10,17"); negative positions are not supported. Within include or exclude, a task matches when any list names it.
• Give the eval, debug, validate and GEPA configs a top-level select block. The four entrypoints select only through Taskset.select, so the copied "shuffle, then head" blocks are gone.
• Keep SelectConfig free of aliases, so that a config with several select blocks (prime-rl's group defaults and sources) has no ambiguous flags. SelectCLIConfig adds the short flags for the one select block of an entrypoint: -n sets select.limit and -s sets select.shuffle.
• Make idx a framework field: iteration sets TaskData.idx to the task's position in load() and overwrites any value that the taskset sets. Iteration also applies the config system_prompt before the views, so a keys match compares the keys that traces record. The built-in tasksets, the example environments, the vf-init template and the docs no longer pass idx.
• Add an optional durable TaskData.id.
• Add Taskset.bounded. take bounds a view, and so does an include that names only closed idx ranges. shuffle, an include by ids, keys or names (which could wait forever for a match), and an exclude of an open idx range (which drops every later task) raise on a view that is not bounded. select applies limit after the shuffle, so there only closed include.idx ranges can bound it; the error says so. An idx-only include stops reading load() after its last position.
• Log a warning when an include or exclude entry matches no task (usually a typo).
• Document the views and the select block in docs/v1/tasksets.md#selecting-tasks.

Companion PRs:

• prime-rl: feat: select taskset tasks per train and eval source prime-rl#3703 (consumes select per train/eval source)
• prime-envs: fix: slice the taskset instead of Taskset.head() prime-envs#852 (one script stops calling head())

Breaking

• EvalConfig, DebugConfig and ValidateConfig drop num_tasks and shuffle. Use select.limit and select.shuffle (-n and -s still work). In TOML, change a top-level num_tasks = 5 to select.limit = 5.
• GEPAConfig drops shuffle, and GEPA no longer shuffles by default. Pass -s to keep the old train/val split behaviour.
• Taskset.head(n) becomes take(n). Taskset.view and the SEED constant are removed.
• Tasksets that never set idx get new default task keys, because idx is part of the content hash. An interrupted eval of such a taskset cannot resume across this change.

Verification

• uv run pytest tests -m "not e2e": 113 passed. ruff check and ruff format --check pass. tests/v1/test_taskset.py has one test per behaviour: idx assignment, each view, lazy reads, the bounds that allow shuffle, idx parsing, keys matches under a system_prompt override, the SelectConfig defaults and SelectCLIConfig aliases, and the fixed order of select().
• vf-validate on a dummy 20-task taskset and a dummy infinite taskset: 27 CLI combinations of --select.*, -n, -s and a TOML [select] block matched an independent oracle. The combinations included every idx form, the union inside include, exclude.idx, and the infinite-taskset errors.
• Real configs against a local mock OpenAI server (no inference spend). The mock counted the model calls:
  • vf-eval @ configs/gsm8k.toml: tasks 0–4 × 3 rollouts, 15 calls. With -s: the oracle's shuffled 5 of 1319.
  • With --select.include.idx 100:103 --select.exclude.idx 101: tasks 100 and 102, 6 calls.
  • vf-eval @ configs/wordle.toml (infinite): tasks 0–4.
  • vf-eval @ configs/alphabet_sort.toml -s: refused with "cannot shuffle", 0 calls. With --select.include.idx 0:4 -s -n 2: runs.
  • vf-validate gsm8k … and vf-debug reverse-text -n 3 -s: task order matches the oracle.
• vf-eval @ configs/<each>.toml --dry-run for all 12 configs: each resolves to its intended select.limit.

🤖 Generated with Claude Code

---

Note

Medium Risk
Breaking config and task-key semantics affect eval resume and all CLIs; selection logic on infinite tasksets is easy to misconfigure but is guarded with explicit errors.

Overview
Introduces a unified [select] block (and --select.* / -n / -s) for choosing which tasks run, replacing top-level num_tasks and shuffle on eval, debug, validate, and GEPA. Taskset.select(SelectConfig) applies lazy views in fixed order: include / exclude (by idx slices, ids, keys, names), shuffle, skip, limit (take in Python; head is removed).

TaskData.idx is now set from the load() stream position during iteration (overwriting author-set values); built-in and example tasksets drop manual idx, and TaskData.id is added for durable source IDs. Taskset.bounded gates shuffle and identity-based filters on infinite tasksets; unmatched include/exclude entries log warnings.

Sample configs/*.toml move limits to [select].limit. Docs describe selection and updated task identity. GEPA no longer shuffles by default—use select.shuffle (-s) for the old split behavior. Task keys may change for tasksets that relied on embedded idx in the hash, affecting resume across this upgrade.

^{Reviewed by Cursor Bugbot for commit c305bde. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add lazy taskset selection views and select config replacing num_tasks

• Iterating a taskset now assigns idx automatically, so all environment loaders drop manually supplied TaskData.idx; TaskData gains an optional durable id field and Task gains a with_data copy helper (taskset.py, task.py).
• Taskset gains lazy include, exclude, shuffle, skip, and take views (replacing head), plus a select view that applies include/exclude/shuffle/skip/limit in a fixed order. Matching supports idx ranges, id, key, and name, warns on unmatched selectors, and rejects unbounded shuffles of infinite streams.
• New SelectConfig/SelectCLIConfig models in select.py replace top-level num_tasks and shuffle in EvalConfig, DebugConfig, ValidateConfig, and GEPAConfig; CLI aliases map -n to select.limit and -s to select.shuffle. Exported via init.py.
• Runners for eval, validate, debug, and GEPA now apply config.select and require the resulting view to be bounded; example TOML configs and docs updated.
• Behavioral Change: config files using top-level num_tasks/shuffle no longer parse — migrate to [select] with limit/shuffle. head() on tasksets is replaced by take(); check configs/ and the GEPA runner's shuffle default (now off unless select.shuffle is set).

^{Macroscope summarized c305bde.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d664149. Configure here.}

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a framework-wide task-selection capability and changes the eval/debug/validate/GEPA execution paths, task identity semantics, and public configuration APIs. The breaking migration and altered selection behavior have a broad enough runtime impact to warrant human review.

^{You can add or adjust custom eligibility rules. Learn more.}