Skip to content

feat(orchestrator)!: rename env collections to sources, compose the verifiers config blocks #3150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hallerite merged 15 commits into from
Jul 29, 2026
Merged

feat(orchestrator)!: rename env collections to sources, compose the verifiers config blocks #3150

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
e81f265
rename training environments to sources
hallerite Jul 28, 2026
25f91cb
fix CPU test config fixtures
hallerite Jul 28, 2026
0304ec0
rename evaluation environments to sources
hallerite Jul 29, 2026
aef4f8f
chore(orchestrator)!: rename max_inflight_rollouts to max_inflight_ep…
hallerite Jul 28, 2026
f7f600f
feat(orchestrator)!: compose the verifiers env, serve and legacy blocks
hallerite Jul 29, 2026
8b289ff
fix the source-config readers the compose commit missed
hallerite Jul 29, 2026
5f423e7
spell the env annotation out instead of vf.EnvField
hallerite Jul 29, 2026
e061233
declare the env default with the factory verifiers exports
hallerite Jul 29, 2026
5319825
inline the legacy helpers, use the config class as the default
hallerite Jul 29, 2026
a7924f8
drop the serving-key pointers here too
hallerite Jul 29, 2026
8185ca6
bump verifiers: no max_concurrent alias on the env block
hallerite Jul 29, 2026
cfb3597
bump verifiers to 0.2.2.dev43 (merged main)
hallerite Jul 29, 2026
96984a7
fix: spawn the env server again, and let a pip-only install resolve v…
hallerite Jul 29, 2026
73c039c
fix: make "no address" a value again, and migrate + cover the k8s con…
hallerite Jul 29, 2026
b797eaf
drop the narrative comment on the verifiers floor
hallerite Jul 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion configs/basic/alphabet-sort/rl.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ group_size = 8
[orchestrator.train.sampling]
max_completion_tokens = 768

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "alphabet-sort"
env.taskset = { id = "alphabet-sort-v1", min_turns = 3, max_turns = 5, task = { power_per_turn = false } }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/basic/hendrycks-sanity/rl.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ batch_size = 128
group_size = 8
seq_len = 8192

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "hendrycks-math"
env.taskset = { id = "math-env-v1", dataset_name = "mikasenghaas/Sanity-Test-R1D-1.5B", dataset_subset = "default" }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/basic/reverse-text/rl.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ group_size = 16
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/basic/wiki-search/rl.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ group_size = 16
[orchestrator.train.sampling]
max_completion_tokens = 512

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "wiki-search"
env.taskset = { id = "wiki-search-v1" }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/basic/wordle/rl.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ name = "PrimeIntellect/Qwen3-1.7B-Wordle-SFT"
batch_size = 128
group_size = 8

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "wordle"
env.taskset = { id = "wordle-v1" }
env.player.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/ci/integration/alphabet_sort.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ group_size = 8
[orchestrator.train.sampling]
max_completion_tokens = 384

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "alphabet-sort"
env.taskset = { id = "alphabet-sort-v1", min_turns = 2, max_turns = 2, min_names_per_turn = 1, max_names_per_turn = 3, task = { similarity_power = 4, power_per_turn = false } }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/ci/integration/reverse-text-lora/resume.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ name = "r8-1e-4"
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/ci/integration/reverse-text-lora/start.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ name = "r8-1e-4"
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/ci/integration/reverse-text-moe/start.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ group_size = 16
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/ci/integration/reverse-text-multi-run/orchestrator.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ lr = 3e-5
[train.sampling]
max_completion_tokens = 128

[[train.env]]
[[train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
4 changes: 2 additions & 2 deletions configs/ci/integration/reverse-text-rl-opd/start.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ name = "prime-qwen3"
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand All @@ -50,7 +50,7 @@ num_examples = 16
[orchestrator.eval.sampling]
max_completion_tokens = 128

[[orchestrator.eval.env]]
[[orchestrator.eval.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
4 changes: 2 additions & 2 deletions configs/ci/integration/reverse-text-rl-sft/start.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ name = "prime-qwen3"
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand All @@ -56,7 +56,7 @@ num_examples = 16
[orchestrator.eval.sampling]
max_completion_tokens = 128

[[orchestrator.eval.env]]
[[orchestrator.eval.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/ci/integration/reverse-text/resume.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ group_size = 16
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/ci/integration/reverse-text/start.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ group_size = 16
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/ci/nightly-fft/alphabet-sort.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ name = "Qwen/Qwen3-4B-Instruct-2507"
batch_size = 512
group_size = 16

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "alphabet-sort"
env.taskset = { id = "alphabet-sort-v1", min_turns = 3, max_turns = 5, task = { power_per_turn = false } }
env.agent.harness = { id = "null" }
Expand Down
4 changes: 2 additions & 2 deletions configs/ci/nightly-fft/hendrycks-sanity.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ batch_size = 512
group_size = 8
seq_len = 8192

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "hendrycks-math"
env.taskset = { id = "math-env-v1", dataset_name = "mikasenghaas/Sanity-Test-R1D-1.5B", dataset_subset = "default" }
env.agent.harness = { id = "null" }
Expand All @@ -27,7 +27,7 @@ env.agent.runtime = { type = "subprocess" }
[orchestrator.eval]
interval = 50

[[orchestrator.eval.env]]
[[orchestrator.eval.source]]
name = "aime2024"
env.taskset = { id = "aime24-v1" }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/ci/nightly-fft/reverse-text.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"
batch_size = 128
group_size = 16

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/ci/nightly-fft/wiki-search.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ oversampling_factor = 2.0
type = "zero_advantage"
enforce = true

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "wiki-search"
env.taskset = { id = "wiki-search-v1" }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/ci/nightly-fft/wordle.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ name = "PrimeIntellect/Qwen3-1.7B-Wordle-SFT"
batch_size = 512
group_size = 16

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "wordle"
env.taskset = { id = "wordle-v1" }
env.player.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/ci/nightly/multimodal_color_codeword.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ group_size = 16
[orchestrator.train.sampling]
max_completion_tokens = 64

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "color-codeword"
env.taskset = { id = "color-codeword-v1", images_per_turn = 1 }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion configs/debug/algo/echo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ type = "echo"
[orchestrator.algo.roles.user]
alpha = 0.1

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "alphabet-sort"
env.taskset = { id = "alphabet-sort-v1", min_turns = 3, max_turns = 5, task = { power_per_turn = false } }
env.agent.harness = { id = "null" }
Expand Down
4 changes: 2 additions & 2 deletions configs/debug/algo/grpo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ name = "prime-qwen3"
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand All @@ -34,7 +34,7 @@ num_examples = 128
[orchestrator.eval.sampling]
max_completion_tokens = 128

[[orchestrator.eval.env]]
[[orchestrator.eval.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
4 changes: 2 additions & 2 deletions configs/debug/algo/max_rl.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ name = "prime-qwen3"
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand All @@ -34,7 +34,7 @@ num_examples = 128
[orchestrator.eval.sampling]
max_completion_tokens = 128

[[orchestrator.eval.env]]
[[orchestrator.eval.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
10 changes: 5 additions & 5 deletions configs/debug/algo/mixed_grpo_opd.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,22 +31,22 @@ name = "prime-qwen3"
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text-grpo"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
env.agent.runtime = { type = "subprocess" }

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text-opd"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
env.agent.runtime = { type = "subprocess" }

[orchestrator.train.env.algo]
[orchestrator.train.source.algo]
type = "opd"

[orchestrator.train.env.algo.teacher]
[orchestrator.train.source.algo.teacher]
name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-RL"
base_url = ["http://localhost:8001/v1"]

Expand All @@ -57,7 +57,7 @@ num_examples = 128
[orchestrator.eval.sampling]
max_completion_tokens = 128

[[orchestrator.eval.env]]
[[orchestrator.eval.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
4 changes: 2 additions & 2 deletions configs/debug/algo/opd.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ name = "prime-qwen3"
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand All @@ -46,7 +46,7 @@ num_examples = 128
[orchestrator.eval.sampling]
max_completion_tokens = 128

[[orchestrator.eval.env]]
[[orchestrator.eval.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
4 changes: 2 additions & 2 deletions configs/debug/algo/opd_lora.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ name = "prime-qwen3"
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand All @@ -46,7 +46,7 @@ num_examples = 128
[orchestrator.eval.sampling]
max_completion_tokens = 128

[[orchestrator.eval.env]]
[[orchestrator.eval.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
4 changes: 2 additions & 2 deletions configs/debug/algo/rae.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ name = "prime-qwen3"
[orchestrator.train.sampling]
max_completion_tokens = 512

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "kuhn-poker"
env.taskset = { id = "kuhn-poker-v1" }
env.player0.harness = { id = "null" }
Expand All @@ -36,7 +36,7 @@ num_examples = 16
[orchestrator.eval.sampling]
max_completion_tokens = 512

[[orchestrator.eval.env]]
[[orchestrator.eval.source]]
name = "kuhn-poker"
env.taskset = { id = "kuhn-poker-v1" }
env.player0.harness = { id = "null" }
Expand Down
4 changes: 2 additions & 2 deletions configs/debug/algo/self_distill.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ name = "prime-qwen3"
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand All @@ -42,7 +42,7 @@ num_examples = 128
[orchestrator.eval.sampling]
max_completion_tokens = 128

[[orchestrator.eval.env]]
[[orchestrator.eval.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
4 changes: 2 additions & 2 deletions configs/debug/algo/sft_distill.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ name = "prime-qwen3"
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand All @@ -51,7 +51,7 @@ num_examples = 128
[orchestrator.eval.sampling]
max_completion_tokens = 128

[[orchestrator.eval.env]]
[[orchestrator.eval.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
4 changes: 2 additions & 2 deletions configs/debug/algo/sft_distill_lora.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ name = "prime-qwen3"
[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
[[orchestrator.train.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand All @@ -51,7 +51,7 @@ num_examples = 128
[orchestrator.eval.sampling]
max_completion_tokens = 128

[[orchestrator.eval.env]]
[[orchestrator.eval.source]]
name = "reverse-text"
env.taskset = { id = "reverse-text-v1" }
env.agent.harness = { id = "null" }
Expand Down
2 changes: 1 addition & 1 deletion deps/verifiers
Open in desktop
Submodule verifiers updated 35 files
+46 −7 .github/workflows/test.yml
+17 −0 docs/v1/env.md
+3 −1 docs/v1/evaluation.md
+1 −1 docs/v1/gepa.md
+1 −1 environments/compact/compact/program.py
+1 −1 pyproject.toml
+2 −2 skills/evaluate-environments/SKILL.md
+27 −21 skills/evaluate-environments/references/REFERENCE.md
+7 −7 tests/v1/conftest.py
+7 −24 uv.lock
+12 −7 verifiers/v1/__init__.py
+9 −8 verifiers/v1/agent.py
+1 −1 verifiers/v1/cli/dashboard/eval.py
+9 −7 verifiers/v1/cli/eval/main.py
+12 −7 verifiers/v1/cli/eval/runner.py
+17 −14 verifiers/v1/cli/serve.py
+16 −123 verifiers/v1/configs/cli/env.py
+66 −6 verifiers/v1/configs/cli/eval.py
+53 −7 verifiers/v1/configs/cli/serve.py
+11 −4 verifiers/v1/configs/env.py
+24 −0 verifiers/v1/configs/legacy.py
+63 −0 verifiers/v1/configs/serve.py
+24 −16 verifiers/v1/env.py
+3 −3 verifiers/v1/gepa/config.py
+6 −1 verifiers/v1/harness.py
+1 −1 verifiers/v1/harnesses/bash/program.py
+1 −1 verifiers/v1/harnesses/null/program.py
+2 −0 verifiers/v1/harnesses/rlm/harness.py
+1 −1 verifiers/v1/harnesses/terminus_2/harness.py
+12 −6 verifiers/v1/legacy.py
+1 −1 verifiers/v1/push.py
+10 −0 verifiers/v1/runtimes/base.py
+8 −6 verifiers/v1/serve/pool.py
+8 −6 verifiers/v1/serve/server.py
+43 −54 verifiers/v1/tasksets/harbor/taskset.py
Loading