feat!: standalone env servers

[mikasenghaas]

Summary

Removes the env-server "sidecaring" path entirely: the orchestrator never spawns env servers via subprocess anymore — it always connects.

• Addresses are derived, not configured per source (matching how transport wiring works): OrchestratorConfig.env_addresses maps each (split, name) source to tcp://127.0.0.1:<base + i> in config order (train → eval). There is no address field on the source's serve block — the launcher and the orchestrator independently derive the same answer from the config alone. The single wiring knob is orchestrator.env_server_base_port (default 5000, like inference.server.port): concurrent runs on one host give each run its own base, keeping orchestrator↔env-server 1:1.
• uv run env-server serves one environment, and is now the only CLI for hosting an env server: the merged companion feat!: remove the serve CLI, rename ServingConfig to ServeConfig verifiers#2237 removed verifiers' unused serve CLI (and renamed ServingConfig → ServeConfig). The entrypoint moves to entrypoints/env_server.py next to the other console scripts, and its config (EnvServerConfig) is [env] + [serve] (vf.ServeConfig: pool, address, max_concurrent) + [legacy], plus prime-rl's [log] — and validates the env identity at parse time (a missing env or a v0 legacy.id mixed with a v1 env id fails immediately, as it does on orchestrator sources).
• Orchestrator only connects, in parallel: envs.py loses all the multiprocessing spawn/teardown machinery; each Env gets its address at construction, Env.start() builds an EnvClient and polls health (600s budget), and Envs.start() connects all sources with a single asyncio.gather — since every address is known up front, spawn (launcher) and connect (orchestrator) happen in parallel.
• Local rl launcher spawns the servers: write_subconfigs writes one EnvServerConfig TOML per source (configs/envs/{train,eval}/<name>.toml — the source's EnvConfig fields plus its derived address) and rl_local starts one env-server process per source, logged to logs/envs/{train,eval}/<name>.log (same paths as before) and monitored like inference/trainer/orchestrator.
• Multinode template: multi_node_rl.sbatch.j2 launches the env servers on the orchestrator node right before the orchestrator (both regular and disaggregated layouts).
• The source-level serve block is now prime-rl's own two-field config (pool, max_concurrent), consumed by the env-server process via the written per-source TOML.
• The multi-run integration test starts one env server per orchestrator (1:1), each pair on its own env_server_base_port; a resumed run keeps its port and reconnects to the same server. Docs and the monitor-run skill are updated to the new process topology.
• New debug config configs/debug/multi-env/rl.toml: two reverse-text train sources + one eval source — the minimal multi-source run exercising one env server per source.

Breaking

• uv run orchestrator standalone no longer spawns env servers. Env servers must be running at each source's derived address — start one per source with uv run env-server (an [env] block matching the source, with serve.address set to the source's derived address; it defaults to tcp://127.0.0.1:5000, the first source's address at the default base port).
• serve.address is removed from sources. Env-server locations are no longer configurable per source: previously None meant "spawn a subprocess on a free port" and a set value meant "connect to this external server"; now every source's server lives at its deterministic local address (offset by env_server_base_port), so pointing a source at a remote env server is no longer expressible. Concurrent runs on one host must set distinct env_server_base_ports.
• env-server config restructured: [serve] and [legacy] are top-level blocks and [env] is the verifiers env block directly (previously everything nested under [env] as a source-shaped config).
• The source-level ServeConfig is prime-rl's own two-field config (pool, max_concurrent) instead of a vf.ServeConfig subclass with an optional address.

Verification

All runs on 2 GPUs from this branch (with main merged in), via uv run rl:

• reverse-text single env (configs/ci/integration/reverse-text/start.toml, 5 steps): launcher spawns PRIME-RL::EnvServer bound at tcp://127.0.0.1:5000, orchestrator connects ("Train environment(s) ready" before inference is even up), reward 0.16 → 0.28, "Training finished!", clean teardown (no leftover processes, exit 0).
• multi-env debug run (configs/debug/multi-env/rl.toml, 10 steps): three env servers spawned in parallel at derived 5000/5001/5002, per-source configs under configs/envs/{train,eval}/ (each a vf-shaped [env]/[serve] TOML carrying its derived serve.address), per-source logs under logs/envs/{train,eval}/, both train sources contribute rollouts, evals run through the eval env server (reward 0.13 → 0.72 across steps 1/5/10), train reward 0.09 → 0.66, "Training finished!" with exit 0 and no leftover processes or bound ports.
• Raising taskset exits the run immediately: same config with one train source's taskset pointed at a nonexistent dataset split. All three env servers come up healthy; the orchestrator raises ValueError: Unknown split "bogus" at taskset load (~5s after it starts), the full traceback lands in orchestrator.log and the launcher output (which tails it), the launcher logs Orchestrator failed with exit code 1 → Terminating all processes..., and exits 1. The env-server logs show clean startup then a graceful EnvServer down / EnvServerPool down from the launcher's teardown — no orphaned processes or bound ports.

tests/unit passes (484 passed after merging main; the test_qwen3_vl_e2e failure is pre-existing on main).

---

Note

High Risk
This is a breaking change to RL launch topology and standalone orchestrator workflows; misconfigured ports or missing env-server processes will cause rollout hangs or startup failures.

Overview
Breaking: The orchestrator no longer spawns env servers as child processes. Standalone uv run orchestrator requires one uv run env-server per source already bound at each source’s derived address.

Wiring: Each train/eval source gets tcp://127.0.0.1:<env_server_base_port + i> in config order (train then eval). Per-source serve.address is removed; concurrent runs on one host use distinct orchestrator.env_server_base_port values.

Launcher & config: uv run rl writes per-source EnvServerConfig TOMLs under configs/envs/{train,eval}/<name>.toml and starts monitored env-server processes (logs at logs/envs/{train,eval}/<name>.log). EnvServerConfig is now top-level [env], [serve], and [legacy] with parse-time validation. Source serve on the orchestrator side is only pool and max_concurrent for the written env-server configs.

Orchestrator: envs.py drops multiprocessing spawn/teardown; Env takes a fixed address, start() polls health (600s), and Envs.start() connects all sources in parallel with asyncio.gather.

Ops & tests: Multinode multi_node_rl.sbatch.j2 starts env servers on the orchestrator node before the orchestrator. Multi-run CI starts one env server per orchestrator with matching base ports. Adds configs/debug/multi-env/rl.toml and documents the new process topology.

^{Reviewed by Cursor Bugbot for commit d0a79b2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ac2cd8d. Configure here.}