feat(v1)!: two-level concurrency, and serving as its own config block

docs/v1/env.md

@@ -52,3 +52,20 @@ Just like tasksets and harnesses, an `Env` can be user-defined for full expressi
 | `single-agent` | `agent` | (default) one `agent` plays the taskset |
 | `best-of-n` | `agent` | `n` independent attempts per episode; its metrics mark the argmax-reward sibling (`best`) and whether any reached `--env.threshold` (`pass_at_n`) — rejection sampling and pass@k. |
 | `agentic-judge` | `solver`, `judge` | the solver plays the task; a code-executing judge agent verifies the finished attempt with real execution. |
+
+## Concurrency
+
+Write independent agents as independent (`asyncio.gather`, a `TaskGroup`) — how many actually run at once is the run's call, not the env's. Two knobs bound it, and the **episode is the unit** at the outer one:
+
+| knob | bounds |
+| --- | --- |
+| `max_concurrent` / `-c` | episodes in flight (per worker when served) |
+| `env.max_concurrent_agents` | agent runs inside one episode — **1** by default |
+
+At the default, `-c 128` is 128 live agent runs whatever the env does internally. Set `max_concurrent_agents` higher (or `None` for no limit) when you want an episode's fan-out to run together — `-c` still caps the episodes carrying it:
+
+```bash
+uv run eval gsm8k-v1 --env.id best-of-n --env.n 16 --env.max-concurrent-agents None -c 128
+```
+
+Turn-taking envs are unaffected: an interaction holds its agent permit only around an active segment, never while awaiting its caller, so `user-sim` and games alternate at any setting.

docs/v1/evaluation.md

@@ -40,7 +40,9 @@ The output from evaluations are written into `outputs/<env>--<model>--<harness>/
 - `num_tasks` — how many tasks to evaluate. Not setting a value means all tasks; an
   infinite taskset (a procedural generator, e.g. `wordle-v1`) requires it
 - `num_rollouts` — rollouts per task
-- `max_concurrent` — caps how many rollouts are in flight at once
+- `max_concurrent` — caps how many episodes are in flight at once; `env.max_concurrent_agents` caps the agent runs inside one episode (1 by default — see [The Env § Concurrency](env.md#concurrency))
+- `[serve]` — how the env is hosted under `--server`: `serve.pool` (worker pool), `serve.address`, and `serve.max_concurrent` (a per-worker episode bound; unset takes `max_concurrent`)
+- `[legacy]` — run a classic (v0) environment through the bridge instead of `[env]`: `legacy.id`, `legacy.args`
 - `verbose` — log at debug instead of info
 - `shuffle` — randomizes the order of tasks (fixed seed); a no-op on an infinite taskset
 

docs/v1/gepa.md

@@ -31,7 +31,7 @@ Validate the config by using `uv run gepa @ config.toml --dry-run`. To run GEPA,
 - `reflection_model` / `reflection_client` — model/endpoint that proposes new prompts (default: reuse `model` / `client`)
 - `num_train` / `num_val` — train tasks for reflection minibatches and held-out val tasks for the pareto frontier (defaults: 100 / 50)
 - `max_total_rollouts` — total rollouts the run may spend (default: 500)
-- `max_concurrent` / `-c` — caps how many rollouts are in flight at once (default: 128)
+- `max_concurrent` / `-c` — caps how many episodes are in flight at once (default: 128)
 
 ## Output
 

skills/evaluate-environments/SKILL.md

@@ -204,7 +204,7 @@ Do not average these categories together without reporting failure rate.
 For a real v0 package only:
 
 ```bash
-prime eval run --id legacy-env --args.split test -n 5
+prime eval run --legacy.id legacy-env --legacy.args.split test -n 5
 ```
 
-Label the result as bridged v0. Do not present `args` as the v1 config contract.
+Label the result as bridged v0. Do not present `legacy.args` as the v1 config contract.

skills/evaluate-environments/references/REFERENCE.md

@@ -2,7 +2,7 @@
 
 A complete reference of every settable config field for **evaluating tasksets in `verifiers.v1`**. The config tree is parsed from CLI flags (dotted, e.g. `--env.agent.runtime.type docker`) and/or `@ file.toml` by `prime-pydantic-config`; every field below is settable either way unless noted.
 
-The root config the eval CLI parses is [`EvalConfig`](#evalconfig--the-run). It composes the environment (`env` — the whole `[env]` block: taskset, agents, limits) with the run knobs (model, sampling, counts) and the worker pool. The tree:
+The root config the eval CLI parses is [`EvalConfig`](#evalconfig--the-run). It composes three orthogonal blocks — the environment (`env`: taskset, agents, limits), how it's hosted (`serve`: worker pool, address, per-worker bound), and the classic v0 bridge (`legacy`) — with its own run knobs (model, sampling, counts). The tree:
 
 ```text
 EvalConfig                          (the run)
@@ -20,19 +20,25 @@ EvalConfig                          (the run)
 │  │  └─ max_turns / max_input_tokens / max_output_tokens / max_total_tokens
 │  ├─ timeout: TimeoutConfig        (episode / finalize — the env's own hooks)
 │  ├─ retries: RetryConfig          (whole-episode fallback for faults no agent owns)
+│  ├─ max_concurrent_agents          (agent runs inside one episode — 1 by default)
 │  └─ interception
-└─ pool: PoolConfig                 (static | elastic) — env-server only
+├─ serve: ServingConfig             (how it's hosted — the `--server` path)
+│  ├─ pool: PoolConfig              (static | elastic)
+│  ├─ address                       (where the ROUTER binds)
+│  └─ max_concurrent                (episodes per worker; unset = the run's `-c`)
+└─ legacy: LegacyEnvConfig          (a classic v0 env instead of `env`)
+   └─ id / args / extra_env_kwargs
 ```
 
-There is no run-level harness: each agent pins its own (`--env.agent.harness.*` on the single-agent env), an unpinned agent runs the taskset's default harness (its bundled one, else `bash`), and a declared pin is the env author's default. The retired flat axes error with a pointer: `--taskset.*` → `--env.taskset.*`, `--harness.*` → `--env.<agent>.harness.*`.
+There is no run-level harness: each agent pins its own (`--env.agent.harness.*` on the single-agent env), an unpinned agent runs the taskset's default harness (its bundled one, else `bash`), and a declared pin is the env author's default. Nothing is flat on the run config — a taskset is `--env.taskset.*`, a harness `--env.<agent>.harness.*`, the pool and bind address `--serve.*`, and a v0 env's id/args `--legacy.*`.
 
 Sibling entrypoints reuse the same tree: [`ServeConfig`](#serveconfig--the-env-server-cli) (env server) and [`ValidateConfig`](#validateconfig--the-validate-cli) (per-task validation). All three live in `verifiers/v1/configs/cli/`.
 
 ---
 
 ## EvalConfig — the run
 
-`verifiers/v1/configs/cli/eval.py` — `EvalConfig(EnvServerConfig)`. The single config object the eval CLI parses. Inherits [`EnvServerConfig`](#envserverconfig--the-pool) (the `env` block + `--pool.*` + the legacy v0 fields) and adds the run knobs. Everything environment-shaped lives under `--env.*` / `[env]`.
+`verifiers/v1/configs/cli/eval.py` — `EvalConfig(BaseConfig)`. The single config object the eval CLI parses: it declares [`env`](#envconfig--the-environment), [`serve`](#servingconfig--how-its-hosted) and [`legacy`](#legacy-v0-envs) as blocks and adds the run knobs. Nothing is inherited — the `serve` CLI and a trainer declare the same blocks. Everything environment-shaped lives under `--env.*` / `[env]`.
 
 | Field | Type | Default | Aliases | Notes |
 | --- | --- | --- | --- | --- |
@@ -43,18 +49,18 @@ Sibling entrypoints reuse the same tree: [`ServeConfig`](#serveconfig--the-env-s
 | `num_tasks` | `int \| None` | `None` | `batch_size`, `num_examples`, `num_tasks`, `n` | How many tasks to evaluate (None = all). |
 | `num_rollouts` | `int` | `1` | `group_size`, `rollouts_per_example`, `num_rollouts`, `r` | Independent episodes per task — the trainer's group size. Env-internal fan-out (e.g. best-of-n attempts) is the env's own knob, not `-r`. |
 | `shuffle` | `bool` | `False` | `shuffle`, `s` | Shuffle tasks before taking the first `num_tasks`. |
-| `max_concurrent` | `int \| None` | `128` | `max_concurrent`, `c` | Max rollouts in flight at once. |
+| `max_concurrent` | `int \| None` | `128` (≥1) | `max_concurrent`, `c` | Episodes in flight at once (`None` = no limit); under `--server` it seeds each worker's bound unless `--serve.max-concurrent` pins one. Agent runs inside one episode are `--env.max-concurrent-agents`. |
 | `verbose` | `bool` | `False` | `verbose`, `v` | Log at debug level instead of info. |
 | `dry_run` | `bool` | `False` | — | Resolve + validate the config and dump it, then exit. |
 | `rich` | `bool` | `True` | — | Live dashboard instead of per-rollout logs (in-process only). |
-| `server` | `bool` | `False` | — | Drive rollouts through the env-server worker pool (sized by `pool`) instead of in-process — the path prime-rl trains through. Incompatible with `--rich`. |
+| `server` | `bool` | `False` | — | Drive rollouts through the env-server worker pool (sized by `[serve]`) instead of in-process — the path prime-rl trains through. Incompatible with `--rich`. |
 | `push` | `bool` | `True` | — | Upload the finished run to the private Evaluations tab. Disable with `--no-push`. |
 | `output_dir` | `Path \| None` | `None` | `output_dir`, `o` | Where to write the run (`config.toml` + `traces.jsonl`). None = a fresh per-run dir under `outputs/<env>--<model>--<harness>/<uuid>`. |
 | `resume` | `Path \| None` | `None` | — | Set by `--resume <dir>`: re-run missing/errored rollouts, episode-atomically (a multi-trace rollout is kept or redone as a unit; `Env.complete` is the keep verdict). Excluded from the saved config; takes no other args. |
 
 Validator: `--rich` + `--server` together is rejected (the dashboard is in-process only).
 
-Inherited from `EnvServerConfig`: [`env`](#envconfig--the-environment), [`pool`](#pool-config), and the legacy `id` / `args` / `extra_env_kwargs`.
+Blocks: [`env`](#envconfig--the-environment), [`serve`](#servingconfig--how-its-hosted), [`legacy`](#legacy-v0-envs). Derived: `is_legacy` (a `legacy.id` and no v1 taskset), `env_id` (the env's id, else the v0 one), `worker_max_concurrent` (`serve.max_concurrent`, else `max_concurrent`).
 
 ---
 
@@ -137,26 +143,27 @@ Per-run caps (turns, tokens, stage timeouts, retries) are agent fields, not env
 
 Trainability is not a config field: it is env truth, set in place by the env's `setup(agents)` hook (default: every agent trains) and stamped on each trace. An env that wants the flip run-configurable exposes its own switch (e.g. proposer-solver's `--env.train_solver false`).
 
-### Legacy (v0) backwards-compat fields
+### Legacy (v0) envs
 
-On `EnvServerConfig` (below): set `id` (leave `env.taskset` unset) to run a classic `verifiers.load_environment` env through the legacy bridge.
+`verifiers/v1/configs/legacy.py` — `LegacyEnvConfig(BaseConfig)`, the `[legacy]` block. Set `legacy.id` (and leave `env.taskset` unset) to run a classic `verifiers.load_environment` env through the bridge. A v0 id next to a v1 taskset is refused, not silently ignored.
 
 | Field | Type | Default | Notes |
 | --- | --- | --- | --- |
 | `id` | `ID \| None` | `None` | Classic v0 env id (`name`, `org/name`, or `org/name@version`). |
 | `args` | `dict` | `{}` | Construction kwargs forwarded to `load_environment(id, **args)`. |
 | `extra_env_kwargs` | `dict` | `{}` | Post-load kwargs applied via `env.set_kwargs(**...)` (e.g. `max_total_completion_tokens`, `max_seq_len`, `timeout_seconds`). |
 
-`EnvServerConfig.is_legacy` → `id is not None and no v1 taskset`.
+`is_legacy` → a `legacy.id` is set and no v1 taskset.
 
-### EnvServerConfig — the pool
+### ServingConfig — how it's hosted
 
-`EnvServerConfig(BaseConfig)`. The `env` block plus the worker pool sizing and the legacy v0 fields. Shared by the `serve` CLI, server-backed eval, and prime-rl's orchestrator.
+`verifiers/v1/configs/serve.py` — `ServingConfig(BaseConfig)`, the `[serve]` block. Read by whoever hosts the env: the `serve` CLI, a server-backed eval, a trainer's orchestrator. An in-process eval ignores it.
 
 | Field | Type | Default | Notes |
 | --- | --- | --- | --- |
-| `env` | `EnvConfig` | `SingleAgentEnvConfig()` | The environment (above). `SerializeAsAny`, so the resolved subclass's agents and knobs survive `model_dump` onto the wire. |
-| `pool` | `PoolConfig` | `ElasticPoolConfig()` | See [Pool config](#pool-config). |
+| `pool` | `PoolConfig` | `ElasticPoolConfig()` | Worker-pool sizing — see [Pool config](#pool-config). |
+| `address` | `str` | `"tcp://127.0.0.1:5000"` | ZMQ address the ROUTER binds (and clients connect to). |
+| `max_concurrent` | `int \| None` | `None` (≥1) | Episodes in flight per worker. None = take the run's own bound (an eval's `-c`). |
 
 ---
 
@@ -195,7 +202,7 @@ Rerun when the run ends with a captured error. Matching is by the error's **exce
 
 ## Pool config
 
-`verifiers/v1/configs/cli/env.py`. Discriminated on `type`; selected with `--pool.type static|elastic`. Drives the env-server worker pool (the `--server` path).
+`verifiers/v1/configs/serve.py`. Discriminated on `type`; selected with `--serve.pool.type static|elastic`. Drives the env-server worker pool (the `--server` path).
 
 ### `StaticPoolConfig` — `type: "static"`
 
@@ -212,7 +219,7 @@ Elastic pool: start at one worker and scale up on demand.
 | Field | Type | Default | Notes |
 | --- | --- | --- | --- |
 | `max_workers` | `int \| None` | `None` | Upper bound on workers (None = unbounded). |
-| `multiplex` | `int` | `128` (≥1) | Rollouts per worker for the scale-up trigger: add a worker once in-flight rollouts reach 90% of `workers * multiplex`. |
+| `multiplex` | `int` | `128` (≥1) | Episodes per worker for the scale-up trigger: add a worker once in-flight episodes reach 90% of `workers * multiplex`. |
 
 ---
 
@@ -534,15 +541,14 @@ A judge class may define:
 
 ## ServeConfig — the env-server CLI
 
-`verifiers/v1/configs/serve.py` — `ServeConfig(EnvServerConfig)`. The env-server CLI. Inherits the `env` block + pool, so `--env.*` / `--pool.*` are the same flags as eval. Adds only CLI-specific serving knobs.
+`verifiers/v1/configs/cli/serve.py` — `ServeConfig(BaseConfig)`. The env-server CLI. Declares the same blocks as eval — [`env`](#envconfig--the-environment), [`serve`](#servingconfig--how-its-hosted), [`legacy`](#legacy-v0-envs) — so `--env.*` / `--serve.*` are the same flags there, and adds only its own two.
 
 | Field | Type | Default | Aliases | Notes |
 | --- | --- | --- | --- | --- |
-| `address` | `str` | `"tcp://127.0.0.1:5000"` | `address`, `a` | ZMQ address the ROUTER binds. |
 | `verbose` | `bool` | `False` | `verbose`, `v` | Log at debug level. |
 | `dry_run` | `bool` | `False` | — | Resolve + validate and dump, then exit. |
 
-Plus all inherited `EnvServerConfig` fields (`env`, `pool`, legacy).
+Where it binds is `--serve.address`; how many workers is `--serve.pool.*`.
 
 ---
 
@@ -584,7 +590,7 @@ Use `only_gold` or `only_setup` to select one mode; setting both is rejected. Th
 - **Plugin resolution.** The run's `env` field narrows to the selected env's config class
   (`--env.id`, else the taskset's exported env, else `SingleAgentEnvConfig`) *before* validation; inside it, `taskset` narrows by its id and each pinned agent `harness` by its id. Entries in `TaskConfig.judges` are similarly narrowed by each judge `id`. This is why local and Hub plugin fields remain typed and appear in CLI validation instead of living in an untyped arguments dictionary.
 - **Dotted flags.** Every nested field is part of the same CLI tree
-  (`--env.agent.runtime.type docker`, `--env.taskset.split test`, `--pool.max_workers 8`, `--env.agent.retries.max_retries 2`). An `@ file.toml` describes the identical tree; explicit CLI values layer over values loaded from the file.
+  (`--env.agent.runtime.type docker`, `--env.taskset.split test`, `--serve.pool.max-workers 8`, `--env.agent.retries.max_retries 2`). An `@ file.toml` describes the identical tree; explicit CLI values layer over values loaded from the file.
 - **Runtime precedence.** An explicit, non-default CLI/TOML `workdir` or resource field wins over
   `TaskData`; otherwise a non-`None` row value fills it, and otherwise the runtime/provider default remains. `TaskData.image` is the required image for that row and replaces the runtime's base image. Unsupported resource fields are ignored; evaluation warns once per runtime/field.
 - **Timeout precedence.** For eval stages, a non-`None` agent-level `TimeoutConfig` value

verifiers/v1/__init__.py

@@ -14,16 +14,18 @@
     resolve_client,
 )
 from verifiers.v1.configs.agent import AgentConfig
-from verifiers.v1.configs.cli.env import (
-    ElasticPoolConfig,
-    EnvServerConfig,
-    StaticPoolConfig,
-    pool_serve_kwargs,
-)
+from verifiers.v1.configs.cli.env import narrowed_env_annotation, resolve_env_field
 from verifiers.v1.configs.env import EnvConfig, default_agent_harness
 from verifiers.v1.configs.harness import HarnessConfig
 from verifiers.v1.configs.judge import JudgeConfig, Judges
+from verifiers.v1.configs.legacy import LegacyEnvConfig
 from verifiers.v1.configs.retries import RetryConfig
+from verifiers.v1.configs.serve import (
+    ElasticPoolConfig,
+    ServingConfig,
+    StaticPoolConfig,
+    pool_serve_kwargs,
+)
 from verifiers.v1.configs.task import TaskConfig
 from verifiers.v1.configs.taskset import TasksetConfig
 from verifiers.v1.decorators import metric, reward, stop, tool
@@ -248,7 +250,10 @@
     "Env",
     "SingleAgentEnv",
     "EnvConfig",
-    "EnvServerConfig",
+    "ServingConfig",
+    "LegacyEnvConfig",
+    "resolve_env_field",
+    "narrowed_env_annotation",
     "SingleAgentEnvConfig",
     "AgentConfig",
     "StaticPoolConfig",

verifiers/v1/agent.py

@@ -276,9 +276,9 @@ def __init__(
             max_total_tokens=config.max_total_tokens,
         )
         self.timeout = config.timeout
-        # Env episode agents replace this with the eval concurrency semaphore.
-        # Interactions acquire it only around active lifecycle work, never while
-        # awaiting the caller between segments.
+        # Env episode agents replace this with the episode's agent semaphore
+        # (`--env.max-concurrent-agents`). Interactions acquire it only around active
+        # lifecycle work, never while awaiting the caller between segments.
         self._gate: asyncio.Semaphore | None = None
         # Env-owned standing, not config: `Env.setup` marks fixed agents
         # untrainable and traces are stamped from here; inert outside an env.
@@ -578,9 +578,9 @@ class _EpisodeAgent(Agent):
     bundle of references — expensive resources are env-owned and borrowed, so no
     state spans concurrent episodes): traces get their agent standing the moment
     they're created, finished ones land in `completed` (the episode's traces),
-    each run takes the eval's gate. The taskset's shared tool servers ride only
-    its own tasks — on an env-minted task they'd wrongly put MCP in play
-    (`tools=` overrides)."""
+    each run takes one of the episode's agent permits. The taskset's shared tool
+    servers ride only its own tasks — on an env-minted task they'd wrongly put MCP
+    in play (`tools=` overrides)."""
 
     def __init__(
         self,
@@ -664,8 +664,9 @@ async def interaction(
         """The agent's `interaction`, with every trace stamped with its standing
         at mint and captured in `completed` at close — an interaction driven from
         `Env.run` stays crash-safe. Setup, each active segment, and close acquire
-        the eval gate independently; the interaction holds no permit while awaiting
-        its caller, so peer interactions can interleave even at concurrency one."""
+        an agent permit independently; the interaction holds no permit while awaiting
+        its caller, so peer interactions still interleave where an episode plays one
+        agent at a time."""
         trace: Trace | None = None
 
         def remember(current: Trace) -> None:

verifiers/v1/cli/dashboard/eval.py

@@ -95,7 +95,7 @@ def _limits(config: EvalConfig) -> list[str]:
             toks.append(f"{label}≤{v}")
     turns = _seat_value(config, lambda spec: spec.max_turns)
     return [
-        f"≤{config.max_concurrent} concurrent"
+        f"≤{config.max_concurrent} episodes"
         if config.max_concurrent
         else "no concurrency cap",
         "per-seat turn caps"