feat: multiplex interception servers + tunnels across rollouts

[mikasenghaas]

Summary

Behind a remote runtime, each rollout's interception endpoint needs its own tunnel, and tunnel creation is rate-capped per API token — so one-tunnel-per-rollout caps remote fan-out, and at scale the per-rollout tunnel cost (creation pacing folds into each rollout's generation.duration, plus tunnel-infra strain) inflates latency. This makes interception multiplexable: many rollouts share one server + one tunnel. Rebased directly onto feat/nano-as-v1, this PR also folds in the prime tunnel rate-limiter and the v1 single-turn runtime benchmark that motivated the change (previously #1593, now superseded).

Multiplexing

• Split InterceptionServer into a per-rollout RolloutSession (its trace/limits/stops/user) + a shared, secret-routed server ({secret: session}). The harness is unchanged — it already authenticates with its per-rollout secret, which is exactly what the server routes by.
• New InterceptionPool (mirrors Environment.shared_tools): brings up ceil(concurrency / multiplex) shared servers, exposes each once (one tunnel per server, via a host-side exposer runtime), and hands each rollout a session slot — shared endpoint + its own secret. The pool is elastic (servers grown on demand) and lives for the eval/server lifetime, so tunnels are reused across requests.
• First-class EnvConfig.multiplex (ge=1, default 32) — inherited by both the eval CLI (--multiplex) and the env server, so prime-rl, which drives the server, benefits too. multiplex=1 = a server+tunnel per rollout. Drops remote tunnels O(N) → O(N/multiplex). (interception split into interception/{server,pool}.py; PooledServer is public.)

Prime tunnels

• Rate-limit prime tunnel creation to stay under the per-token cap, and bump the limiter 100 → 512 (Prime raised the cap). modal still has no tunnel limiter at all — separate fix.

Benchmark (bench/)

• General single-turn runtime benchmark: benchmark.sh (providers × batch sizes) → aggregate.py → committed benchmark.json → plot.py (image gitignored). Compares per-rollout generation-duration p10/p50/p90 (e2e wall clock is gated by a single endpoint-queue straggler).

Results — prime, gsm8k-v1 (capped max_tokens=2048, 512 tunnel limiter)

Per-rollout generation-duration p50/p90 (s) — baseline (tunnel/rollout) vs --multiplex 32. (e2e wall clock is gated by a single endpoint-queue straggler, so the distribution is the robust comparator.)

n	baseline	mux=32	mux=64
32	22 / 25	19 / 25	20 / 26
64	25 / 36	20 / 29	20 / 26
128	47 / 67	23 / 36	34 / 41
256	67 / 117	38 / 62	38 / 59

Tunnels at n=256: baseline 256, mux=32 8, mux=64 4. Multiplexing roughly halves p50/p90 from 128 up; multiplex=32 is the sweet spot — mux=64 ties it at 256 but is worse at 128 (p50 34 vs 23: 64 rollouts through 2 tunnels vs 32 through 4), so fewer-but-busier tunnels don't help. 0 errors across all runs.

Verification

• Correctness (subprocess + prime): baseline path, pooled single-turn, pooled colocated-tool (2 turns), pooled multi-turn user-sim (3–5 turns) — all reward ~1.0, 0 errors. Prime smoke: 4 rollouts → 1 shared server, 1 tunnel.

Notes

• Default is now 32 (on); multiplex=1 reproduces the per-rollout baseline (the benchmark's "baseline" column).
• Supersedes exp: v1 runtime benchmark (subprocess vs docker vs prime) #1593 — the runtime benchmark + prime tunnel rate-limiter are folded in here.

Note

Multiplex interception servers and tunnels across concurrent rollouts

• Refactors InterceptionServer in server.py from single-rollout to multi-session, routing requests by per-rollout Bearer token secrets via a new RolloutSession dataclass.
• Adds InterceptionPool in pool.py to share interception servers and tunnel endpoints across rollouts up to a configurable multiplex count (default 32), growing the pool on demand.
• Threads the pool through Environment.interception_pool(), EnvServer, Episode.run, run_with_retry, and Rollout.run; rollouts fall back to per-rollout servers when no pool is provided.
• Adds a leaky-bucket rate limiter in prime.py to cap tunnel creation at 512/min per API token, preventing provider 429s under high concurrency.
• Adds benchmarking scripts (bench/benchmark.sh, bench/aggregate.py, bench/plot.py) for measuring and visualizing rollout throughput across runtimes and batch sizes.

^{Macroscope summarized 1fc6bcb.}

---

Note

Medium Risk
Changes the default eval/env-server rollout path (multiplex=32) and shared-secret routing on the interception server; mis-pooling or auth bugs could cross-wire rollouts, while Prime tunnel pacing affects all concurrent remote exposes.

Overview
Adds shared interception so many concurrent rollouts reuse one localhost proxy (and, on Prime, one tunnel) instead of provisioning per rollout—addressing remote fan-out limits and straggler-heavy e2e latency at scale.

InterceptionServer is split into per-rollout RolloutSession state and a server that registers sessions under bearer secrets and routes /v1/chat/completions by token. InterceptionPool grows ceil(N / multiplex) servers on demand, exposes each once via a host-side exposer runtime, and acquire hands rollouts a shared endpoint plus their own secret. EnvConfig.multiplex defaults to 32 (1 restores one server/tunnel per rollout); the eval runner and env server enter the pool for the whole run and pass it through episode → retry → rollout (optional pool; no pool keeps the old per-rollout expose path). LegacyEnvServer overrides the pool to a no-op.

Prime tunnel creation is paced with a process-wide AsyncLimiter at 512/min (capacity 1) so burst pool growth stays under the per-token cap.

New bench/ tooling runs single-turn evals across runtimes × batch sizes, aggregates per-rollout generation.duration (plus e2e, reward, errors) into benchmark.json, and plots p10/p50/p90 with plot.py.

^{Reviewed by Cursor Bugbot for commit 1fc6bcb. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1fc6bcb. Configure here.}

[cursor]

Pool load counter race

Medium Severity

In InterceptionPool.acquire, entry.load is incremented under _lock but decremented in finally without the lock. Concurrent rollouts finishing together can lose decrements, so load drifts below the number of active sessions and more than multiplex rollouts can register on one shared server.

 

^{Reviewed by Cursor Bugbot for commit 1fc6bcb. Configure here.}

[cursor]

Missing multiplex documentation

Low Severity

This PR adds first-class EnvConfig.multiplex (default 32) for eval and the env server, but no updates appear in docs/ (for example docs/evaluation.md or docs/reference.md) describing the flag or when to set multiplex=1.

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit 1fc6bcb. Configure here.}

[cursor]

Missing skills multiplex update

Low Severity

The new multiplex eval/env-server knob is not reflected in workflow skills such as skills/evaluate-environments/SKILL.md, which document eval and env-server configuration patterns.

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit 1fc6bcb. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a new multiplexing feature for interception servers with significant infrastructure changes affecting concurrent rollout behavior. An unresolved medium-severity review comment identifies a potential race condition in the pool's load counter that could allow over-subscription of shared servers.

^{You can customize Macroscope's approvability policy. Learn more.}