feat(gym): route rollouts through an externally managed vLLM router

[aoshen02]

Summary

• When env.nemo_gym.router_url is set, register each vLLM replica with an externally managed vLLM Router (POST /workers blocks until the router's own health probe succeeds, so a 200 means registered and routable) and point Gym's policy_base_url at the router. Unset keeps today's direct-to-replica behavior.
• Add GET /health to the async worker's HTTP server, which the router uses to health-probe replicas at registration.

Complements #3518 (managed-router lifecycle owned by NeMo RL) with the external-router mode: the router can be started and owned outside the trainer, and NeMo RL only registers its replicas.

Workload

Deterministic replay throughout: recorded coding-agent trajectories replayed turn by turn, so every arm sees a byte-identical request sequence. Context accumulates naturally through the multi-turn loop — turn t re-sends everything from turns 1..t-1, which is what makes session→replica affinity worth anything at all. NVIDIA-Nemotron-3-Super-120B-A12B-BF16, TP2 vLLM engines, greedy decoding with ignore_eos.

Two datasets, drawn from the same trace pool and differing only in how the tail is weighted:

	p30_all	longtail
Sessions	610	600
Turns	19,582	10,308
Turns per session (min / p50 / p90 / p99 / max)	5 / 29 / 56 / 85 / 99	8 / 13 / 20 / 78 / 95
Cumulative context per session, p50	6.28M chars	2.03M chars
Cumulative context per session, max	50.7M chars	50.7M chars
max / p50	8.1x	25.0x
Top 1% of sessions, share of total work	4.6%	9.6%
Top 5%	18.2%	36.5%
Top 10%	31.2%	56.0%

longtail keeps the same longest session (identical replay_turns) and thins the middle: 540 short sessions (8-20 turns) carrying 44% of the work, plus 60 long ones (27-95 turns) carrying 56%. It is the harder workload for a router, not the bigger one — total work is lower, concentration is 3x higher.

Experiment results

Zero preemptions and zero HTTP 500s in every run below.

16 nodes / 28 engines, p30_all, max_concurrent_rollouts: 112

Arm	Rollout wall	Prefix-cache hit rate	Per-engine load (req)
cache_aware	18:57	93.5%	505-942 (1.9x)
consistent_hash	20:26	93.5%	446-1100 (2.5x)
none (direct to engines)	22:03	85.6%	562-828

Code under test: this PR's commits on NeMo-RL upstream daf46ff3 (2026-08-06), with NVIDIA-NeMo/Gym#2570's commits on Gym upstream 473f446f (2026-07-24). 16 GB200 nodes = 2 training + 14 generation, 2 engines/node. max_num_batched_tokens 8480, gpu_memory_utilization 0.9.

8 nodes / 12 engines, p30_all, max_concurrent_rollouts: 0 (uncapped)

8 GB200 nodes = 2 training + 6 generation, 2 engines/node; all other engine settings unchanged.

Arm	Rollout wall	Prefix-cache hit rate
cache_aware	13:57	91.6%
consistent_hash	15:39	91.8%
none (direct to engines)	15:51	91.8%

Workload counters are identical across the three (1,570,738,525 prompt tokens, 0 preemptions).

One asymmetry worth stating plainly: the two routed arms ran on Gym base 473f446f, whose vllm_model picks a replica per session with a per-uvicorn-process counter. The none arm ran on that base plus Gym's stable assignment, sha256(session_id) % len(clients), which is what current Gym main does. Running none on the counter instead does not reproduce this row. With router_url set the routed arms hold a single client (policy_base_url is the router alone), so that picker is inert for them and the comparison stands.

8 nodes / 12 engines, longtail, max_concurrent_rollouts: 0 (uncapped)

Same topology and engine settings as the row above; the dataset is the only change. All three arms on the same eight nodes, chained through --dependency so no two share hardware in time.

Arm	Rollout wall	Time to 90% done	Prefix-cache hit	queries/prompt	Per-engine load	Load CV
cache_aware	09:31	218s	90.5%	1.13	728-1008 (1.38x)	8.6%
cache_aware (repeat)	09:41	235s	90.6%	1.13	—	—
consistent_hash	10:06	263s	88.2%	1.20	647-1110 (1.72x)	16.4%
none (direct to engines)	10:47	263s	88.5%	1.19	618-1044 (1.69x)	13.1%

Identical request counts across all four (10,320 each).

Run-to-run noise floor, from the repeat: 1.8% on wall clock, 7.8% on time-to-90%.
cache_aware's 11.7% is six times the wall-clock floor, so it is a result rather than a
hypothesis. The repeat also settles a question we could not answer before: 09:31's final
session finished in 4s against none's 67s, which looked like placement luck, but the
repeat lands at 09:41 — the tail is short because everything before it finished sooner,
not because of where one session landed.

cache_aware's advantage reproduces: 11.7% over none here, 12.0% on p30_all — across a 3x change in tail concentration.

Its mechanism does not reproduce, and the earlier explanation was dataset-specific. On p30_all, cache_aware was the lowest hit rate of the three (91.6%) and we attributed the win to first-turn placement. On longtail it is the highest (90.5% vs 88.2 / 88.5%) and also the most evenly loaded (CV 8.6% vs 16.4 / 13.1%). With context per session growing faster, returning a session to the replica that already holds its prefix starts paying in hit rate directly. Same 12%, two different sources.

Wall clock is the wrong headline metric on this workload, and the completion curve says why. Time to reach N sessions done, relative to none:

Sessions done	none	consistent_hash	cache_aware
100	137s	134s (+2.0%)	117s (+14.6%)
300	180s	179s (+0.6%)	156s (+13.1%)
500	234s	236s (-0.9%)	202s (+13.8%)
540 (90%)	263s	263s (0.0%)	218s (+17.1%)
585	489s	505s (-3.3%)	457s (+6.5%)
599	580s	602s (-3.8%)	567s (+2.2%)
600	647s	606s (+6.3%)	571s (+11.7%)

consistent_hash is behind none for the first 599 sessions and passes it only on the last one: the final session took 67s under none and 4s under both routed arms. That is not a policy effect, it is where the single 25x-outlier session happened to land once the fleet had drained. cache_aware's lead, by contrast, is established by session 100 and holds through 90%, and the repeat run reproduces the wall clock to within 1.8% — so unlike consistent_hash's, it is not a last-session effect.

Read both columns. Time to 90% separates the policies on mechanism (218s vs 263s vs 263s, and it does not hinge on one session's placement), but the repeat shows it is the noisier of the two here — 7.8% run-to-run against wall clock's 1.8%. We had assumed the opposite from first principles and were wrong; noise character is a property of the configuration, not of the metric, so it has to be measured per campaign rather than argued.

What the tail costs, and what routing cannot do about it

Measured on the longtail runs, from the per-engine /metrics timeline sampled every 30s:

	none	consistent_hash	cache_aware
Last 10% of sessions, share of wall clock	59.4%	56.6%	61.8%
Last 5%	35.2%	29.5%	32.4%
Last 1%	14.2%	11.4%	10.0%
Turns finished in the first half of wall clock	82%	82%	82%
Wall clock spent below half of peak throughput	77%	76%	79%
Peak fleet throughput	48 turns/s	47 turns/s	54 turns/s

The last 10% of sessions take ~60% of the wall clock, and this is identical across all three policies — cache_aware has the highest tail share (61.8%) precisely because it shortened everything else.

The reason is not that tail sessions are slow but that there is nothing left to parallelise. Concurrency measured on the same runs falls from ~530 in-flight requests at peak to 43 at t=451s and 2 at t=785s, while a single session is capped at ~0.45 turns/s by serial decoding. Saturating these 12 engines needs 150-220 concurrent requests; the tail runs one to two orders of magnitude below that. No routing policy changes this — it is a question of what the idle capacity is reconfigured to do, which is #3520's direction, not this PR's.

Caveats

• The noise floor was measured for cache_aware only; none and consistent_hash are assumed to sit in the same range rather than measured.
• The p30_all 12-engine cache_aware row is likewise a single run.
• All longtail jobs exit non-zero: the harness interrupts after the rollout, and the 120B training step starts before the interrupt lands and OOMs. Ordering is rollout complete → metrics snapshot written → OOM, so measurement is unaffected. The published p30_all runs exit the same way.

Full report, acceptance scripts, and trimmed evidence runs: https://github.com/aoshen02/RL/blob/feat/gym-router-url/experiments/routing/README.md

Testing

• End-to-end in the campaigns above.
• python -m py_compile on touched files; no unit tests added yet — happy to add router-registration unit tests if maintainers want this direction.

AI assistance was used for this work; the submitting human has reviewed every line.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.