feat(nemo-gym): route rollouts through vLLM Router - #3518
Conversation
a471870 to
e6fd6e4
Compare
| ) | ||
| vllm_model_config = dict( | ||
| responses_api_models_config.get("vllm_model") or {} | ||
| ) |
There was a problem hiding this comment.
maybe we can simiplify to:
from omegaconf import DictConfig, OmegaConf
if self._vllm_router is not None:
initial_global_config_dict = OmegaConf.merge(
initial_global_config_dict,
{"policy_model": {"responses_api_models": {
"vllm_model": {"session_affinity_header": "X-Session-ID"}}}},
)
There was a problem hiding this comment.
Thanks, simplified this with OmegaConf.merge in 26549e6. I kept an explicit DictConfig cast because OmegaConf.merge can return either a dict or list config. Existing nested values are preserved, and the unit test verifies that num_workers remains alongside session_affinity_header.
| "--worker-urls", | ||
| *self.worker_base_urls, | ||
| "--policy", | ||
| self.config.policy, |
There was a problem hiding this comment.
The intended header is X-Session-ID (with "Session"). The vLLM Router consistent-hash policy reads x-session-id directly. I removed --request-id-headers x-session-id in 26549e6 because that option configures request-ID propagation, while Gym continues to send X-Session-ID for affinity.
| self.worker_base_urls = [ | ||
| base_url.rstrip("/").removesuffix("/v1") for base_url in worker_base_urls | ||
| ] | ||
| self.host = host |
There was a problem hiding this comment.
I think we can also support https://github.com/smg-project/smg, should be no further todo needed.
26549e6 to
df930bf
Compare
1ee6e4f to
8dc2ead
Compare
8dc2ead to
ae455c6
Compare
Signed-off-by: NolenLiang <nliang@nvidia.com>
ae455c6 to
4e7fc2a
Compare
github-actions
Bot
added
the
Documentation
Add reproducible NeMo Gym rollout benchmarks, cache-aware and consistent-hash routing, Prometheus/RL-Insight evidence, and paired Phase 2 reporting. Co-authored-by: Jonas Yang <joyang@nvidia.com> Signed-off-by: Nolen Liang <nliang@nvidia.com>
32d4fbc to
4380042
Compare
2 participants
Summary
Behavior and product decision
The routing feature remains disabled by default. When enabled, NeMo RL starts the Router after the vLLM worker URLs are available, reserves the Router HTTP/metrics ports, waits for readiness, registers dedicated Prometheus targets, and routes Gym policy requests through the selected policy. Startup failure and shutdown clean up the owned Router and monitoring processes.
All NeMo Gym environments resolve the pinned
vllm-router==0.1.15dependency group. This is an intentional product tradeoff for a single reproducible Gym runtime, even when routing is disabled; it is not an accidental lockfile side effect.External audit: measurement contract
The formal matrix uses identical replay inputs within each repeat and changes only the routing arm:
Qwen/Qwen2.5-1.5B-Instructat revision989aa7980e4cf806f80c7fef2b1adb7bc71aa306The repository contains the reproduction and audit tooling under
experiments/nemo_gym_phase2/,tools/nemo_gym_phase2_report.py, andtools/nemo_gym_phase2_compare.py. Generated reports include manifests and SHA-256 inventories. Raw cluster logs and Prometheus snapshots are retained in the external run bundle rather than committed to the source tree.External audit: formal result
All six accepted single-run reports and all ten paired-matrix gates passed.
Paired against Direct over 512 common outcomes:
The evidence demonstrates auditable cache-aware behavior and a 36.85% Router cache-hit rate, but it does not demonstrate a statistically significant accuracy improvement or a cache-aware p99 improvement in this matrix. Consistent-hash has the lowest observed mean request p99.
Publication provenance
The formal runs recorded NeMo RL
755c0033ec1e82ccfbb155090477cde0bee5c314and Gymf075ce8b23ce872642444e762fce38a96fef37f3. Publication history was later rewritten solely to correct the Git author identity and DCO trailers:f075ce8->4d6b55a, with identical source treee0807bfc5c40cf282c07b33aa8178c124dda4e9f32d4fbc->438004203; all non-submodule content is identical, and the only tree difference is the gitlink to the tree-equivalent Gym commit aboveNolen Liang <nliang@nvidia.com>; Jonas Yang attribution is retained withCo-authored-byThe post-run RL change only normalizes volatile
uv sync --checkstdout/stderr when comparing run compatibility. Each source report, raw invariant hash, lock hash, package inventory, command, return code, workload, topology, and artifact checksum remains audited.Validation
git diff --check