build: bump vllm 0.20.0 → 0.25.1

[terrykong]

What does this PR do ?

Bumps vLLM from 0.20.0 to 0.25.1 (supersedes the 0.24.0 draft #3137). Rebased onto current main (bc382dc); all validation below is against that base.

Changes

Bullets are prefixed with the commit that made the change, as of head 69a3e5a.

Dependencies (pyproject.toml / uv.lock)

• a34945a — vLLM wheel URLs → v0.25.1 (manylinux_2_28, changed from manylinux_2_35 in 0.24+)
• a34945a — flashinfer-python/-cubin/-jit-cache → 0.6.13 in the vllm extra (pinned by vLLM 0.25.1's requirements/cuda.txt)
• a34945a — xgrammar override relaxed ==0.1.33 → >=0.2.1,<1.0.0 (vLLM 0.25 imports xgrammar.normalize_tool_choice, added in 0.2.x; resolver picks 0.2.3)
• a34945a — openai stays at 2.6.1 (nemo-gym pins openai<=2.7.2 and its child server venvs must exactly match the parent's openai version, so a global override is not viable). vLLM 0.25.1's import of openai.types.responses.NamespaceTool (added in openai 2.25.0) is handled by a new source patch instead — see below.

Async OpenAI-compatible server (vllm_worker_async.py)

• a34945a — vLLM 0.25 moved chat preprocessing from OpenAIServingRender (module deleted) to OnlineRenderer (vllm/renderers/); both /v1/chat/completions and /tokenize still route through preprocess_chat, so the NeMo-RL prefix-token override moves to an OnlineRenderer subclass
• a34945a — OpenAIServingTokenization → ServingTokenization (takes online_renderer, no engine_client)
• a34945a — preprocess_chat's tool_parser/reasoning_parser args were folded into a single parser arg upstream
• a34945a — Max-context-length log filter now targets vllm.entrypoints.openai.chat_completion.serving (module logger name)

fp8 generation (quantization/fp8.py)

• a34945a — FusedMoE is a factory function in vLLM 0.25 (returns MoERunner delegating to a weight-owning RoutedExperts submodule) — fp8 weight classification now detects MoERunner/RoutedExperts
• a34945a — Linear block-quant patch keeps the scale under weight_scale_inv (0.25's Fp8BlockScaledMMLinearKernel forward path reads weight_scale_inv, no more rename to weight_scale)
• a34945a — make_fp8_moe_kernel call updated to 0.25 signature (routing_tables=layer._expert_routing_tables(), layer=layer; shared_experts kwarg removed upstream)

vLLM source patches (patches.py)

• a34945a — ADDITIONAL_ENV_VARS no longer exists in vLLM 0.25; env propagation is prefix-based (NCCL_*, HF_*, HUGGING_FACE_* auto-copied — covers the PR fix: fix async vllm nccl fail on dsv3 tp16pp2 and non-colocated on single node #898 NCCL workaround) plus the additive VLLM_RAY_EXTRA_ENV_VARS_TO_COPY hook, which we now set for RAY_ENABLE_UV_RUN_RUNTIME_ENV + user extra_env_vars
• a34945a — Dropped the Hermes tool-parser thread-safety patch: vLLM 0.25 no longer runs tokenizer encode/decode in Hermes2ProToolParser.__init__, which was the race the patch guarded
• a34945a — New patch _patch_vllm_tool_parser_namespace_tool: guards vLLM's NamespaceTool import (tool_parsers/utils.py) with a never-matching stub for openai < 2.25. NamespaceTool is only isinstance-checked for Responses-API namespace tools, which an openai 2.6.1 client cannot construct
• a34945a — py_executable runtime_env patch and llama_eagle3 lm_head patch verified to still apply cleanly against the 0.25.1 wheel

Refit race fix (nemo_rl/utils/packed_tensor.py)

• f49ce0c — vLLM 0.25's PyNcclCommunicator enqueues broadcasts on the current stream without blocking, exposing a latent race: packed_broadcast_producer/consumer returned while the final broadcast / unpack+load_weights copies were still in flight on their side streams. In non-colocated refit this let generation race with weight loading (token_mult_prob_error blew up to ~1e5–1e6 in grpo_non_colocated.sh; transport checksums were verified byte-identical, so it was purely an ordering bug). The join itself is no longer this PR's change: main landed the same for s in streams: s.synchronize() independently via feat(generation): add TensorRT-LLM rollout backend #2420 while this branch was in flight, so after the rebase f49ce0c contributes only the comments explaining why the join is required — which main's version lacks. Kept for that reason.

Tests

• a34945a — test_vllm_generation.py: mock modules follow the new layout (vllm.renderers.online_renderer); test_vllm_http_server golden response updated for 0.25.1 schema (empty tool_calls omitted on serialization, reasoning_content removed, new routed_experts/prompt_text/metrics fields)

ModelOpt quantization (nemo_rl/modelopt/...)

• ec8d7df — _drop_nonclass_quant_registry_keys(): purges non-class keys from ModelOpt's QuantModuleRegistry before the fakequant prolog (vLLM 0.25 registers factory functions there, which broke issubclass checks)
• ec8d7df — Eager input_amax_loader attach for fakequant refit: vLLM 0.25's per-module LinearBase.load_weights calls param.weight_loader directly and never iterates named_parameters(), so the old lazy shim never fired ('Tensor' object has no attribute 'weight_loader')
• ec8d7df — MoE fakequant ported to the RoutedExperts layout: ModelOpt registers its MoE quant module on FusedMoE, which 0.25 turned into a factory function, so no MoE quantizers were inserted at all. Registering on RoutedExperts (which owns the expert weights in 0.25) puts the quantizer buffers exactly where the expert mapping resolves incoming input_quantizer._amax names, and the kernel swap wraps forward_modular/forward_monolithic since RoutedExperts.forward asserts against being called. The swap also has to patch experts.triton_moe, which binds invoke_fused_moe_triton_kernel at import time — patching only fused_moe.fused_moe leaves the executing callsite on the original kernel and fakequant silently never engages.
• 773b485 — Gated MoE real-quant refit loads per expert: RoutedExperts.load_weights' fused-3D branch infers orientation by comparing the last dim against the unpadded hidden size, which packed NVFP4 weights (K/2 uint8) and block scales (K/16) never match — it transposed and re-chunked them, tripping the layerwise-reload numel assert. Gated models now emit per-expert 2-D shards (the same loader path the initial disk load uses); non-gated models keep the batched layout, since they load through the model's own expert loop and its heuristic-free 3-D full_load path.
• 61347c8 — W4A16 NVFP4 rollout: 0.25 installs LinearMethodCls as an instance attribute keyed off the quant algo, shadowing the NeMo subclass override (W4A16 silently ran the W4A4 linear method); 0.25's Marlin converter also pads MoE tiles natively, so the NeMo-side pre-padding double-padded and tripped upstream shape asserts.
• 9d3841c — MXFP8 rollout: FusedMoeWeightScaleSupported moved out of fused_moe.layer (hard ImportError in both MoE refit helpers). The linear-refit path also has to force the CUTLASS kernel: 0.25 prefers FlashInferCutedslMxfp8LinearKernel, which stores the weight column-major [K, N], while our refit-friendly process_weights_after_loading override (and the MXFP8 refit loader) keep the canonical [N, K] layout — leaving CuTe-DSL selected made the forward pass read K as the output dim (shape '[-1, 6144]' is invalid for input of size 16777216).
• cedf21c — Sharded NIXL expert refit: ported to the 0.25 MoE layout — expert weights now live on the nested RoutedExperts submodule (parameter names gain a .routed_experts. segment), and TP/EP metadata moved from the layer to moe_config.

Fixes found during CI validation (H100/GB200 nightly + DSv3 perf babysitting)

• ba47f53 — TCPStore/MessageQueue port collision for node-spanning engines: vLLM 0.25's RayExecutorV2 probes NeMo-RL's per-engine VLLM_PORT for the torch TCPStore while the cross-node broadcast MessageQueue allocates from the same range between probe and bind → deterministic EADDRINUSE at engine startup for any engine spanning nodes (DSv3 TP=32, qwen3.5-397B TP=16). Fix: offset only the TCPStore search to VLLM_PORT + 32, keeping both ports inside the engine's existing 100-port window. configure_worker's port assignment is unchanged from main — see the reviewer note below on why unsetting VLLM_PORT was rejected. Confirmed on H100 hardware: distributed_init_method=tcp://…:7032, zero EADDRINUSE (was 4). Fixing it then exposed a second, independent port bug at a different call site — see "Results — performance (DSv3)".
• 9c894c4 — fp8 refit memory churn: the 0.25 port of our patched Fp8LinearMethod.process_weights_after_loading rebound weight/weight_scale_inv storage on every refit (0.20 copied in place), fragmenting device memory across sleep/wake cycles until CuMemAllocator wake-up OOM'd mid-run in the fp8-rollouts nightlies. Restored in-place copies once the processed layout is stable.
• add423a — fp8 wake-up OOMs — root-caused and fixed: at an identical gpu_memory_utilization, vLLM 0.25 sizes moonlight's KV cache to 34.29 GiB where 0.20 sized it to 19.83 GiB (sleep pool 49.96 vs ~36 GiB), so the pool can no longer be re-mapped at wake-up next to the colocated Megatron policy. This is why utilization tuning kept failing (0.5 still OOM'ed) and why cudagraph mode was a red herring (PIECEWISE captured 0.45 GiB and still OOM'ed). Fix: pin vllm_kwargs.kv_cache_memory_bytes: 20 GiB (the 0.20-proven size, which takes precedence over gpu_memory_utilization) on the fp8-rollouts and moonlight fp8-e2e recipes. All three fp8 tests now pass their full step budgets.
• 068cd6c — No CI thresholds are relaxed. An earlier revision raised the ppo_automodel critic grad-norm ceiling 350 → 1500 and the eval_async score ceiling 0.14 → 0.2. Both are reverted — see commit 20. All metric bounds now match main.
• 7f118df — test_vllm_megatron_pipeline_parallel timeout 120s → 240s. It began failing intermittently on wall clock, not on an assertion. Measured call time: 103.80s on vLLM 0.20 (PR perf: avoid CPU round trip during NVFP4 refit #3308, job 90163013717) vs 113.10s on 0.25 (this branch, job 89878378208) — ~9.3s / +9% slower engine init, cutting headroom under the old budget from 16.2s to 6.9s, which is less than run-to-run variance on a shared runner. The budget was already at 86% on 0.20. Both measurements are recorded at the marker.
• bb2dd26 — A second port collision, exposed by fixing the first: with the TCPStore no longer taking port 7000, startup got one step further and every RayWorkerProc on a non-driver node then raced for the same VLLM_PORT building its worker_response_mq (ZMQError: Address already in use, ×563). MessageQueue.__init__ probes-and-releases a port then ZMQ-binds it several statements later, and _init_message_queues runs right after init_device()'s collective barrier, so the workers on a node arrive together. Fix: retry at the bind, advancing past the lost port, which also covers the executor's own rpc_broadcast_mq. Byte-identical on vLLM main, so upstream is affected too. Reproduced with no GPU and no second node.
• dde85d3 — Refit staging buffer could not hold DSv3's embedding: the IPC staging buffers are sized from free memory with no floor at the largest parameter, so model.embed_tokens.weight (1.73 GiB) exceeded its 1.65 GiB buffer and the refit aborted on an assertion. Such a parameter is now shipped alone in a buffer sized to fit, mirroring the HTTP streaming path. Pre-existing on main and unrelated to vLLM — it fails there too on 0.20. The new branch is reachable only where the old code raised, so working refits are byte-for-byte unaffected.
• 1a37190, 1827620 — Two defects in this PR's own port tests, both found by CI: the "pristine" fixture was copying an already-patched vLLM (the patch mechanism rewrites site-packages in place, so the fixture was order-dependent), and the negative-control asserted one environment's port number — CI runs as root, where unpatched vLLM returns 100 rather than falling through to VLLM_PORT. Now reverses the patch to build the fixture, and asserts the invariant instead of a number.
• 69a3e5a — uv.lock relocked against the recorded Gym submodule commit: 3rdparty/Gym-workspace/Gym is a tool.uv.workspace member, so a checkout at the wrong commit made uv lock resolve Gym's dependencies from the wrong tree and silently drop 20 packages (506 locked vs 526 correct), failing the container build on both platforms at uv lock --check. Note uv lock --check passes locally against a dirty submodule — it is only as good as the workspace it reads.

Notes for reviewers

• Two corrections worth calling out, both squashed into the commit they fix and covered by unit tests: (a) this PR's original node-spanning VLLM_PORT guard could never evaluate true, so the earlier claim that the DSv3 port collision was validated-fixed was wrong; (b) the follow-up that dropped VLLM_PORT for those engines would have sent them to kernel-ephemeral ports, regressing the TOCTOU protection from fix: configure port ranges to avoid TOCTOU port contention #2380/fix: Port contention issues between Ray/vLLM/Gym and sandbox workers #3103 — the port assignment is now unchanged from main and the collision is fixed by offsetting only the TCPStore search inside the engine's existing window. Details under "Results — performance (DSv3)".

• Dependency note from the rebase onto main: this PR raises the xgrammar override floor to >=0.2.1,<1.0.0, because vLLM 0.25.1 imports xgrammar.normalize_tool_choice. main's override was xgrammar==0.1.33, which also happened to override tensorrt-llm's own xgrammar>=0.2.1 requirement — so once the floor rises, tensorrt-llm==1.3.0rc21 (added to main after this branch's merge base) becomes unsatisfiable: it pins apache-tvm-ffi==0.1.6, xgrammar 0.2.1–0.2.3 need >=0.1.9, and xgrammar ≥0.2.4 ships no cp313 wheels. Resolved by adding an apache-tvm-ffi>=0.1.9 override, following the existing llguidance precedent. This moves tensorrt-llm off its pinned tvm-ffi — worth a TRT-LLM owner's eyes, though narrower than it first looks: TRT-LLM's own dependency already declares apache-tvm-ffi>=0.1.6,!=0.1.8,!=0.1.8.post0,<0.2, so 0.1.9 is explicitly in range rather than untested, and the !=0.1.8 exclusions are the known-bad release that 0.1.9 fixed. tensorrt-llm is also in no-build-isolation-package, so build-time and runtime tvm-ffi are both 0.1.9, ruling out header/ABI skew. Verified uv lock succeeds against main's own pyproject.toml in the same tree, so the conflict is genuinely introduced by the bump and not a stale lockfile.

  A GB200 run of the two trtllm nightly recipes (nemo-ci#59844796) failed, but not on this override: both died with ModuleNotFoundError: No module named 'tensorrt_llm', with zero mentions of tvm_ffi anywhere — the package was absent, so the override was never exercised. The runtime worker venv comes from uv run --locked --extra trtllm (virtual_cluster.py:80) and must compile tensorrt_llm from its path source; the wheel cache is set only in tests/unit/L0_Unit_Tests_Trtllm.sh, not in the recipes. This PR touches neither file. That points to pre-existing, but it is not proven — these two recipes have no green baseline recorded on this branch or on main, so a main control run is still needed to classify it.

• Bottom line: both platforms are at parity-or-better vs main. Every test that failed at head was re-run on the rebase base (bc382dc, plain main + vLLM 0.20) to classify it. Two regressions were found and fixed (moonlight fp8-e2e on H100, nvfp4 distillation on GB200); three tests now pass at head that fail on main (both fp8-rollouts variants, w4a4-real). Everything else that still fails, fails identically on main.

• The bump fixes 2 of the 4 disabled.txt qwen3.5 entries (both AutoModel variants); this PR re-enables them and raises the nightly GPU-hour cap to 3600 to fit them. main independently raised the same cap 3420 → 3480 for its own additions, so the rebase had to accommodate both: the merged nightly measures 3595 hours over 114 tests (5 h headroom). The Megatron and 397B entries stay disabled for reasons unrelated to this bump. Full per-test breakdown in the Test plan below.

• fp8 refit path validated on the fp8 nightly recipes; upstream replace_parameter now preserves weight_loader, so a follow-up could drop parts of the fp8 monkeypatch entirely.

• DSv3 32n perf had three independent blockers, all three now fixed and confirmed on H100 hardware — the test passes, which it has never done at any commit including plain main. Two were 0.25 port bugs at different call sites (the second only became visible once the first was fixed), and the third is a pre-existing NeMo-RL refit bug that also fails on main with 0.20. Detail under "Results — performance (DSv3)".

Test plan

Per Yuki's guidance on #3137: (1) all H100 / GB200 nightly tests, (2) one DSv3 perf run per platform, (3) the qwen3.5 runs from disabled.txt.

Status against the three requested items

#	Requested	Status	Detail
1	All (or most of) the H100 / GB200 nightly tests	✅ Done — parity-or-better	Full suite run on both SKUs (100 H100 + 28 GB200). Every failure at head was re-run on the rebase base to classify it. 2 bump regressions found and fixed; 3 tests now pass at head that fail on main; every other failure reproduces identically on main. Per-test table below.
2	One DSv3 perf run on both H100 and GB200	✅ H100 passes (first time ever); GB200 queued	The test has no green baseline at any commit — it is red on plain main too. Root-causing it turned up three independent failures: two vLLM 0.25 port bugs at different call sites (RayExecutorV2's TCPStore vs each RayWorkerProc's response queue — the second only visible once the first was fixed, and both present on vLLM main upstream), plus a pre-existing NeMo-RL refit-buffer bug that also fails on 0.20. All three are fixed on this branch and H100 now passes end-to-end (nemo-ci#375685907, 42.8 min, reached step 10): zero EADDRINUSE, zero ZMQError, zero refit-buffer assertions, Generation KL Error 0.0019–0.0022. GB200 is queued for cluster resources. See "DSv3 perf" below.
3	The qwen3.5 runs from disabled.txt	✅ 2 of 4 fixed and re-enabled; 2 stay disabled for non-bump reasons	The bump fixes both AutoModel variants (validated passing post-rebase), and this PR now re-enables them in nightly.txt and raises the nightly GPU-hour guardrail 3480 → 3600 to fit them. The Megatron geo3k variant still hangs — but so does its already-enabled Megatron sibling on plain main, so that is the pre-existing qwen3.5+Megatron+EP hang, not something this bump regressed or promised to fix. The 397B test turned out not to be a vLLM problem at all — it was failing because Qwen/Qwen3.5-397B-A17B had never been seeded into CI's offline HF cache. The model has now been seeded and verified to resolve offline (all 94 shards), and the recipe moves out of disabled.txt into release.txt (one run costs 1024 GPU-hours, too large for nightly).

3 — qwen3.5 disabled.txt, test by test

Test (in disabled.txt)	Fixed by the bump?	Evidence	Why it is still disabled
grpo-qwen3.5-35ba3b-2n8g-automodel-ep16	✅ yes	passes post-rebase, nemo-ci#370922142/nemo-ci#59133030	Re-enabled in this PR
vlm_grpo-qwen3.5-35ba3b-geo3k-2n8g-automodel-ep16	✅ yes	passes post-rebase, nemo-ci#370922174/nemo-ci#59133030	Re-enabled in this PR
vlm_grpo-qwen3.5-35ba3b-geo3k-2n8g-megatron-ep16	❌ no	hangs twice — TimeoutError: RPC call to sample_tokens timed out — nemo-ci#370922175/nemo-ci#59133030 and nemo-ci#371188276/nemo-ci#59178648	Still broken. Not a bump regression: the already-enabled sibling grpo-qwen3.5-35ba3b-2n8g-megatron-ep16tp2cp2 hangs identically on plain main (nemo-ci#371232795/nemo-ci#59178656) and on this branch (nemo-ci#371188273/nemo-ci#59178648). This is the pre-existing qwen3.5 + megatron + EP hang.
grpo-qwen3.5-397ba17b-32n8g-megatron.v2	n/a — was never a vLLM failure	OSError: We couldn't connect to 'https://huggingface.co' ... and couldn't find them in the cached files (nemo-ci#367597981/nemo-ci#58665710)	Root-caused and unblocked. CI runs offline, so a model only resolves if it is already in the shared HF cache — and this one had never been seeded there (the Qwen3.5 35B and 9B models had been). It therefore could not have passed on any vLLM version. The model is now seeded (752 GiB, all 94 shards, verified to resolve with local_files_only=True), and the recipe moves to release.txt rather than nightly: at 32 nodes × 8 GPUs × 4 h it costs 1024 GPU-hours per run, which would grow the nightly suite by 29% on its own.

Nightly GPU-hour guardrail raised to fit them. Each re-enabled test costs 64 GPU-hours (2 nodes × 8 GPUs × 4 h). After rebasing onto main — which had independently raised the cap to 3480 for its own additions — the merged nightly measures 3595 hours over 114 tests, so the cap moves to 3600 (test_nightly_compute_stays_below_3600_hours), leaving 5 h headroom.

Note the total is order-dependent: tools/launch evals every script's CONFIG block into one shell and resets only SEGMENT_SIZE/JOB_REAPER_COMMENT, so NUM_RUNS leaks between scripts. The two distillation-xtoken-off-policy-* entries define none and inherit 1 from the preceding script, contributing 6 h that a naive per-script reading counts as 0. Pre-existing and not changed here, but it means reordering nightly.txt can shift the measured total. The Qwen3.5-397B recipe moves to release.txt instead (1024 GPU-hours/run), and the Megatron geo3k recipe stays disabled, so this is the only nightly cost increase in the PR.

Results — nightly

Legend: ✅ pass · ❌ fail · — not run. Sorted by suite, then by failure status (fixed/better-than-main → still failing → flaky → clean), then by SKU. Run references are nemo-ci#<job-id>/nemo-ci#<pipeline-id>.

Test	Pass on branch	Pass on main	Run (branch)	Run (main)	Notes
llm_grpo_llama3_1_8b_instruct_1n8g_megatron_fp8_rollouts_v3 H100 · nightly	✅	❌	nemo-ci#371937317/nemo-ci#59283710	nemo-ci#371232800/nemo-ci#59178656	Better than main. Same wake-up OOM class; fails on main too. Two fixes: fp8 linear weights are copied in place on refit (the 0.25 port rebound storage every refit, fragmenting device memory across sleep/wake), plus the same kv_cache_memory_bytes pin. Passes the full 100 steps.
llm_grpo_llama3_1_8b_instruct_1n8g_megatron_fp8_rollouts_v3_tq_simple H100 · nightly	✅	❌	nemo-ci#371937320/nemo-ci#59283710	nemo-ci#371232807/nemo-ci#59178656	Better than main. Same fixes and outcome as the _v3 variant above.
llm_grpo_moonlight_16ba3b_4n8g_megatron_fp8_e2e H100 · nightly	✅	✅	nemo-ci#371937319/nemo-ci#59283710	nemo-ci#371232802/nemo-ci#59178656	Fixed regression. Only H100 test that passed on main but failed at head. vLLM 0.25 sizes this KV cache to 34.3 GiB where 0.20 sized it to 19.8 GiB at the same gpu_memory_utilization, growing the CuMemAllocator sleep pool past what can be re-mapped at wake-up next to the colocated Megatron policy (CUDA Error: out of memory at cumem_allocator.cpp). Fix: pin vllm_kwargs.kv_cache_memory_bytes: 20 GiB on the recipe.
llm_distillation_nano3_30ba3b_4n4g_megatron_qa_nvfp4_modelopt_spec GB200 · nightly	✅	✅	nemo-ci#373274921/nemo-ci#59466552	nemo-ci#372686041/nemo-ci#59378561	Fixed regression. Only GB200 test that passed on main but failed at head. Two causes, both fixed: (1) ModelOpt registers MoE fakequant on FusedMoE, which 0.25 turned into a factory function, so no MoE quantizers were inserted at all and refit hit KeyError: ...routed_experts.w13_input_quantizer._amax — now registered on RoutedExperts, where the expert weights live; (2) the fakequant kernel swap only patched fused_moe.fused_moe, but 0.25's modular TritonExperts binds invoke_fused_moe_triton_kernel into experts.triton_moe at import time, so the executing callsite kept the original kernel and fakequant silently never engaged (quantizers collected no amax) — the swap now covers both namespaces.
llm_grpo_qwen3_30ba3b_4n4g_megatron_qa_nvfp4_w4a4_real GB200 · nightly	✅	❌	nemo-ci#373045624/nemo-ci#59425270	nemo-ci#372686039/nemo-ci#59378561	Better than main. Crashed during real-quant refit on 0.25: RoutedExperts.load_weights' fused-3D branch infers tensor orientation by comparing the last dim to the unpadded hidden size, which packed NVFP4 weights (K/2 uint8) and block scales (K/16) never match, so it transposed and re-chunked them and tripped the layerwise-reload numel assert. Fix: emit per-expert 2-D shards for gated MoEs, the same loader path the initial disk load uses. Now passes including convergence; main fails convergence (gen_kl 0.0166 vs <0.003).
llm_grpo_llama3_1_8b_instruct_2n8g_megatron_fp8_e2e H100 · nightly	❌	❌	nemo-ci#371188280/nemo-ci#59178648	nemo-ci#371232801/nemo-ci#59178656	Pre-existing. Fails identically on main — not a bump regression.
llm_grpo_llama3_1_8b_instruct_2n8g_megatron_fp8_e2e_tq_mooncake H100 · nightly	❌	❌	nemo-ci#371188287/nemo-ci#59178648	nemo-ci#371232808/nemo-ci#59178656	Pre-existing. Fails identically on main — not a bump regression.
llm_grpo_nemotron3_super_120BA12B_16n8g_megatron H100 · nightly	❌	❌	nemo-ci#371188285/nemo-ci#59178648	nemo-ci#371232806/nemo-ci#59178656	Pre-existing. Fails identically on main — not a bump regression.
llm_grpo_qwen3_1_7b_6n8g_megatron_super_rlvr H100 · nightly	❌	❌	nemo-ci#371188284/nemo-ci#59178648	nemo-ci#371232805/nemo-ci#59178656	Pre-existing. Fails identically on main — not a bump regression.
llm_grpo_qwen3_30ba3b_4n8g_megatron_qa_nvfp4 H100 · nightly	❌	❌	nemo-ci#371188277/nemo-ci#59178648	nemo-ci#371232798/nemo-ci#59178656	Pre-existing. Fails identically on main — not a bump regression.
llm_grpo_qwen3_30ba3b_4n8g_megatron_zmq_deltaweight_noncolocated H100 · nightly	❌	❌	nemo-ci#371188283/nemo-ci#59178648	nemo-ci#371232804/nemo-ci#59178656	Pre-existing. Fails identically on main — not a bump regression.
llm_grpo_qwen3_30ba3b_thinking_swe1_16n8g_megatron_cp2_r3_async_gym H100 · nightly	❌	❌	nemo-ci#371188278/nemo-ci#59178648	nemo-ci#371232799/nemo-ci#59178656	Pre-existing. Fails identically on main — not a bump regression.
llm_grpo_qwen3_5_35ba3b_2n8g_megatron_ep16tp2cp2 H100 · nightly	❌	❌	nemo-ci#371188273/nemo-ci#59178648	nemo-ci#371232795/nemo-ci#59178656	Pre-existing. Fails identically on main — not a bump regression.
llm_grpo_qwen3_8b_base_1n8g_fp8_kvcache_megatron H100 · nightly	❌	❌	nemo-ci#371188282/nemo-ci#59178648	nemo-ci#371232803/nemo-ci#59178656	Pre-existing. Fails identically on main — not a bump regression.
vlm_vlm_grpo_qwen3_5_35ba3b_geo3k_2n8g_megatron_ep16 H100 · nightly	❌	—	nemo-ci#371188276/nemo-ci#59178648	— (not run)	No main control (not part of the failing set re-run on main). Hangs in sample_tokens (RPC timeout) — the same qwen3.5-family hang as llm_grpo_qwen3_5_35ba3b_2n8g_megatron_ep16tp2cp2, which does reproduce on main. Passed on 0.25 before the rebase (nemo-ci#367597980/nemo-ci#58665710), so this reads as the pre-existing qwen3.5 hang rather than a bump regression.
vlm_vlm_grpo_qwen3_omni_30ba3b_audiomcq_4n8g_megatron_v1 H100 · nightly	❌	❌	nemo-ci#371188275/nemo-ci#59178648	nemo-ci#371232797/nemo-ci#59178656	Pre-existing. Fails identically on main — not a bump regression.
llm_grpo_nanov3_30ba3b_4n4g_megatron_qa_nvfp4_w4a16_real GB200 · nightly	❌	❌	nemo-ci#372317452/nemo-ci#59311151	nemo-ci#372686038/nemo-ci#59378561	Pre-existing. Same convergence failure and nearly the same magnitude on both sides (branch gen_kl 0.0107 / token_mult 1.092; main 0.0119 / 1.096). Kept change: the test script's log assertion was relaxed to quantization=(nemo_)?modelopt because 0.25 prints the registered NeMo quant name (nemo_modelopt_w4a16_nvfp4); without it the test failed on a log-string check while the run itself was healthy.
llm_grpo_nemotron3_super_120BA12B_8n4g_megatron GB200 · nightly	❌	❌	nemo-ci#372317455/nemo-ci#59311151	nemo-ci#372686040/nemo-ci#59378561	Pre-existing. Fails on main with a Ray worker OOM. First branch attempt was an external Slurm cancellation; the rerun reproduces the main-side failure.
llm_prorlv2_qwen2_5_math_1_5b_instruct_1n8g_fsdp2tp1_v2_tq_mooncake H100 · nightly	✅	✅	nemo-ci#371188288/nemo-ci#59178648	nemo-ci#371232809/nemo-ci#59178656	Flaky. Failed once on the branch, passed on retry; passes on main. No code change.
vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n8g_megatrontp2_v1 H100 · nightly	✅	✅	nemo-ci#371188274/nemo-ci#59178648	nemo-ci#371232796/nemo-ci#59178656	Flaky. Failed once on the branch, passed on retry; passes on main. No code change.
llm_grpo_llama3_2_1b_instruct_1n4g_fsdp2tp1_v3 GB200 · nightly	✅	—	nemo-ci#372317451/nemo-ci#59311151	— (not run)	Flaky. 2.6% perf-threshold miss with healthy convergence; passed on retry. No main control run. No code change.

Remaining 107 nightly tests — passed on branch, no main control run needed (84 H100 · 23 GB200)

Test	Pass on branch	Pass on main	Run (branch)	Run (main)	Notes
llm_dapo_gemma4_e2b_it_1n8g_fsdp2_automodel H100 · nightly	✅	—	nemo-ci#370922237/nemo-ci#59133030	— (not run)	—
llm_distillation_qwen3_1_7b_1n8g_megatron_qa_nvfp4 H100 · nightly	✅	—	nemo-ci#370922262/nemo-ci#59133030	— (not run)	—
llm_distillation_qwen3_32b_to_1_7b_base_1n8g_fsdp2tp1_v1 H100 · nightly	✅	—	nemo-ci#370922260/nemo-ci#59133030	— (not run)	—
llm_distillation_qwen3_32b_to_1_7b_base_1n8g_megatron_tp2pp2cp2_pack H100 · nightly	✅	—	nemo-ci#370922261/nemo-ci#59133030	— (not run)	—
llm_dpo_llama3_1_8b_instruct_4n8g_megatrontp2pp2_quick H100 · nightly	✅	—	nemo-ci#370922258/nemo-ci#59133030	— (not run)	—
llm_dpo_llama3_2_1b_instruct_1n8g_fsdp2tp1_v2 H100 · nightly	✅	—	nemo-ci#370922256/nemo-ci#59133030	— (not run)	—
llm_dpo_mistral_nemo_instruct_2407_1n8g_fsdp2tp8_actckpt_long H100 · nightly	✅	—	nemo-ci#370922259/nemo-ci#59133030	— (not run)	—
llm_dpo_nanov3_30B3AB_1n8g_fsdp8ep8_automodel H100 · nightly	✅	—	nemo-ci#370922257/nemo-ci#59133030	— (not run)	—
llm_dpo_qwen2_5_math7b_1n8g_megatron_fused_linear_logprobs H100 · nightly	✅	—	nemo-ci#370922250/nemo-ci#59133030	— (not run)	—
llm_gdpo_qwen2_5_1_5b_1n8g_gym_multireward H100 · nightly	✅	—	nemo-ci#370922267/nemo-ci#59133030	— (not run)	—
llm_grpo_cispo_mm1_async_lag1_highoffpolicy_qwen3_30ba3b_3n8g_megatron_cispo H100 · nightly	✅	—	nemo-ci#370922186/nemo-ci#59133030	— (not run)	—
llm_grpo_deepscaler_1_5b_16K H100 · nightly	✅	—	nemo-ci#370922177/nemo-ci#59133030	— (not run)	—
llm_grpo_deepscaler_1_5b_24K H100 · nightly	✅	—	nemo-ci#370922179/nemo-ci#59133030	— (not run)	—
llm_grpo_deepscaler_1_5b_8K H100 · nightly	✅	—	nemo-ci#370922181/nemo-ci#59133030	— (not run)	—
llm_grpo_deepscaler_1_5b_8K_tq_simple H100 · nightly	✅	—	nemo-ci#370922222/nemo-ci#59133030	— (not run)	—
llm_grpo_gemma3_1b_it_1n8g_fsdp2tp1_tq_simple H100 · nightly	✅	—	nemo-ci#370922221/nemo-ci#59133030	— (not run)	—
llm_grpo_gspo_deepscaler_1_5b_8K H100 · nightly	✅	—	nemo-ci#370922182/nemo-ci#59133030	— (not run)	—
llm_grpo_gspo_deepscaler_1_5b_8K_tq_simple H100 · nightly	✅	—	nemo-ci#370922223/nemo-ci#59133030	— (not run)	—
llm_grpo_llama3_1_8b_instruct_2n8g_fsdp2tp1_noncolocated H100 · nightly	✅	—	nemo-ci#370922197/nemo-ci#59133030	— (not run)	—
llm_grpo_llama3_1_8b_instruct_2n8g_fsdp2tp1_noncolocated_tq_simple H100 · nightly	✅	—	nemo-ci#370922226/nemo-ci#59133030	— (not run)	—
llm_grpo_llama3_2_1b_instruct_1n8g_fsdp2tp1_v3_tq_simple H100 · nightly	✅	—	nemo-ci#370922220/nemo-ci#59133030	— (not run)	—
llm_grpo_llama3_2_1b_instruct_1n8g_fsdp2tp2_temp0_8_topp0_9_topk50 H100 · nightly	✅	—	nemo-ci#370922216/nemo-ci#59133030	— (not run)	—
llm_grpo_llama3_2_1b_instruct_1n8g_fsdp2tp2_temp0_8_topp0_9_topk50_tq_mooncake H100 · nightly	✅	—	nemo-ci#370922230/nemo-ci#59133030	— (not run)	—
llm_grpo_llama3_2_1b_instruct_1n8g_megatron_generation_tq_mooncake H100 · nightly	✅	—	nemo-ci#370922229/nemo-ci#59133030	— (not run)	—
llm_grpo_llama3_2_1b_instruct_1n8g_megatron_temp0_8_topp0_9_topk50 H100 · nightly	✅	—	nemo-ci#370922217/nemo-ci#59133030	— (not run)	—
llm_grpo_llama3_2_1b_instruct_1n8g_megatron_temp0_8_topp0_9_topk50_tq_mooncake H100 · nightly	✅	—	nemo-ci#370922231/nemo-ci#59133030	— (not run)	—
llm_grpo_llama3_2_1b_instruct_1n8g_megatron_tq_simple H100 · nightly	✅	—	nemo-ci#370922224/nemo-ci#59133030	— (not run)	—
llm_grpo_llama3_2_1b_instruct_2n8g_megatron_generation_noncolocated H100 · nightly	✅	—	nemo-ci#370922198/nemo-ci#59133030	— (not run)	—
llm_grpo_math_qwen3_30ba3b_megatron_tp4_32k H100 · nightly	✅	—	nemo-ci#370922184/nemo-ci#59133030	— (not run)	—
llm_grpo_moonlight_16ba3b_4n8g_megatron H100 · nightly	✅	—	nemo-ci#370922140/nemo-ci#59133030	— (not run)	—
llm_grpo_moonlight_16ba3b_4n8g_megatron_tq_simple H100 · nightly	✅	—	nemo-ci#370922227/nemo-ci#59133030	— (not run)	—
llm_grpo_nano_v2_12b_1n8g_megatron H100 · nightly	✅	—	nemo-ci#370922200/nemo-ci#59133030	— (not run)	—
llm_grpo_nano_v2_12b_2n8g_fsdp2tp1 H100 · nightly	✅	—	nemo-ci#370922201/nemo-ci#59133030	— (not run)	—
llm_grpo_nanov3_30BA3B_2n8g_fsdp2 H100 · nightly	✅	—	nemo-ci#370922202/nemo-ci#59133030	— (not run)	—
llm_grpo_nanov3_30BA3B_2n8g_fsdp2_lora H100 · nightly	✅	—	nemo-ci#370922203/nemo-ci#59133030	— (not run)	—
llm_grpo_nanov3_30BA3B_2n8g_fsdp2_tq_mooncake H100 · nightly	✅	—	nemo-ci#370922234/nemo-ci#59133030	— (not run)	—
llm_grpo_nanov3_30BA3B_2n8g_megatron_generation H100 · nightly	✅	—	nemo-ci#370922206/nemo-ci#59133030	— (not run)	—
llm_grpo_nanov3_30BA3B_2n8g_megatron_generation_async_gym H100 · nightly	✅	—	nemo-ci#370922207/nemo-ci#59133030	— (not run)	—
llm_grpo_nanov3_30BA3B_2n8g_megatron_lora H100 · nightly	✅	—	nemo-ci#370922204/nemo-ci#59133030	— (not run)	—
llm_grpo_nanov3_30BA3B_2n8g_megatron_pack_cp H100 · nightly	✅	—	nemo-ci#370922205/nemo-ci#59133030	— (not run)	—
llm_grpo_nanov3_30BA3B_2n8g_megatron_pack_cp_tq_simple H100 · nightly	✅	—	nemo-ci#370922228/nemo-ci#59133030	— (not run)	—
llm_grpo_nemotron3_super_120BA12B_16n8g_automodel_ep8 H100 · nightly	✅	—	nemo-ci#370922212/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen2_5_1_5B_4n8g_megatron_yarn_256k H100 · nightly	✅	—	nemo-ci#370922218/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen2_5_math_1_5b_instruct_1n8g_fsdp2tp1_v3_tq_simple H100 · nightly	✅	—	nemo-ci#370922219/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen3_1_7b_1n8g_megatron_eagle3_tq_mooncake H100 · nightly	✅	—	nemo-ci#370922232/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen3_1_7b_1n8g_megatron_super_swe1 H100 · nightly	✅	—	nemo-ci#370922209/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen3_1_7b_2n8g_megatron_super_rlhf H100 · nightly	✅	—	nemo-ci#370922210/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen3_30ba3b_10n8g_megatron_cp2_r3_async H100 · nightly	✅	—	nemo-ci#370922190/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen3_30ba3b_2n8g_megatron_fused_linear_logprobs H100 · nightly	✅	—	nemo-ci#370922251/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen3_30ba3b_8n8g_megatron_cp2_r3 H100 · nightly	✅	—	nemo-ci#370922188/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen3_30ba3b_8n8g_megatron_cp2_r3_tq_simple H100 · nightly	✅	—	nemo-ci#370922189/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen3_5_35ba3b_2n8g_automodel_ep16 H100 · nightly	✅	—	nemo-ci#370922142/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen3_5_9b_1n8g_megatron H100 · nightly	✅	—	nemo-ci#370922141/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen3_8B_base_1n8g_fsdp2_lora H100 · nightly	✅	—	nemo-ci#370922214/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen3_8B_base_1n8g_fsdp2_lora_tq_mooncake H100 · nightly	✅	—	nemo-ci#370922235/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen3_8b_base_1n8g_megatron_lora H100 · nightly	✅	—	nemo-ci#370922215/nemo-ci#59133030	— (not run)	—
llm_grpo_qwen3_8b_base_dapo_2n8g_long_megatron_qa_nvfp4_w4a16 H100 · nightly	✅	—	nemo-ci#370922196/nemo-ci#59133030	— (not run)	—
llm_mopd_qwen3_1_7b_3n8g_megatron_pack H100 · nightly	✅	—	nemo-ci#370922263/nemo-ci#59133030	— (not run)	—
llm_ppo_qwen2_5_1_5b_gsm8k_1n8g_automodel_valuetp2sp H100 · nightly	✅	—	nemo-ci#370922264/nemo-ci#59133030	— (not run)	—
llm_ppo_qwen2_5_1_5b_gsm8k_1n8g_megatron_valuetp2sp_dynbatch H100 · nightly	✅	—	nemo-ci#370922265/nemo-ci#59133030	— (not run)	—
llm_ppo_qwen2_5_1_5b_gsm8k_1n8g_megatron_valuetp2sp_pp2cp2_pack H100 · nightly	✅	—	nemo-ci#370922266/nemo-ci#59133030	— (not run)	—
llm_sft_gpt_oss_20b_1n8g_fsdp8ep8_automodel H100 · nightly	✅	—	nemo-ci#370922252/nemo-ci#59133030	— (not run)	—
llm_sft_llama3_1_8b_1n8g_fsdp2tp1_lora H100 · nightly	✅	—	nemo-ci#370922243/nemo-ci#59133030	— (not run)	—
llm_sft_llama3_1_8b_1n8g_fsdp2tp2 H100 · nightly	✅	—	nemo-ci#370922241/nemo-ci#59133030	— (not run)	—
llm_sft_llama3_1_8b_1n8g_fsdp2tp4_dynamicbatch H100 · nightly	✅	—	nemo-ci#370922242/nemo-ci#59133030	— (not run)	—
llm_sft_llama3_1_8b_1n8g_megatron H100 · nightly	✅	—	nemo-ci#370922246/nemo-ci#59133030	— (not run)	—
llm_sft_llama3_1_8b_1n8g_megatron_lora H100 · nightly	✅	—	nemo-ci#370922244/nemo-ci#59133030	— (not run)	—
llm_sft_llama3_1_8b_1n8g_megatron_seqpack H100 · nightly	✅	—	nemo-ci#370922247/nemo-ci#59133030	— (not run)	—
llm_sft_llama3_2_1b_1n8g_fsdp2tp1_v3 H100 · nightly	✅	—	nemo-ci#370922239/nemo-ci#59133030	— (not run)	—
llm_sft_llama3_2_1b_2n8g_hsdp H100 · nightly	✅	—	nemo-ci#370922240/nemo-ci#59133030	— (not run)	—
llm_sft_nanov3_30BA3B_2n8g_fsdp2 H100 · nightly	✅	—	nemo-ci#370922253/nemo-ci#59133030	— (not run)	—
llm_sft_nanov3_30BA3B_2n8g_fsdp2_lora H100 · nightly	✅	—	nemo-ci#370922254/nemo-ci#59133030	— (not run)	—
llm_sft_qwen2_5_32b_4n8g_fsdp2tp8sp_actckpt_v3 H100 · nightly	✅	—	nemo-ci#370922245/nemo-ci#59133030	— (not run)	—
llm_sft_qwen2_5_math7b_1n8g_megatron_fused_linear_logprobs H100 · nightly	✅	—	nemo-ci#370922249/nemo-ci#59133030	— (not run)	—
llm_sft_qwen2_5_math7b_2n8g_megatron H100 · nightly	✅	—	nemo-ci#370922248/nemo-ci#59133030	— (not run)	—
llm_sft_qwen3_0_6B_1n8g_megatron_yarn_128k H100 · nightly	✅	—	nemo-ci#370922255/nemo-ci#59133030	— (not run)	—
vlm_vlm_grpo_gemma4_e4b_geo3k_1n8g_automodel H100 · nightly	✅	—	nemo-ci#370922238/nemo-ci#59133030	— (not run)	—
vlm_vlm_grpo_nemotron_omni_30ba3b_clevr_1n8g_automodel_ep8_v1 H100 · nightly	✅	—	nemo-ci#370922158/nemo-ci#59133030	— (not run)	—
vlm_vlm_grpo_nemotron_omni_30ba3b_mmpr_4n8g_automodel_ep8_v1 H100 · nightly	✅	—	nemo-ci#370922167/nemo-ci#59133030	— (not run)	—
vlm_vlm_grpo_qwen2_5_omni_3b_avqa_1n8g_megatron_v1 H100 · nightly	✅	—	nemo-ci#370922147/nemo-ci#59133030	— (not run)	—
vlm_vlm_grpo_qwen2_5_omni_7b_audiomcq_1n8g_megatron_v1 H100 · nightly	✅	—	nemo-ci#370922148/nemo-ci#59133030	— (not run)	—
vlm_vlm_grpo_qwen2_5_omni_7b_intent_1n8g_megatron_v1 H100 · nightly	✅	—	nemo-ci#370922149/nemo-ci#59133030	— (not run)	—
vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n8g_dtensor2tp1_v1 H100 · nightly	✅	—	nemo-ci#370922145/nemo-ci#59133030	— (not run)	—
vlm_vlm_grpo_qwen3_5_35ba3b_geo3k_2n8g_automodel_ep16 H100 · nightly	✅	—	nemo-ci#370922174/nemo-ci#59133030	— (not run)	—
llm_distillation_qwen3_32b_to_1_7b_base_1n4g_fsdp2tp1_v1 GB200 · nightly	✅	—	nemo-ci#371771063/nemo-ci#59224654	— (not run)	—
llm_distillation_qwen3_32b_to_1_7b_base_1n4g_megatron_tp1pp2cp2_pack GB200 · nightly	✅	—	nemo-ci#371771064/nemo-ci#59224654	— (not run)	—
llm_dpo_llama3_1_8b_instruct_4n4g_megatrontp1pp2_quick GB200 · nightly	✅	—	nemo-ci#371771061/nemo-ci#59224654	— (not run)	—
llm_dpo_llama3_2_1b_instruct_1n4g_fsdp2tp1_v2 GB200 · nightly	✅	—	nemo-ci#371771058/nemo-ci#59224654	— (not run)	—
llm_dpo_nanov3_30B3AB_1n4g_fsdp4ep4_automodel GB200 · nightly	✅	—	nemo-ci#371771059/nemo-ci#59224654	— (not run)	—
llm_grpo_deepscaler_1_5b_1n4g_8K GB200 · nightly	✅	—	nemo-ci#371771040/nemo-ci#59224654	— (not run)	—
llm_grpo_gemma3_1b_it_1n4g_fsdp2tp1 GB200 · nightly	✅	—	nemo-ci#371771017/nemo-ci#59224654	— (not run)	—
llm_grpo_llama3_1_8b_instruct_2n4g_fsdp2tp1_noncolocated GB200 · nightly	✅	—	nemo-ci#371771042/nemo-ci#59224654	— (not run)	—
llm_grpo_llama3_2_1b_instruct_1n4g_megatron GB200 · nightly	✅	—	nemo-ci#371771020/nemo-ci#59224654	— (not run)	—
llm_grpo_llama3_2_1b_instruct_1n4g_megatron_generation GB200 · nightly	✅	—	nemo-ci#371771024/nemo-ci#59224654	— (not run)	—
llm_grpo_moonlight_16ba3b_4n4g_megatron GB200 · nightly	✅	—	nemo-ci#371771028/nemo-ci#59224654	— (not run)	—
llm_grpo_nano_v2_12b_1n4g_megatron GB200 · nightly	✅	—	nemo-ci#371771044/nemo-ci#59224654	— (not run)	—
llm_grpo_nano_v2_12b_2n4g_fsdp2tp1 GB200 · nightly	✅	—	nemo-ci#371771046/nemo-ci#59224654	— (not run)	—
llm_grpo_qwen2_5_7b_instruct_4n4g_fsdp2tp2_v3 GB200 · nightly	✅	—	nemo-ci#371771018/nemo-ci#59224654	— (not run)	—
llm_grpo_qwen2_5_math_1_5b_instruct_1n4g_fsdp2tp1_v3 GB200 · nightly	✅	—	nemo-ci#371771014/nemo-ci#59224654	— (not run)	—
llm_grpo_qwen3_1_7b_1n4g_megatron_eagle3 GB200 · nightly	✅	—	nemo-ci#371771022/nemo-ci#59224654	— (not run)	—
llm_sft_gpt_oss_20b_1n4g_fsdp4ep4_automodel GB200 · nightly	✅	—	nemo-ci#371771053/nemo-ci#59224654	— (not run)	—
llm_sft_llama3_2_1b_1n4g_fsdp2tp1_v3 GB200 · nightly	✅	—	nemo-ci#371771049/nemo-ci#59224654	— (not run)	—
llm_sft_nanov3_30BA3B_2n4g_fsdp2 GB200 · nightly	✅	—	nemo-ci#371771055/nemo-ci#59224654	— (not run)	—
llm_sft_nanov3_30BA3B_2n4g_fsdp2_lora GB200 · nightly	✅	—	nemo-ci#371771056/nemo-ci#59224654	— (not run)	—
llm_sft_qwen2_5_math7b_2n4g_megatron GB200 · nightly	✅	—	nemo-ci#371771051/nemo-ci#59224654	— (not run)	—
vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n4g_dtensor2tp1_v1 GB200 · nightly	✅	—	nemo-ci#371771036/nemo-ci#59224654	— (not run)	—
vlm_vlm_grpo_qwen2_5_vl_3b_instruct_clevr_1n4g_megatrontp1_v1 GB200 · nightly	✅	—	nemo-ci#371771038/nemo-ci#59224654	— (not run)	—

Results — performance (DSv3)

Neither platform passes, and neither did before this PR — the test is red at the base commit too, so there is no green baseline to regress from. Re-running on the current head produced a log that finally exposed the real blocker (earlier runs only showed a downstream symptom):

1. 0.25-specific port collision — root-caused and fixed. For an engine that spans nodes, RayExecutorV2 picks the torch TCPStore port with a bind-probe (Step 3) and releases it, then Step 4 builds the broadcast MessageQueue, which needs a real TCP socket and binds and holds a port from the same VLLM_PORT scan range (shm_broadcast.py: remote_subscribe_port = get_open_port() → remote_socket.bind(...)). The queue takes the very port the probe just released, and the rank-0 worker then dies with DistNetworkError: ... port: 7000 ... EADDRINUSE. Engines that fit on one node bind an ipc:// socket and never allocate a TCP port there, which is why only node-spanning engines are affected.

   Fix: offset only the TCPStore search to VLLM_PORT + 32, past the queue's scan range. Both ports stay inside the engine's 100-port window and therefore below the OS ephemeral floor. vLLM already applies the same disjoint-window idea to co-located DP engines a few lines below (master_port + 100 + rank * 32).

   Deliberately not fixed by unsetting VLLM_PORT. An earlier revision of this PR did that, which is wrong: with VLLM_PORT unset, _get_open_port() falls through to s.bind(("", 0)) — kernel-assigned ephemeral ports — reintroducing exactly the TOCTOU port contention the reserved port layout exists to prevent (fix: configure port ranges to avoid TOCTOU port contention #2380, fix: Port contention issues between Ray/vLLM/Gym and sandbox workers #3103), and only for the large node-spanning runs. configure_worker's port assignment is therefore unchanged from main. The patch also verifies its own result by reading the file back and warns if it did not land, since a silent no-op is what made an earlier attempt look ineffective.

2. Downstream symptom, expected to clear with (1). With the port stolen, startup fails in wait_for_engine_startup with Engine core initialization failed. See root cause above. Failed core proc(s): {} — an empty dict, and the EngineCore child's stderr is not captured in the CI job log, which is why earlier runs looked like an unexplained silent death. The EADDRINUSE above it was the missing "root cause".

3. A second 0.25 port bug, revealed by fixing the first. With the TCPStore no longer taking port 7000, startup got one step further and died in RayWorkerProc._init_message_queues with zmq.error.ZMQError: Address already in use (addr='tcp://…:7000'), repeated 563×. Same probe-then-bind shape, different consumer: MessageQueue.__init__ picks its remote-socket port with get_open_port() (which probes and releases), then ZMQ-binds it several statements later. Every RayWorkerProc on a non-driver node takes n_local_reader=0, so each needs a real TCP port and they all scan from the same VLLM_PORT. Because _init_message_queues runs immediately after init_device() — whose process-group setup is a collective barrier — the workers on a node reach the probe together, all see the port free, and all but one die. Workers on the driver node use ipc:// and take no TCP port, which is why only node-spanning engines are affected.

   Fix: retry at the bind rather than trusting the probe, advancing past the port that was lost. This terminates because a port a peer already holds with ZMQ is visible to the next probe, and it keeps ports anchored at VLLM_PORT instead of falling back to ephemeral. Patching the bind rather than handing each worker a private start port also covers every other MessageQueue with a remote reader, including the executor's own rpc_broadcast_mq. Reproduced and fixed with no GPU, no Ray and no second node — the mechanism is pure port arithmetic, so 8/32 concurrent processes against the real 0.25.1 MessageQueue reproduce it (unpatched 6/8 and 30/32 die; patched, all bind distinct in-band ports). Tracked as RL-1111. ⚠️ vLLM main carries byte-identical code, so this is an upstream bug too.

4. Pre-existing NeMo-RL refit bug — also fixed here. At the base commit on vLLM 0.20, the same test fails during refit with AssertionError: Parameter model.embed_tokens.weight too large for buffer: 1853358080 > 1771261132 (nemo_rl/models/policy/utils.py). The IPC staging buffers are sized from free memory (NRL_REFIT_BUFFER_MEMORY_RATIO, default 0.3, halved again for ping-pong) with no floor at the largest parameter, and DSv3's model.embed_tokens.weight is 129280 × 7168 bf16 = 1.73 GiB against a 1.65 GiB buffer — 4.6% over. Such a parameter is now shipped on its own in a buffer sized to fit, mirroring the HTTP streaming path which already gives an oversized parameter a bucket of its own. The new branch is reachable only where the old code raised AssertionError, so every refit that works today takes byte-for-byte the same path. Unrelated to vLLM; it fails on plain main.

Test	Pass on branch	Pass on main / base	Run (branch)	Run (main / base)	Notes
llm_performance_grpo_deepseek_v3_32n8g H100 · performance	✅	❌	nemo-ci#375685907/nemo-ci#59818076 (all three fixes)	nemo-ci#58698490 (base commit, vLLM 0.20)	Passes in 42.8 min, reaching step 10 with Generation KL Error 0.0019–0.0022 and refits completing across all 256 ranks. Zero EADDRINUSE, zero ZMQError, zero too large for buffer, zero tracebacks. Prior heads showed EADDRINUSE on 7000 (1), then after fixing it ZMQError ×563 (3). Base-commit control fails during refit on the buffer assertion (4).
llm_performance_grpo_deepseek_v3_32n4g GB200 · performance	⏳ queued	— (not run)	nemo-ci#59818142 (all three fixes)	—	The prior GB200 attempt (nemo-ci#374559159) was void — it died during git checkout with remote did not send all necessary objects, 0 Slurm submissions — so GB200 has no datapoint on fixes 2–4 yet.

Status: H100 passes. All three blockers are fixed and confirmed together on hardware — the run reached step 10 with healthy on-policy metrics and none of the three failure signatures. Blockers 2 and 3 had only unit tests until this run; blocker 3 in particular is a pre-existing main bug unrelated to vLLM, so this PR does not merely restore parity on DSv3, it turns a test that was red before the bump green. GB200 is queued behind cluster capacity and remains unconfirmed; the mechanism is architecture-independent port arithmetic plus pure-Python buffer handling, but that is inference, not evidence.

Why one branch: the refit fix (4) was briefly a separate PR against main, since it is a pre-existing bug unrelated to the bump. It was folded in here because validating DSv3 needs all three fixes together, and keeping them apart would have meant running the DSv3 perf suite twice on two SKUs for no extra signal. If this PR slips, that fix is the piece worth re-splitting out and landing on its own.

Results — release suite

Not run — the review guidance scoped validation to nightly + targeted perf.

Local validation (2× RTX 6000 Ada)

• uv lock / uv sync --extra vllm resolve and install cleanly; all ported modules import against the real 0.25.1 wheel; file patches apply cleanly; pre-commit (ruff, pyrefly, taplo) passes
• Unit: test_vllm_http_server and test_vllm_http_server_correct_merged_tokens_matches_baseline pass end-to-end against a real 0.25.1 engine (plus the mock-based async-server tests)
• Functional (vLLM-touching subset runnable locally): grpo.sh ✅ (gen KL 0.0005–0.0007, limit 0.002) · grpo_multiturn.sh ✅ (1.05, limit 1.1) · gdpo_async_grpo.sh ✅ (0.00065, limit 0.001) · grpo_non_colocated.sh ❌→✅ after the packed-broadcast stream-sync fix (1.015, limit 1.05) — measured against the pre-rebase base, which genuinely lacked the join; today's main has it via feat(generation): add TensorRT-LLM rollout backend #2420, so this is evidence the race is real, not evidence of a fix unique to this PR · eval.sh ✅ (0.1333, expected [0.1, 0.14)) · Megatron-backed suites and test_decode_vs_prefill.sh (needs 80GB) deferred to CI runners

Open follow-ups (none block this PR)

1. DSv3 32n perf end-to-end: all three blockers are fixed; the runs in the table above are the outstanding confirmation. The "silent EngineCore death" reported in earlier revisions of this description was never a separate bug — it was the uncaptured EngineCore stderr hiding the EADDRINUSE.
2. Nightly coverage for node-spanning engines: zero of the 140 nightly tests run an engine with tp*pp > cluster.gpus_per_node, which is the only reason two separate startup bugs shipped undetected — all 17 such recipes live in the performance/release suites. Both bugs now have cheap unit-level reproductions, but a ≤2-node functional test would close the class properly.
3. qwen3.5 Megatron + EP hang: vlm_grpo-qwen3.5-35ba3b-geo3k-2n8g-megatron-ep16 and the enabled grpo-qwen3.5-35ba3b-2n8g-megatron-ep16tp2cp2 both hang in sample_tokens on main and on this branch. Pre-existing and independent of this PR; the latter currently burns its full 4 h wall clock failing every night, so it is a candidate for retirement.
4. qwen3.5-397B: the model is now seeded into CI's offline cache and the recipe sits in release.txt, so it can be validated on the next release run. It has never actually been exercised in CI, so its first run should be treated as new-test bring-up rather than a regression signal.
5. Upstream vLLM (RL-1111): the MessageQueue remote-socket probe-then-bind race is byte-identical on vLLM main, so the bind-retry change is worth proposing upstream. RL-1104 and RL-1111 carry the full root-cause write-ups, reference runs and rejected approaches.
6. Upstream vLLM: file the RayExecutorV2 TCPStore port-selection TOCTOU (probe in _init_executor vs later bind in the rank-0 worker, colliding with the broadcast MessageQueue's get_open_port() scan when VLLM_PORT is set and the engine spans nodes).
7. Upstream vLLM / ModelOpt: RoutedExperts.load_weights' fused-3D orientation heuristic is unsound for packed (sub-byte) quantized expert weights; ModelOpt's vLLM plugin also needs a first-party port to the 0.25 MoERunner/RoutedExperts layout so downstreams don't have to re-register it.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[terrykong]

/ok to test bf5fa58

[terrykong]

/ok to test b253775

[terrykong]

/ok to test 8034a9f

[terrykong]

/ok to test 66284ce

[terrykong]

/ok to test f4f2986

[terrykong]

/ok to test 49f3449

[terrykong]

/ok to test 992ac5b

[terrykong]

/ok to test 251949e

[terrykong]

nemo-ci validation runs (per Yuki's guidance on #3137)

Triggered at head 251949eb7 (GitHub CI L1 fully green at this SHA — all 33 unit shards + all 30 functional suites on H100 & GB200; only the intentional pinned-base check is red):

Coverage	Pipeline
H100 nightly (full suite)	nemo-ci#58665697
GB200 nightly (full suite)	nemo-ci#58665698
H100 DSv3 perf (grpo-deepseek-v3-32n8g)	nemo-ci#58665705
GB200 DSv3 perf (grpo-deepseek-v3-32n4g)	nemo-ci#58665707
qwen3.5 disabled.txt tests (via throwaway branch terryk/bump-vllm-0.25.1-qwen35ci)	nemo-ci#58665710

Results will be compared against reference pipeline nemo-ci#56454576.

---

🤖 Generated with Claude Code

[terrykong]

Correction: the two GB200 runs were retriggered — the original triggers were misconfigured on my end and generated no tests.

Updated GB200 runs:

• GB200 nightly: nemo-ci#58674214 (replaces #58665698)
• GB200 DSv3 perf: nemo-ci#58674216 (replaces #58665707, canceled)

---

🤖 Generated with Claude Code

[terrykong]

GB200 runs retriggered once more: the previous attempts hit a CI infra issue that killed every test job at startup, before any RL code ran — independent of this PR.

Final validation set:

• H100 nightly: nemo-ci#58665697 (100 test jobs running)
• H100 DSv3 perf: nemo-ci#58665705
• qwen3.5 disabled-tests: nemo-ci#58665710
• GB200 nightly: nemo-ci#58677937
• GB200 DSv3 perf: nemo-ci#58677940

---

🤖 Generated with Claude Code

[terrykong]

/ok to test a213ac5

[terrykong]

DSv3 perf: deterministic startup failure found & fixed (a213ac534)

llm_performance_grpo_deepseek_v3_32n8g (H100) failed twice with the same error at vLLM engine init:

torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address.
port: 20001 ... EADDRINUSE

Root cause (vLLM 0.25 RayExecutorV2._init_executor): Step 3 picks the torch TCPStore port by probing from VLLM_PORT (NeMo-RL assigns 20001 + engine_idx*100 per engine) but only binds it later in the rank-0 worker's init_process_group. In between, Step 4 creates the broadcast MessageQueue; for an engine that spans nodes its remote ZMQ socket also allocates via get_open_port(), which scans from the same VLLM_PORT — so it binds the exact port the TCPStore probe just returned. Engines that span nodes (DSv3 gen TP=32 → 4 H100 nodes/engine) hit EADDRINUSE deterministically; single-node engines use a shm-only MessageQueue, which is why every other test is unaffected.

Fix: new source patch _patch_vllm_ray_executor_v2_tcpstore_port (in nemo_rl/models/generation/vllm/patches.py) makes _select_tcpstore_port search from VLLM_PORT+32, keeping the TCPStore window disjoint from the MessageQueue's scan range while staying inside the 100-port per-engine spacing. (Upstream vLLM has the same TOCTOU when VLLM_PORT is set; may be worth filing.)

Retriggered with the fix:

• DSv3 perf H100: nemo-ci#58683457
• DSv3 perf GB200: nemo-ci#58683459 (queued pipeline on the old SHA was canceled)
• GitHub CI (L1) rerun on a213ac534: run 29700226964

[terrykong]

/ok to test 5f563d7

[terrykong]

/ok to test 47d835e

[terrykong]

fp8-rollouts: GPU memory creep found & fixed (47d835e89)

Both H100 nightly fp8-rollouts variants (...megatron_fp8_rollouts_v3 and ..._tq_simple) died with:

CUDA Error: out of memory at /workspace/csrc/cumem_allocator.cpp:163

at vLLM wake-up — at step 74/100 and step 87/100 respectively (then the step stalls and the job hits the Slurm wall clock, so it reports as TIMEOUT). Reference (vLLM 0.20) and today's main-branch nightly pass these, and non-fp8 GRPO tests on this PR pass, so this was a real bump regression isolated to the fp8 refit path.

Cause: NeMo-RL calls process_weights_after_loading after every refit. The 0.25 port of our patched Fp8LinearMethod.process_weights_after_loading rebound layer.weight.data / layer.weight_scale_inv.data to the fresh tensors returned by process_fp8_weight_block_strategy on every refit (the 0.20 version copied the scale in place). ~226 linear layers × 100 refit cycles of free/alloc churn interleaved with sleep/wake slowly fragments device memory until CuMemAllocator can't restore its pool on wake.

Fix: copy into the existing parameter storages when the processed layout is stable (every refit after the first); only rebind on the first call when layout transforms change shapes. This restores 0.20's in-place behavior and removes all steady-state per-refit churn.

Validation: nemo-ci#58693850 reruns both fp8-rollouts variants on 47d835e89. (The in-flight retry of the original job on the old SHA is left running as a determinism control — expected to OOM again ~step 75.)

[terrykong]

/ok to test db34114

[terrykong]

/ok to test 61c66de

[terrykong]

nemo-ci validation summary — vLLM 0.20 → 0.25.1 (head 61c66de71)

Validation per the review guidance on #3137: H100+GB200 nightly, one DSv3 perf run per platform, and the qwen3.5 disabled.txt tests. Reference for comparison: nemo-ci#56454576 (same base commit 26effe27).

Item	Verdict
H100 nightly	✅ 31 failed vs 41 on the reference — zero unexplained deltas
GB200 nightly	✅ 22/24 pass — both failures pre-existing/environmental
qwen3.5 disabled tests	✅ 3/4 now pass on 0.25.1 (confirms the bump fixes vllm-project/vllm#36237); 397B blocked by an infra gap
DSv3 perf (H100+GB200)	⚠️ test is also broken at the base commit — no green baseline exists; 0.25-specific startup blocker was found & fixed, one further init issue needs on-cluster debugging
GitHub CI (L1)	✅ green on 61c66de71 — only reds are the intentional pinned-base check + its aggregate

H100 nightly — nemo-ci#58665697 vs reference

• 12 tests that failed on the reference pass on this PR (incl. grpo_..._megatron_fp8_e2e_tq_mooncake, megatron_generation_tq_mooncake, yarn_256k, VLM megatron tests).
• 29 of 31 PR failures fail identically on the reference → pre-existing.
• Delta triage:
  • ~14 Slurm DUE TO TIME LIMIT kills during a slow-cluster window (mostly SFT tests, which don't touch vLLM; healthy metrics up to the kill) — all passed on retry.
  • llm_distillation_qwen3_1_7b_1n8g_megatron_qa_nvfp4 — fails identically on today's scheduled main run and on a base-commit control (nemo-ci#58682073): Megatron strict-instantiation allowlist rejects a _target_ in the shared pretrained NVFP4 checkpoint's run_config.yaml. Environment drift on the shared checkpoint, unrelated to this PR.
  • fp8-rollouts — see next section.
• (One long retry, yarn_256k, still finishing at time of writing; it already passed once in this pipeline's first attempt wave on the reference side.)

fp8-rollouts (H100): wake-up OOM — mitigated, residual flakiness

Both nightly fp8-rollouts variants initially died with CUDA Error: out of memory at csrc/cumem_allocator.cpp:163 at vLLM wake-up (steps 74 and 87 of 100), then stalled to the Slurm limit. Fix 47d835e89 restores 0.20's in-place weight/scale updates in our patched Fp8LinearMethod.process_weights_after_loading (the 0.25 port rebound every linear layer's weight/weight_scale_inv storage on every refit, fragmenting device memory across sleep/wake cycles).

Validation (nemo-ci#58693850): ..._tq_simple passes the full 100 steps with the fix; the plain variant OOM'd once at step 37 and is on a second attempt. Combined tally across all attempts: 0.20 = 2/2 pass (reference), 0.25 = 2 pass / 3 OOM. So 0.25 leaves the fp8-rollouts colocated setup with thinner wake-up headroom (its default FULL_AND_PIECEWISE cudagraph capture set is much larger). If flakiness persists, suggested follow-up: trim cudagraph_capture_sizes or lower gpu_memory_utilization in the two fp8-rollouts recipes.

GB200 nightly — nemo-ci#58695371

The reference run's GB200 results are unusable as a baseline (all of its GB200 jobs failed to a storage-quota infra issue), so comparison is against the scheduled main-branch nightly run from the same day.

• 22/24 pass (4 infra failures at job startup all passed on retry).
• llm_dpo_nanov3_30B3AB_1n4g_fsdp4ep4_automodel — also fails on main's same-day run → pre-existing.
• llm_grpo_moonlight_16ba3b_4n4g_megatron — fails at tokenizer load in the driver (Couldn't instantiate the backend tokenizer... need sentencepiece or tiktoken), reproducible (2/2), while main's same-day run passes. Everything relevant is byte-identical between the base lock and ours (transformers 5.8.1, tiktoken/blobfile/sentencepiece are explicit base deps with identical versions and ARM wheels; the tokenizer loads fine locally with this exact package set). Needs someone with cluster access to inspect the driver venv in the freshly built ARM image — not attributable to any dependency change in this PR.

DSv3 perf — the honest picture

The 32-node DSv3 perf test has no green baseline at this base commit. A base-commit control (nemo-ci#58698490, vLLM 0.20) fails during refit with a pre-existing NeMo-RL bug: AssertionError: Parameter model.embed_tokens.weight too large for buffer: 1853358080 > 1771261132 in stream_weights_via_ipc_zmq_impl.

On 0.25 the test failed earlier, at engine startup, with EADDRINUSE on the TCPStore port — root cause: NeMo-RL assigns each engine a deterministic VLLM_PORT, and in 0.25 RayExecutorV2 probes that range for the TCPStore while the cross-node broadcast MessageQueue allocates from the same range in between probe and bind. Only engines that span nodes hit this (DSv3 TP=32), which is why nothing else was affected. After two patch-based attempts didn't take effect inside the CI container's EngineCore, the operative fix (61c66de71) drops VLLM_PORT for node-spanning engines so all consumers use ephemeral ports, exactly like vanilla vLLM. Validated: no more port collisions on either platform (H100 nemo-ci#58718348, GB200 nemo-ci#58718647).

Remaining: with ports fixed, the DSv3 EngineCore now dies silently during executor init on both platforms (Engine core initialization failed... Failed core proc(s): {} — SIGKILL profile, all 32 RayWorkerProcs already up; suggests host-memory pressure during TP32 init on colocated nodes). Diagnosing this needs Ray session logs / dmesg from the run itself — not reachable from the CI logs. Given the test is equally broken at the base commit (different stage, same net result), I'd suggest not blocking the bump on it and tracking both issues (the pre-existing refit buffer assertion + the 0.25 init death) as follow-ups.

qwen3.5 disabled tests — nemo-ci#58665710

Test	Result
grpo-qwen3.5-35ba3b-2n8g-automodel-ep16	✅ pass
vlm_grpo-qwen3.5-35ba3b-geo3k-2n8g-automodel-ep16	✅ pass
vlm_grpo-qwen3.5-35ba3b-geo3k-2n8g-megatron-ep16	✅ pass
grpo-qwen3.5-397ba17b-32n8g-megatron.v2	❌ infra: Qwen/Qwen3.5-397B-A17B is not in CI's offline model cache — fails at tokenizer load before any vLLM code runs. Needs cache seeding; independent of the bump.

The three passes confirm the 0.25.1 bump resolves the hang/crash (vllm-project/vllm#36237) these tests were disabled for → they can be re-enabled with this PR.

Fixes added during validation (beyond the original port)

• a213ac534/db34114f2/61c66de71 — TCPStore/MessageQueue port-collision fixes for node-spanning engines (ephemeral-port approach is the operative one; the two patch-based attempts are retained as harmless hardening).
• 47d835e89 — fp8 refit: in-place weight/scale updates to stop per-refit device-memory churn.

🤖 Generated with Claude Code

[terrykong]

/ok to test 5842dda

[terrykong]

fp8-rollouts update: plain variant needs memory headroom, not just the refit fix

The plain fp8_rollouts_v3 OOM'd again on retry with the refit fix (step 46; attempts across old+new code now: OOM@74, OOM@87→tq_simple, pass, OOM@37, OOM@46 — vs consistent passes on vLLM 0.20). The refit fix is validated for tq_simple (full 100-step pass) but the plain recipe sits too close to the memory edge under vLLM 0.25, which keeps more device memory outside the sleep pool (larger default FULL_AND_PIECEWISE cudagraph capture set).

5842ddac4 lowers gpu_memory_utilization 0.6 → 0.55 for the fp8-rollouts recipe (tq_simple inherits it). Validation: nemo-ci#58764723 reruns both variants on the new SHA.

[terrykong]

fp8-rollouts: resolved ✅

With 5842ddac4 (gpu_memory_utilization 0.6 → 0.55) on top of the in-place refit fix (47d835e89), both variants pass the full 100 steps on nemo-ci#58764723:

Test	Result
llm_grpo_llama3_1_8b_instruct_1n8g_megatron_fp8_rollouts_v3	✅ pass (was 1/4 before the headroom bump)
..._tq_simple	✅ pass

This closes the last open bump-attributable H100 nightly delta. Also for the record: the final straggler yarn_256k failed on retry with a Slurm time-limit kill — it fails on the reference pipeline as well (pre-existing, not a delta).

GitHub CI on 5842ddac4 is green (the flaky PPO shard passed on rerun); remaining reds are only the intentional pinned-base check + its aggregate, which clear on rebase.

[terrykong]

/ok to test dd0f128

[terrykong]

Rebased onto main + revalidation kicked off

The branch is rebased from 26effe27 onto current main HEAD (bc382dc4); head is now dd0f128cc. As part of the rebase the 20 iterative commits were restructured into 11 atomic ones (identical tree, minus 4 stray tests/unit/unit_results/*.json artifacts; the two TCPStore patch attempts that never took effect in the EngineCore are dropped, keeping only the operative ephemeral-port fix).

New in the rebase: porting main-side features that landed since the old base

Main gained several vLLM-facing features between 26effe27 and HEAD that were written against vLLM 0.20. Audited all of them against the 0.25.1 source; three needed porting (each change verified against the 0.25.1 tree):

• Sharded NIXL expert refit (feat: add checkpoint-engine refit interface and integrate NIXL #2608): expert weights moved onto RoutedExperts (...experts.routed_experts.w13_weight naming, owner.moe_config.* for tp/eplb metadata) — the old lookups silently dropped expert weights from transfer and defaulted to rank-0/size-1 sharding.
• MXFP8 rollout (feat: MXFP8 rollout support #1887): FusedMoeWeightScaleSupported import moved (hard ImportError in both MoE refit helpers); the linear-refit kernel guard now also accepts the Cutedsl kernel that 0.25 prefers (identical scale swizzle).
• ModelOpt W4A16 NVFP4 rollout (feat(modelopt): support real NVFP4 QAT rollout for MoE and Mamba #2983): 0.25 installs LinearMethodCls as an instance attribute keyed off the quant algo, which shadowed the NeMo subclass override (W4A16 silently ran the W4A4 linear method); 0.25's Marlin converter also pads MoE tiles natively, so the NeMo-side pre-padding would double-pad and trip upstream shape asserts.

Checkpoint-engine/delta-refit, MTP spec dec, NUMA binding, and the sampled-token logprob fix audited clean — no changes needed.

Also re-enabled the 3 qwen3.5 tests in nightly.txt/disabled.txt that were disabled for vllm-project/vllm#36237 and validated passing on 0.25.1 (nemo-ci#58665710). The 397B test stays disabled (different bug + model missing from the CI cache).

Revalidation runs (head dd0f128cc)

Coverage	Pipeline
H100 nightly (full suite)	nemo-ci#59133030
GB200 nightly (full suite)	nemo-ci#59133046

Any failures will be re-run against the rebase base (bc382dc4, current main) to separate pre-existing failures from bump regressions.

---

🤖 Generated with Claude Code

[terrykong]

Correction: the GB200 nightly run is nemo-ci#59149043 (replaces nemo-ci#59133046, which was triggered with a malformed suite name on my end and generated no tests).

[terrykong]

Second GB200 correction: the GB200 nightly run for this validation is nemo-ci#59153735. nemo-ci#59149043 was routed to a GB200 cluster whose CI runner hosts are currently rejecting the pipeline's own setup step (a PEP 668 externally-managed-environment error in nemo-ci's job preamble, before any RL code runs — reproducible there for any ref, infra-side). The rerun targets the same GB200 cluster the earlier validation runs and the scheduled main nightlies use.

[terrykong]

/ok to test c0a5ef5

[terrykong]

/ok to test 0ec18c9