[faresobeid]

Bumps vLLM 0.32.0 -> 0.33.0. vLLM 0.33 is compiled against and pins torch 2.14.0, so this is also a torch bump for the trainer.

Kernel wheels rebuilt for torch 2.14 in PrimeIntellect-ai/prime-kernels#24 (release v0.1.0-b050074; DeepGEMM needed -std=c++20 for the torch 2.14 headers) and pinned here.

Dependencies

• vllm 0.33.0 wheel. The GitHub release is attached to tag v0.33.0rc3 (same commit as v0.33.0, sha256s match PyPI), so the wheel URLs use download/v0.33.0rc3/vllm-0.33.0-…; download/v0.33.0/… 404s.
• torch 2.13.0 -> 2.14.0, pinned exactly in override-dependencies (>= let the override replace vLLM's ==2.14.0 and resolve 2.14.1). torchvision 0.29.0, torchaudio 2.11.0, triton 3.8.0, NCCL 2.30.7. 2.14.1 only fixes MPS and the CUDA 13.2 builds, so 2.14.0 is right for cu130.
• transformers 5.19.0 -> 5.20.2 (vLLM 0.33 allows <5.21 and its CI tests 5.20.2).
• flashinfer-python / flashinfer-cubin / flashinfer-jit-cache 0.7.0.post1 -> 0.7.2.post1; tokenspeed-mla 2.0.3, humming-kernels 0.1.19 (already exempt from the cooldown), nvidia-cudnn-frontend >= 1.31.0 (resolves to 1.32.0). Merged with main (verifiers bump, prime CLI as a uv tool).
• NIXL 0.10.1 -> 1.5.0: vLLM 0.33's NIXL connector raises below 1.5 ([NIXL] Prepare KV transfer descriptors as NIXL 1.5 strided lists vllm-project/vllm#61032). nixl is pinned in disagg (uv otherwise kept 0.10.1, which drags in nixl-cu12) and the dependency-metadata override moves to 1.5.0 — 1.5's metadata still requires both nixl-cu12 and nixl-cu13. The NIXL Python API prime-rl uses (transports/weights/nixl/agent.py) is unchanged between 0.10.1 and 1.5. Dockerfile.cuda and scripts/install_nixl_from_source.sh use the tag v1.5.0 (tags gained a v prefix in 1.0).
• flash-attn 2.8.3 wheels -> cu130torch2.14 (mjun0812 v0.10.0 x86_64, v0.10.1 aarch64). Checked: imports on torch 2.14, carries sm_80/90/100/120 cubins.
• flash-mla dropped. There is no torch 2.14 build (skyrl-wheels stops at 2.13) and the 2.13 wheel links the unstable libtorch ABI. The trainer's cudnn_flashmla backend now calls the FlashMLA vLLM bundles (vllm.third_party.flashmla, _flashmla_C.abi3.so, FlashMLA 2026.09.30 / [Kernel][DSv4.1] Bump FlashMLA to the 2026.09.30 release and its interleaved V4.1 KV cache vllm-project/vllm#61371): same flash_mla_sparse_fwd signature, with attn_sink and topk_length. This also removes the duplicate-FlashMLA symbol clash the DSv4 kernel tests worked around, and brings the backend to aarch64.
• Not torch-version-bound (checked symbols): torchao (stable ABI), flash_attn_3 (abi3), mamba-ssm (Triton-only build), quack / FA4 / sonic-moe / dion / torchtitan / modelexpress (pure Python). torchtitan @ 23e4dfc needs no bump.

Breakages fixed

• serving_tokens.py (every non-streaming /inference/v1/generate):
  • 0.33 calls serve_tokens_full_generator(..., derender_request=...) ([Frontend] Add derender output mode to generate vllm-project/vllm#60082) → accept and forward it.
  • 0.33 reads sampling_params.logprobs before consuming the result generator, so clearing it inside the loop no longer worked: the default flat logprobs<=1 path and the compact path hit AssertionError: Did not output logprobs (500). The packing decision is now made up front and upstream gets a copy with logprobs off. New test test_flat_sampled_logprobs_are_packed covers the default RL path.
  • Our detokenize/flat-logprob tweaks now apply only to output_mode="tokens" (0.33's text/derender modes need detokenization).
• server.py: vllm.entrypoints.launchers.cli_args moved to vllm.entrypoints.common.cli_args ([Frontend] Consolidate startup utils. vllm-project/vllm#62390).
• qwen38_weight_loading.py: compute_ple_shard_overlap became PLEVocabParallelEmbedding._compute_shard_overlap ([Qwen4Exp] Clean up HC and PLE implementations for Qwen3.8-Flash-Next vllm-project/vllm#61454). This ImportError sat at the top of apply_shared_vllm_patches, and vLLM swallows plugin errors, so every later patch was silently skipped.
• gpt_oss_weight_loading.py: _load_weights_other lost heads_per_rank/head_start, and sinks now load through their own TP-aware loader ([Bugfix][RL] Load attention sinks through weight_loader for layerwise reload vllm-project/vllm#57797). The sink branch is removed (it would now narrow twice under TP>1); expert routing through loaders is kept (upstream still param.copy_s them).
• monkey_patch_routed_experts_cut_loaded_prefix_hits: Scheduler._get_local_prefix_cache_hit(request, max_cache_hit_length) ([Bugfix][AuxOutput] Retain routed experts while a KV cache tier holds their blocks vllm-project/vllm#62591); the old wrapper raised TypeError on every new request. New test.
• monkey_patch_routed_experts_with_pd_connectors: #62591 caps local and external hits at the blocks whose routing rows this instance retains (max_cache_hit / cap_external_hit). A P/D decode instance has no rows for a remotely prefilled prompt, so the cap would zero the NIXL load and recompute the whole prompt on decode. Decode-side P/D requests now skip both caps. New test.
• KV offload reuse with routed experts (monkey_patch_clamp_kv_offload_loads_for_routed_experts): #62591 admits an external hit only when every block's routing rows are retained, which it does for at most 2x the GPU KV cache, so with a CPU offload tier every multi-turn reload was refused. GPU A/B (OLMoE-1B-7B, RTX PRO 6000, OffloadingConnector, 160 GPU blocks, 48 conversations x 4 turns, routed_experts_prompt_start = previous turn): external hit tokens per turn were 0 / 0 / 0 / 0 with routed experts on (with or without the 0.32 clamp) vs 0 / 18432 / 24576 / 13312 with them off. A load within the clamp needs none of its rows, so cap_external_hit now admits it whole: with the fix the routed-on run hits exactly like the routed-off run, every row count is right, generated tokens are identical (192/192) and there is no BlockObjectStoreError. The clamp therefore stays (it bounds those loads).
• Trainer fp8 casts (trainer/models/kernels/fp8_utils.py): [Perf] per_token_group_quant_8bit: multiply by reciprocal instead of dividing per element vllm-project/vllm#61343 quantizes per-token groups as x * (1/s) instead of x / s, so the trainer's Triton casts, which are meant to be bit-identical to vLLM's per_token_group_quant_fp8, differed in 0.03-0.08% of fp8 values (scales unchanged). The non-ue8m0 path now multiplies by the exact reciprocal (div_rn(1, s)); test_fp8_utils passes again (25/25).
• sampling_order.py: accept the new has_topp_only_rows kwarg (PRL_TOP_P_BEFORE_TOP_K=1 raised TypeError).
• test_routed_experts_pd.py: AuxOutputSchedulerConnector now needs a kv_cache_manager.

Patches deleted (fixed upstream in 0.33)

• monkey_patch_deepseek_v4_c128_boundary: [Bugfix][DSv4] Don't bake the C128 compressor skip into CUDA graphs vllm-project/vllm#59709 (skip is no longer captured into CUDA graphs).
• monkey_patch_deepseek_v4_attn_sink_loading: [Bugfix][RL] Load attention sinks through weight_loader for layerwise reload vllm-project/vllm#57797 (sinks load via load_attention_sink, for every DSv4/V4.1 variant). The patch would now raise AttributeError and would mis-shard under TP>1.
• server.py LoRA reload: the load_inplace = False reset on the stored request — [bug] Fix lora reloading from disk every step when using AsyncLLM vllm-project/vllm#41482 resets it before caching (the only store point; error paths return first). We still force load_inplace = True on the request.
• monkey_patch_aux_output_store's eviction override: [Perf][AuxOutput] Do not rescan referenced keys in the routed-experts store vllm-project/vllm#62392 keeps referenced keys out of the LRU. The 2x store headroom is kept (with a KV connector the scheduler now retains up to 2x KV blocks; a 1x store would grow by copying its arena on the engine path); its upstream-equivalence test is replaced by a headroom test.

Every other patch was checked against the 0.33 source and its target is unchanged in name, signature and the behavior we rely on; _patch_lora_key_prefix's copy is still in sync (lora_model.py is unchanged). Still needed on 0.33: KV-transfer freed assert (#46240 open), MiniMax think-end, online FP8 parameter cast, online MXFP8 kernel on reload (#62406 closed unmerged), GLM5-Next KDA conv weights (#60991 merged after the 0.33 cut), routed-experts chat stripping, DP coordinator timeout, no-MoE-LoRA, request replay.

Torch 2.14

• No prime-rl torch API breaks: every private FSDP2 / DTensor / dynamo / c10d API we import is unchanged, and the BC-breaking changes in the release notes are not used.
• activation_checkpointing.py: pass respect_saved_tensors_hooks=False to keep today's behavior (SAC-saved tensors are not offloaded); 2.14 otherwise warns whenever activation offloading is on.
• Worth knowing: 2.14 fixes FSDP2 PREMUL_SUM producing all-zero expert grads with reduce_dtype="bfloat16" + EP (pytorch#190747; our default reduce_dtype is float32). Inductor's aten_distributed_optimizations.enable_simple_overlap is now on by default and reorders the funcol collectives in deepseek_v4/attention.py and distributed/ngram_lookup.py — watch for memory or hangs there (=False restores 2.13 behavior).

Behavior changes checked, no action

• Requests sent while generation is paused now get 503 ([Core] Reject new requests while generation is paused vllm-project/vllm#51488) — only in wait/abort modes; prime-rl always pauses with mode="keep" (server.py, orchestrator/clients.py, inference/dynamo.py).
• Weight-transfer API (ParamMeta / iter_full / iter_local, #63256, #35911, #60934, #62555): prime-rl uses its own PyNccl broadcast + layerwise reload, not vllm/distributed/weight_transfer.
• kv_transfer_params moved to the top level of /render output (#61241): prime-rl does not call /render.
• Chat finish_reason (#59324) changes only affect the eval chat path: a length-truncated tool call is now length, not tool_calls.
• All CLI/engine arg names, the 12 tool parsers, 9 reasoning parsers, KV connector names and the vllm:* metrics prime-rl reads still exist. vllm._C is now _C_stable_libtorch, but ops stay under torch.ops._C.

Kernel wheels for torch 2.14

prime-kernels, deep-ep, deep-gemm, deep-select link the unstable libtorch ABI (prime_kernels/flash_moe/_C failed to import on 2.14 with an undefined symbol). PrimeIntellect-ai/prime-kernels#24 builds them against 2.14.0+cu130; the pins point at release v0.1.0-b050074 (prime_kernels-0.1.0+cu130torch2.14.0). DeepGEMM 891d57b compiles its extension as C++17 and torch 2.14's headers require C++20, so the build switches it to -std=c++20 (what vLLM's DeepGEMM fork does). torchao stays on v0.1.1 (stable ABI). prime_kernels.flash_moe._C now imports on torch 2.14; deep-ep / deep-select need a GPU to import.

Recommended separately (prime-kernels#20): DeepGEMM 2.5.0 -> 2.8.1 (vllm-project/DeepGEMM@0820030, what vLLM 0.33 vendors). vLLM imports site-packages deep_gemm before its vendored copy, so ours shadows it in inference. 2.8.1 removes fp8_mqa_logits, so trainer/models/kernels/fp8_indexer.py:277 must move to fp8_fp4_mqa_logits in the same change.

Validation

• CPU: pytest tests/unit -m "not gpu" on vLLM 0.33.0 / torch 2.14.0: 1085 passed, 1 xfailed; ruff check and ruff format --check clean.
• apply_shared_vllm_patches() runs to completion on 0.33 (with PRL_TOP_P_BEFORE_TOP_K=1 and PRIME_NO_MOE_LORA=1), and is idempotent; server.py / serving_tokens.py import.
• AST import audit of every vllm import in src/, tests/, packages/ against 0.33: no missing names after the fixes above. Every patched or subclassed upstream function whose source changed between 0.32 and 0.33 was diffed and reviewed.
• GPU (RTX PRO 6000 sandbox, vllm/vllm-openai:v0.33.0): the routed-experts + KV offload A/B above.
• GPU CI (gpu-ci, 2xH200): requested, but the prime-rl-2xh200 runner has not picked up any GPU job since 11-02 14:48 (main's included), so the GPU suite ran on RTX PRO 6000 (SM120) sandboxes instead:
  • pytest tests/unit -m gpu, file by file: everything passes except tests of kernels SM120 doesn't support: DeepGEMM fp8 (test_fp8_indexer, test_fp8_grouped_gemm: "Unsupported architecture" / "Unknown recipe"), prime-kernels indexed_attention (qwen3_8_flash_next/test_vlm: needs sm_80/90/100/103), and test_fused_lm_head, whose process dies mid-file (to be rechecked on H200). test_fp8_utils caught the vLLM #61343 cast change above.
  • Integration (tests/integration): the runs train normally (e.g. reverse_text: 5 steps, mismatch KL 0.0007-0.008) but this sandbox is several times slower than the CI runners, so they exceed the tests' 5-20 min budgets; rerunning with longer budgets.

Needs GPU before merge:

• reverse_text integration suite, Qwen3-30B-A3B RL (bf16 + fp8) and router replay vs 0.32 baselines (step time, mismatch KL).
• DSv4 cudnn_flashmla kernel tests on SM90 and SM100 (vLLM's FlashMLA fork vs nv_dev@b7643bd).
• P/D with routed experts (NIXL 1.5): prefill is not recomputed on decode, router rows are correct, no BlockObjectStoreError.
• GPT-OSS BF16 and Qwen3.8 PLE reload at TP>1; DSv4 long-context mismatch KL (C128 patch removal).
• Dockerfile.cuda build (NIXL v1.5.0 source build: its [build-system] now requires torch==2.11.*, which only affects the bundled nixl_ep, unused here).

Follow-ups (not in this PR):

• monkey_patch_routed_experts_cut_loaded_prefix_hits: redundant without P/D (on 0.33 every non-P/D external load has retained rows). Still needed for P/D decode instances that also serve non-P/D requests: decode requests load prompt blocks without rows, and upstream request_finished then records those GPU-cached blocks as row-holding. Delete together with the P/D patch if routed experts + P/D is dropped. The same gap makes an offload connector on a P/D decode instance unsafe for external hits on those blocks.
• Qwen3.8 NVFP4 PLE weight_scale shards are not filtered by the reload patch (harmless; falls through to upstream).

🤖 Generated with Claude Code