Repository navigation
test(deepseek_v4): load flash-mla before vLLM's bundled FlashMLA - #4139
Merged
Merged
Conversation
faresobeid
marked this pull request as ready for review
October 18, 2026 10:44
samsja
pushed a commit
that referenced
this pull request
Oct 19, 2026
Bumps vLLM 0.31.0 -> 0.32.0. ## Dependencies - `vllm` 0.32.0 wheel. torch stays 2.13.0, flashinfer stays 0.7.0.post1. - `transformers` 5.6.2 -> 5.19.0 (vLLM 0.32 requires `>=5.16.1,<5.20`; 5.19.0 is what vLLM's CI tests). The `transformers==5.6.2` override is removed. `tokenizers` 0.22.2 -> 0.23.3. - `nvidia-cutlass-dsl` override 4.7.1 -> 4.8.0 (exact pin in vLLM 0.32). `tokenspeed-mla` 0.2.15, `humming-kernels` 0.1.17, `instanttensor` 0.2.2, `xgrammar` 0.2.8 come from vLLM; `humming-kernels` and `instanttensor` are exempt from the 7-day cutoff. - `ifbench` is excluded from the workspace: it pins `setuptools<78`, vLLM 0.32 needs `>=78.1.1`. No config or test references it. ## Patches deleted (fixed upstream in 0.32) - `monkey_patch_shm_buffer_ownership`: vllm-project/vllm#61000. - `monkey_patch_tokenize_params_validation`: vllm-project/vllm#57035 (`max_tokens` is an upper bound, not reserved from the context). - `monkey_patch_nano_v3_reasoning_parser`: upstream `nemotron_v3` parser. The Nemotron 3 Nano / 3.5 auto-detection now picks `nemotron_v3`. Configs that set `reasoning_parser = "nano_v3"` must rename it. - DeepSeek V4 `layer_types` shim (`utils/transformers_compat.py` and its vLLM patch): transformers >= 5.15 knows the V4 layer types. ## Breakages fixed - `serving_tokens.py`: `GenerateResponseChoice` is gone and `GenerateResponse` is now a discriminated union (vllm-project/vllm#58588). Subclass `GenerateTokensChoice` / `GenerateTokensResponse` and check `isinstance` against `GenerateTokensResponse`. - `prefill_logprobs`: validate with `GenerateTokensResponse` (the union alias has no `model_validate_json`). - `_patch_lora_key_prefix`: re-copied from 0.32's `LoRAModel.from_local_checkpoint`; tensorizer support was removed upstream (vllm-project/vllm#60024). - NemotronH config: transformers 5.19 renames legacy layer types (`mamba` → `linear_attention`, `attention` → `full_attention`) by setting `config.layer_types`, which was a read-only property. `layer_types` is now an `attribute_map` alias of `layers_block_type` (as upstream), and the model and conversion code use the new names. Hub configs with legacy `layers_block_type` or `hybrid_override_pattern` load as before. - Qwen3.5 tests: `from_config` through the auto class now swaps a composite config for its text config; the tests build VLMs with `Qwen3_5ForCausalLM`, as the trainer does. - `monkey_patch_kv_xfer_finished_tolerate_freed`: synced the copied method with 0.32 (`_set_kv_fetch_stage(req, KV_FETCH_COMPLETED_WAITING)`). Still needed on 0.32 (fixes land in 0.33 or are not merged): DeepSeek V4 C128 boundary, DeepSeek V4 `attn_sink` loading, GPT-OSS weight loading, KV-transfer freed assert, MiniMax think-end, routed experts with P/D connectors. Every remaining patch target still exists in 0.32; targets whose source changed were checked against the new source. ## Validation CPU: `pytest tests/unit -m "not gpu"` 854 passed; ruff clean. Import/attribute check of every `vllm` import and monkey-patch target in `src/` and `tests/` against 0.32: no missing names; patched methods whose source changed were reviewed. GPU (H200, max 2 nodes; 0.31 baselines use main's lock at the same base commit, same nodes): | Check | 0.32 | 0.31 | |---|---|---| | `tests/unit -m gpu` (NemotronH, Qwen3.5, Mamba CP, DSv4 kernels, …) | pass after the fixes above¹ | – | | Integration: reverse_text, reverse_text_moe, reverse_text_lora, reverse_text_rl_opd, reverse_text_rl_sft, reverse_text_sft, reverse_text_sft_lora, alphabet_sort | pass² | – | | Qwen3-30B-A3B RL, bf16, NCCL (15 steps, median step / mismatch KL) | 15.3 s / 0.028 | 14.9 s / 0.025 | | Qwen3-30B-A3B RL, online fp8 (`fp8_per_block`), NCCL | 15.4 s / 0.061 | 16.4 s / 0.063 | | Qwen3-30B-A3B RL, bf16, filesystem broadcast (trainer step) | 156 s / 0.030 | 155 s / 0.020 | | GLM-4.5-Air RL, router replay, 1 train + 1 infer node (10 steps, median step / mismatch KL / reward) | 23.4 s / 0.0028 / 0.49→0.65 | 24.7 s / 0.0036 / 0.44→0.66 | | Qwen3-30B-A3B P/D (NIXL 1P1D, vllm-router 0.2.3) vs colocated, router replay + sampling masks, batch-invariant: identical tokens, rows, masks, logprobs | c1 10/10, c10 20/20, uint16 c1 10/10 / c10 20/20, decode preemption 20/20 | same as 0.31 results | | GLM-4.5-Air (TP4) P/D vs prefill server queried directly (same node), batch-invariant: identical tokens and rows | 0/10 (decode diverges); 10/10 with `disable_custom_all_reduce = true`³ | 10/10 | | Long multi-turn prompt (14.4k prompt + `max_tokens` 4096, `max_model_len` 16384), plain and router-replay server | 200, prompt not truncated; `/generate` 15.9k + 4096 → 200, capped at `max_model_len`; 16.4k prompt → 400 | same | | DeepSeek-V4-Flash TP8+EP serving smoke | loads, coherent completion | – | ¹ The DSv4 `cudnn_flashmla` kernel tests fail the same way on main when vLLM's DSv4 test loads first (FlashMLA symbol clash); with #4139 applied they pass on 0.32 (123 passed). ² Each integration module's `test_dashboard` needs `playwright install --with-deps`, which fails on our nodes; all other tests pass. ³ 0.32 uses the custom all-reduce under `VLLM_BATCH_INVARIANT=1` (vllm-project/vllm#58623). Two TP>1 servers then no longer decode bit-identically, so P/D only matches colocated serving bit-for-bit with `disable_custom_all_reduce = true`. prime-rl does not set batch invariance; RL is unaffected (GLM row above). Not covered: Qwen3.5/Qwen3.8 PLE reload (no checkpoint on disk); GLM-4.5-Air disaggregated RL (needs 3 nodes). Router-replay requests need `top_k > 0` and `temperature > 0` (sampling-mask replay), on 0.31 and 0.32 alike. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Touches core inference dependencies and removes several runtime patches; misalignment with vLLM 0.32 APIs or Nemotron parser changes could break rollouts or serving until configs are updated. > > **Overview** > **Upgrades the inference stack from vLLM 0.31.0 to 0.32.0**, with matching pins: `transformers` 5.19.0, `nvidia-cutlass-dsl` 4.8.0, and direct wheel URLs. `ifbench` is dropped from the uv workspace because its `setuptools<78` pin conflicts with vLLM 0.32. > > Several **vLLM workarounds are removed** now that upstream fixed them: SHM buffer ownership on dequeue, relaxed `TokenizeParams` prompt-length checks, the custom `nano_v3` reasoning parser, and the DeepSeek V4 `layer_types` transformers shim (`transformers_compat.py`). **Nemotron 3 Nano / 3.5** auto-detection now uses the built-in **`nemotron_v3`** parser instead of `nano_v3`. > > **API follow-ups for 0.32:** token-in/token-out serving and `prefill_logprobs` use `GenerateTokensChoice` / `GenerateTokensResponse` instead of the old `GenerateResponse*` types. The LoRA checkpoint patch is re-synced from 0.32 (tensorizer path removed). The KV-transfer “freed request” patch adds `_set_kv_fetch_stage(..., KV_FETCH_COMPLETED_WAITING)` to match upstream. > > **Trainer Nemotron-H** aligns layer-type names with newer transformers (`linear_attention`, `full_attention`) via `attribute_map`, while still accepting legacy `mamba` / `attention` in `layers_block_type`. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit f92627e. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
mikasenghaas
approved these changes
Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tests/unit -m gpufails all 24cudnn_flashmlacases intest_deepseek_v4_kernels.pywithThe chosen implementation does not support all required features ... TOPK_LENGTH. They pass when run alone.Cause:
vllm/_flashmla_C.abi3.soand the flash-mla wheel (flash_mla/cuda.*.so) both export the GNU-unique symbolFwd_Sm90_Impl::features. The dynamic loader binds every library to the copy that was loaded first. vLLM's older list has 6 entries and noTOPK_LENGTH; the wheel reads 8 entries from it (hence the trailingHEAD_64, HEAD_64in the error). In the suite,test_deepseek_v4.py::test_deepseek_v4_on_disk_keys_map_to_the_names_vllm_expectsimports vLLM's DeepSeek V4 model, which loads_flashmla_C, before the kernel tests run.Fix: the kernel test module imports
flash_mlaat collection, so the wheel's list wins. vLLM only reads the first 6 entries, which are the same in both lists.Checked on one H200 node with main's venv:
import vllm._flashmla_Cthenflash_mla_sparse_fwd(..., topk_length=...)reproduces the CI error; without the vLLM import it passes.pytest tests/unit -m gpu: 234 passed, 2 skipped (main CI: 24 failed, 210 passed).Note
Low Risk
Test-only import ordering fix with no runtime or production behavior changes.
Overview
Fixes GPU suite flakes where all
cudnn_flashmlacases intest_deepseek_v4_kernels.pyfail with missingTOPK_LENGTHwhen run after other unit tests that import vLLM.The module now imports the
flash_mlawheel at pytest collection (when installed), before vLLM can load its older bundled_flashmla_C. Both libraries share the GNU-unique symbolFwd_Sm90_Impl::features, so whichever loads first wins for the whole process; the wheel’s feature list includesTOPK_LENGTH, while vLLM’s does not.No production code changes—only test import ordering and an explanatory comment.
Reviewed by Cursor Bugbot for commit c369ed2. Bugbot is set up for automated code reviews on this repo. Configure here.