Repository navigation
[LoRA] Remove tensorizer - #60024
Merged
Merged
[LoRA] Remove tensorizer#60024
Conversation
jeejeelee
requested review from
AndreasKaratzas,
DarkLight1337,
NickLucche,
aarnphm and
robertgshaw2-redhat
as code owners
October 5, 2026 07:37
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Contributor
|
Documentation preview: https://vllm--60024.org.readthedocs.build/en/60024/ |
mergify
Bot
added
documentation
Member
Author
|
/ci run |
|
✅ Triggered Buildkite CI #92845 for commit |
Collaborator
CI selector (shadow): 29 test steps (46 jobs) instead of 63 (85 jobs)Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works. Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.
Selector would run (29)
Would skip (today's rules run them) (43)
Would add (today's rules do not run them) (9)
AMD mirrors: would skip (52)
AMD mirrors: would add (5)
9 changed files · base |
Isotr0py
approved these changes
Oct 5, 2026
jeejeelee
enabled auto-merge (squash)
October 5, 2026 09:10
github-actions
Bot
added
the
ready
1 task done
samsja
pushed a commit
to PrimeIntellect-ai/prime-rl
that referenced
this pull request
Oct 19, 2026
Bumps vLLM 0.31.0 -> 0.32.0. ## Dependencies - `vllm` 0.32.0 wheel. torch stays 2.13.0, flashinfer stays 0.7.0.post1. - `transformers` 5.6.2 -> 5.19.0 (vLLM 0.32 requires `>=5.16.1,<5.20`; 5.19.0 is what vLLM's CI tests). The `transformers==5.6.2` override is removed. `tokenizers` 0.22.2 -> 0.23.3. - `nvidia-cutlass-dsl` override 4.7.1 -> 4.8.0 (exact pin in vLLM 0.32). `tokenspeed-mla` 0.2.15, `humming-kernels` 0.1.17, `instanttensor` 0.2.2, `xgrammar` 0.2.8 come from vLLM; `humming-kernels` and `instanttensor` are exempt from the 7-day cutoff. - `ifbench` is excluded from the workspace: it pins `setuptools<78`, vLLM 0.32 needs `>=78.1.1`. No config or test references it. ## Patches deleted (fixed upstream in 0.32) - `monkey_patch_shm_buffer_ownership`: vllm-project/vllm#61000. - `monkey_patch_tokenize_params_validation`: vllm-project/vllm#57035 (`max_tokens` is an upper bound, not reserved from the context). - `monkey_patch_nano_v3_reasoning_parser`: upstream `nemotron_v3` parser. The Nemotron 3 Nano / 3.5 auto-detection now picks `nemotron_v3`. Configs that set `reasoning_parser = "nano_v3"` must rename it. - DeepSeek V4 `layer_types` shim (`utils/transformers_compat.py` and its vLLM patch): transformers >= 5.15 knows the V4 layer types. ## Breakages fixed - `serving_tokens.py`: `GenerateResponseChoice` is gone and `GenerateResponse` is now a discriminated union (vllm-project/vllm#58588). Subclass `GenerateTokensChoice` / `GenerateTokensResponse` and check `isinstance` against `GenerateTokensResponse`. - `prefill_logprobs`: validate with `GenerateTokensResponse` (the union alias has no `model_validate_json`). - `_patch_lora_key_prefix`: re-copied from 0.32's `LoRAModel.from_local_checkpoint`; tensorizer support was removed upstream (vllm-project/vllm#60024). - NemotronH config: transformers 5.19 renames legacy layer types (`mamba` → `linear_attention`, `attention` → `full_attention`) by setting `config.layer_types`, which was a read-only property. `layer_types` is now an `attribute_map` alias of `layers_block_type` (as upstream), and the model and conversion code use the new names. Hub configs with legacy `layers_block_type` or `hybrid_override_pattern` load as before. - Qwen3.5 tests: `from_config` through the auto class now swaps a composite config for its text config; the tests build VLMs with `Qwen3_5ForCausalLM`, as the trainer does. - `monkey_patch_kv_xfer_finished_tolerate_freed`: synced the copied method with 0.32 (`_set_kv_fetch_stage(req, KV_FETCH_COMPLETED_WAITING)`). Still needed on 0.32 (fixes land in 0.33 or are not merged): DeepSeek V4 C128 boundary, DeepSeek V4 `attn_sink` loading, GPT-OSS weight loading, KV-transfer freed assert, MiniMax think-end, routed experts with P/D connectors. Every remaining patch target still exists in 0.32; targets whose source changed were checked against the new source. ## Validation CPU: `pytest tests/unit -m "not gpu"` 854 passed; ruff clean. Import/attribute check of every `vllm` import and monkey-patch target in `src/` and `tests/` against 0.32: no missing names; patched methods whose source changed were reviewed. GPU (H200, max 2 nodes; 0.31 baselines use main's lock at the same base commit, same nodes): | Check | 0.32 | 0.31 | |---|---|---| | `tests/unit -m gpu` (NemotronH, Qwen3.5, Mamba CP, DSv4 kernels, …) | pass after the fixes above¹ | – | | Integration: reverse_text, reverse_text_moe, reverse_text_lora, reverse_text_rl_opd, reverse_text_rl_sft, reverse_text_sft, reverse_text_sft_lora, alphabet_sort | pass² | – | | Qwen3-30B-A3B RL, bf16, NCCL (15 steps, median step / mismatch KL) | 15.3 s / 0.028 | 14.9 s / 0.025 | | Qwen3-30B-A3B RL, online fp8 (`fp8_per_block`), NCCL | 15.4 s / 0.061 | 16.4 s / 0.063 | | Qwen3-30B-A3B RL, bf16, filesystem broadcast (trainer step) | 156 s / 0.030 | 155 s / 0.020 | | GLM-4.5-Air RL, router replay, 1 train + 1 infer node (10 steps, median step / mismatch KL / reward) | 23.4 s / 0.0028 / 0.49→0.65 | 24.7 s / 0.0036 / 0.44→0.66 | | Qwen3-30B-A3B P/D (NIXL 1P1D, vllm-router 0.2.3) vs colocated, router replay + sampling masks, batch-invariant: identical tokens, rows, masks, logprobs | c1 10/10, c10 20/20, uint16 c1 10/10 / c10 20/20, decode preemption 20/20 | same as 0.31 results | | GLM-4.5-Air (TP4) P/D vs prefill server queried directly (same node), batch-invariant: identical tokens and rows | 0/10 (decode diverges); 10/10 with `disable_custom_all_reduce = true`³ | 10/10 | | Long multi-turn prompt (14.4k prompt + `max_tokens` 4096, `max_model_len` 16384), plain and router-replay server | 200, prompt not truncated; `/generate` 15.9k + 4096 → 200, capped at `max_model_len`; 16.4k prompt → 400 | same | | DeepSeek-V4-Flash TP8+EP serving smoke | loads, coherent completion | – | ¹ The DSv4 `cudnn_flashmla` kernel tests fail the same way on main when vLLM's DSv4 test loads first (FlashMLA symbol clash); with #4139 applied they pass on 0.32 (123 passed). ² Each integration module's `test_dashboard` needs `playwright install --with-deps`, which fails on our nodes; all other tests pass. ³ 0.32 uses the custom all-reduce under `VLLM_BATCH_INVARIANT=1` (vllm-project/vllm#58623). Two TP>1 servers then no longer decode bit-identically, so P/D only matches colocated serving bit-for-bit with `disable_custom_all_reduce = true`. prime-rl does not set batch invariance; RL is unaffected (GLM row above). Not covered: Qwen3.5/Qwen3.8 PLE reload (no checkpoint on disk); GLM-4.5-Air disaggregated RL (needs 3 nodes). Router-replay requests need `top_k > 0` and `temperature > 0` (sampling-mask replay), on 0.31 and 0.32 alike. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Touches core inference dependencies and removes several runtime patches; misalignment with vLLM 0.32 APIs or Nemotron parser changes could break rollouts or serving until configs are updated. > > **Overview** > **Upgrades the inference stack from vLLM 0.31.0 to 0.32.0**, with matching pins: `transformers` 5.19.0, `nvidia-cutlass-dsl` 4.8.0, and direct wheel URLs. `ifbench` is dropped from the uv workspace because its `setuptools<78` pin conflicts with vLLM 0.32. > > Several **vLLM workarounds are removed** now that upstream fixed them: SHM buffer ownership on dequeue, relaxed `TokenizeParams` prompt-length checks, the custom `nano_v3` reasoning parser, and the DeepSeek V4 `layer_types` transformers shim (`transformers_compat.py`). **Nemotron 3 Nano / 3.5** auto-detection now uses the built-in **`nemotron_v3`** parser instead of `nano_v3`. > > **API follow-ups for 0.32:** token-in/token-out serving and `prefill_logprobs` use `GenerateTokensChoice` / `GenerateTokensResponse` instead of the old `GenerateResponse*` types. The LoRA checkpoint patch is re-synced from 0.32 (tensorizer path removed). The KV-transfer “freed request” patch adds `_set_kv_fetch_stage(..., KV_FETCH_COMPLETED_WAITING)` to match upstream. > > **Trainer Nemotron-H** aligns layer-type names with newer transformers (`linear_attention`, `full_attention`) via `attribute_map`, while still accepting legacy `mamba` / `attention` in `layers_block_type`. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit f92627e. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
michael-gpu
pushed a commit
to practical-inference/vllm-ampere-extended
that referenced
this pull request
Oct 20, 2026
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
This was referenced Oct 22, 2026
Closed
6 tasks done
6 tasks done
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Tensorizer is a niche usage — removing it from the LoRA code for maintainability.
Claims
Validation
Details
Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.