Repository navigation
[Bugfix] Fix Qwen4Exp PLE embedding rejecting INC (AutoRound) checkpoints - #59990
Merged
vadiklyutiy merged 2 commits intoOct 5, 2026
Merged
vadiklyutiy merged 2 commits into
vadiklyutiy merged 2 commits into
Conversation
…ints <!-- markdownlint-disable --> ## Overview Fixes vllm-project/vllm issue 59798. `Intel/Qwen3.8-Flash-Next-W4A16-AutoRound` is an AutoRound checkpoint, which vLLM loads as `INCConfig`. It failed at model construction with `NotImplementedError: Qwen4Exp PLE embedding does not support quantization config INCConfig`. `Qwen4ExpPLEEmbeddingMethod.from_quant_config` now asks `INCConfig` whether the PLE layer is quantized. vllm-project#59431 does the same for compressed-tensors. ## Claims - The command from the issue (TP=2, expert parallel) no longer raises. Both workers build the PLE n-gram table with `Qwen4ExpPLEUnquantizedEmbeddingMethod` (bf16), and model construction finishes. - The checkpoint's own INC metadata makes the choice: its `extra_config` has `".*ple.*": {"bits": 16, "data_type": "float"}`. If INC says the PLE layer is quantized, the old `NotImplementedError` is still raised, because there is no INC-quantized PLE format. - The branch is in `vllm/models/qwen4_exp/common/ngram_embedding.py`, so both the NVIDIA and the AMD Qwen4Exp paths get it. - A new unit test covers both cases. ## Validation Hardware: 2 × NVIDIA A100-SXM4-80GB. Before: commit `18f8f96025b556071eb627076f94df560fbd3a22`. After: this PR on top of that commit. Reproducer, from the issue. No request is needed, because the crash happens at startup: ```bash VLLM_ENGINE_READY_TIMEOUT_S=3600 vllm serve Intel/Qwen3.8-Flash-Next-W4A16-AutoRound \ --tensor-parallel-size 2 --enable-expert-parallel ``` | | Runs | Result | |---|---|---| | Before | 1 | Both workers raise `NotImplementedError: Qwen4Exp PLE embedding does not support quantization config INCConfig` during model construction, before weight loading. The engine exits with code 1. The `vllm/vllm-openai:nightly` image at the same commit gives the same error. | | After | 2 | No `NotImplementedError` in either run. Both workers build the PLE table, and model construction finishes (about 56 s). | After the fix, both workers log: ``` Initialized PLE embedding language_model.model.layers.1.ple.ple_embedding.ngram_embedding: quantization_method=Qwen4ExpPLEUnquantizedEmbeddingMethod, weight_dtype=torch.bfloat16, weight_device=cpu, pinned=True ``` Limitation: neither "after" run reached a ready server. Both runs next stopped at the weight download, with a `PermissionError` from `huggingface_hub`. The HF cache on the test machine had only the config and tokenizer files for this model, and we could not write to it. This happens after the bug site, in code the patch does not touch. The "before" run never got that far. No run here loaded the real weights, and no accuracy eval was run. Before merge, someone with the weights should run the reproducer to a ready server and run an accuracy check. Extra checks (not the reproducer): - The same command with `--load-format dummy`: weights loaded (35.44 GiB per GPU), CUDA graphs were captured, and the server became ready. One `/v1/completions` request returned 16 tokens. The text is noise, because the weights are random. - The checkpoint's safetensors headers on the Hub: the PLE n-gram table is 128 plain BF16 shards (`[2500012, 160]` each), with no `qweight`, `scales` or `qzeros`. This matches the unquantized method. - Unit tests: `pytest --noconftest tests/models/qwen4_exp/test_ple.py -k "inc_layer_config or rejects_unsupported or modelopt_exclusion or shard_exclusions or mixed_precision"` gives `6 passed, 69 deselected`. `--noconftest` was used because `tests/conftest.py` needs `tblib`, which the test env did not have. With the INC branch turned off in a separate process (tree unchanged), the new test's config gets the old `NotImplementedError` again. So the test catches the bug. - `pre-commit run --files` on the two changed files: every hook passed or was skipped, including ruff and mypy. ## Details Root cause: `from_quant_config` gets the model-level quantization config. That config describes the model body, which is not always the same as the PLE table. The method could only ask four config families whether the PLE layer is excluded: ModelOpt mixed precision, ModelOpt, compressed-tensors and FP8. Any other config fell through to the `NotImplementedError`. This checkpoint quantizes `model.language_model.layers` to 4 bits, but its `extra_config` marks `.*ple.*` as 16-bit float. `INCConfig` keeps this metadata and resolves it with `config_parser.resolve(layer, prefix)`, but the PLE code never asked it. How it was found: on the unpatched tree, a debug print just before the raise showed three things. The config was a plain `INCConfig`, with no handled class in its MRO. The prefix was `language_model.model.layers.1.ple.ple_embedding.ngram_embedding`. And `resolve(None, prefix)` returned `bits=16, quantized=False`. So the INC metadata is already mapped to vLLM names and matches the prefix. Only the dispatch ignored it. Fix: one `if` in `from_quant_config`. For an `INCConfig`, when `quant_config.config_parser.resolve(None, prefix).quantized` is false, it returns `Qwen4ExpPLEUnquantizedEmbeddingMethod`. `INCConfig.get_quant_method` uses the same `resolve(...).quantized` check for its own layers. Passing `None` as the layer is safe: `_resolve_raw` only uses the layer in `isinstance(layer, ParallelLMHead)` and `layer.__class__.__name__`, and both work with `None`. The compressed-tensors branch also passes `None`. Size: +6 lines in `ngram_embedding.py`, and +19 lines of test in `tests/models/qwen4_exp/test_ple.py`. That test file already runs in Buildkite (`.buildkite/test_areas/models_basic.yaml`). Design choice for review: INC's own `get_quant_method` treats every layer that is not linear or MoE as unquantized, whatever `resolve` says. This patch is stricter. An INC checkpoint that puts the PLE table inside a quantized block, with no 16-bit exclusion, still gets `NotImplementedError`. The new test checks this case on purpose. Please say if you prefer INC's looser behavior. --- <details> <summary> Pull Request Checklist </summary> - [x] I used vLLM's `/pr-checklist` skill. (Mandatory for agents, optional for humans). - [x] AI assistance was used during the creation of this PR. - [ ] **Design Fit:** Minimizes impact on core components, reuses existing functionality, and justifies added complexity. - [ ] **Testing and Validation:** Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI. - [ ] **Code Quality and Style:** Keeps code and comments clear and concise, and updates relevant documentation and examples. - [ ] **Pull Request Contents:** Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide. </details> Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
vadiklyutiy
requested review from
AndreasKaratzas,
DarkLight1337,
gau-nernst and
ywang96
as code owners
October 5, 2026 00:06
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
gau-nernst
approved these changes
Oct 5, 2026
Member
Author
|
/ci run |
|
❌ This PR is 12 commits behind upstream |
vadiklyutiy
enabled auto-merge (squash)
October 5, 2026 00:39
github-actions
Bot
added
the
ready
Member
Author
|
/ci run |
|
✅ Triggered Buildkite CI #92821 for commit |
3 of 4 tasks
Collaborator
CI selector (shadow): 1 test steps (1 jobs) instead of 36 (51 jobs)Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works. Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.
Selector would run (1)
Would skip (today's rules run them) (35)
Would add (today's rules do not run them) (0)none AMD mirrors: would skip (32)
AMD mirrors: would add (0)none 2 changed files · base |
1 task done
a17t
pushed a commit
to a17t/vllm-rdna
that referenced
this pull request
Oct 12, 2026
…ints (vllm-project#59990) Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> (cherry picked from commit 4f52fa3)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Fixes #59798.
Intel/Qwen3.8-Flash-Next-W4A16-AutoRoundis an AutoRound checkpoint, which vLLM loads asINCConfig. It failed at model construction withNotImplementedError: Qwen4Exp PLE embedding does not support quantization config INCConfig.Qwen4ExpPLEEmbeddingMethod.from_quant_confignow asksINCConfigwhether the PLE layer is quantized. #59431 does the same for compressed-tensors.Claims
Qwen4ExpPLEUnquantizedEmbeddingMethod(bf16), and model construction finishes.extra_confighas".*ple.*": {"bits": 16, "data_type": "float"}. If INC says the PLE layer is quantized, the oldNotImplementedErroris still raised, because there is no INC-quantized PLE format.vllm/models/qwen4_exp/common/ngram_embedding.py, so both the NVIDIA and the AMD Qwen4Exp paths get it.Validation
Hardware: 2 × NVIDIA A100-SXM4-80GB. Before: commit
18f8f96025b556071eb627076f94df560fbd3a22. After: this PR on top of that commit.Reproducer, from the issue. No request is needed, because the crash happens at startup:
NotImplementedError: Qwen4Exp PLE embedding does not support quantization config INCConfigduring model construction, before weight loading. The engine exits with code 1. Thevllm/vllm-openai:nightlyimage at the same commit gives the same error.After the fix, both workers log:
Limitation: neither "after" run reached a ready server. Both runs next stopped at the weight download, with a
PermissionErrorfromhuggingface_hub. The HF cache on the test machine had only the config and tokenizer files for this model, and we could not write to it. This happens after the bug site, in code the patch does not touch. The "before" run never got that far. No run here loaded the real weights, and no accuracy eval was run. Before merge, someone with the weights should run the reproducer to a ready server and run an accuracy check.Extra checks (not the reproducer):
--load-format dummy: weights loaded (35.44 GiB per GPU), CUDA graphs were captured, and the server became ready. One/v1/completionsrequest returned 16 tokens. The text is noise, because the weights are random.[2500012, 160]each), with noqweight,scalesorqzeros. This matches the unquantized method.pytest --noconftest tests/models/qwen4_exp/test_ple.py -k "inc_layer_config or rejects_unsupported or modelopt_exclusion or shard_exclusions or mixed_precision"gives6 passed, 69 deselected.--noconftestwas used becausetests/conftest.pyneedstblib, which the test env did not have. With the INC branch turned off in a separate process (tree unchanged), the new test's config gets the oldNotImplementedErroragain. So the test catches the bug.pre-commit run --fileson the two changed files: every hook passed or was skipped, including ruff and mypy.Details
Root cause:
from_quant_configgets the model-level quantization config. That config describes the model body, which is not always the same as the PLE table. The method could only ask four config families whether the PLE layer is excluded: ModelOpt mixed precision, ModelOpt, compressed-tensors and FP8. Any other config fell through to theNotImplementedError. This checkpoint quantizesmodel.language_model.layersto 4 bits, but itsextra_configmarks.*ple.*as 16-bit float.INCConfigkeeps this metadata and resolves it withconfig_parser.resolve(layer, prefix), but the PLE code never asked it.How it was found: on the unpatched tree, a debug print just before the raise showed three things. The config was a plain
INCConfig, with no handled class in its MRO. The prefix waslanguage_model.model.layers.1.ple.ple_embedding.ngram_embedding. Andresolve(None, prefix)returnedbits=16, quantized=False. So the INC metadata is already mapped to vLLM names and matches the prefix. Only the dispatch ignored it.Fix: one
ifinfrom_quant_config. For anINCConfig, whenquant_config.config_parser.resolve(None, prefix).quantizedis false, it returnsQwen4ExpPLEUnquantizedEmbeddingMethod.INCConfig.get_quant_methoduses the sameresolve(...).quantizedcheck for its own layers. PassingNoneas the layer is safe:_resolve_rawonly uses the layer inisinstance(layer, ParallelLMHead)andlayer.__class__.__name__, and both work withNone. The compressed-tensors branch also passesNone. Size: +6 lines inngram_embedding.py, and +19 lines of test intests/models/qwen4_exp/test_ple.py. That test file already runs in Buildkite (.buildkite/test_areas/models_basic.yaml).Design choice for review: INC's own
get_quant_methodtreats every layer that is not linear or MoE as unquantized, whateverresolvesays. This patch is stricter. An INC checkpoint that puts the PLE table inside a quantized block, with no 16-bit exclusion, still getsNotImplementedError. The new test checks this case on purpose. Please say if you prefer INC's looser behavior.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.