[BUG] Exclude SKIP_TENSORS from get_layer_size() + new weight sync example for dpep - #37334
Merged
Merged
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request resolves a bug in get_layer_size by correctly excluding tensors that are not intended for loading, thereby preventing an OOM error with FusedMoE layers. The fix is sound and properly manages circular dependencies. Additionally, a new example for weight synchronization with FSDP is introduced. In this new example file, I identified a portability issue due to a hardcoded PYTHONPATH, which would hinder other users from running the example without changes. A fix has been suggested for this.
kylesayrs
approved these changes
Mar 17, 2026
kylesayrs
left a comment
kylesayrs
left a comment
Contributor
There was a problem hiding this comment.
Safe and good, thanks!
Contributor
|
Documentation preview: https://vllm--37334.org.readthedocs.build/en/37334/ |
mergify
Bot
added
documentation
26 tasks
robertgshaw2-redhat
added
the
ready
robertgshaw2-redhat
approved these changes
Mar 18, 2026
robertgshaw2-redhat
enabled auto-merge (squash)
March 18, 2026 22:29
SouthWest7
pushed a commit
to SouthWest7/vllm
that referenced
this pull request
Mar 27, 2026
…ample for dpep (vllm-project#37334) Signed-off-by: ahao-anyscale <ahao@anyscale.com>
khairulkabir1661
pushed a commit
to khairulkabir1661/vllm
that referenced
this pull request
Mar 27, 2026
…ample for dpep (vllm-project#37334) Signed-off-by: ahao-anyscale <ahao@anyscale.com>
JiantaoXu
pushed a commit
to JiantaoXu/vllm
that referenced
this pull request
Mar 28, 2026
…ample for dpep (vllm-project#37334) Signed-off-by: ahao-anyscale <ahao@anyscale.com>
mtparet
pushed a commit
to blackfuel-ai/vllm
that referenced
this pull request
Apr 9, 2026
…ample for dpep (vllm-project#37334) Signed-off-by: ahao-anyscale <ahao@anyscale.com>
1 task
alankessler
added a commit
to alankessler/vllm
that referenced
this pull request
Apr 13, 2026
…True The layerwise reload mechanism wraps weight loaders for all tensors not in SKIP_TENSORS. This prevents bias parameters from loading correctly during online FP8 quantization, leaving them as zeros. Qwen2 is the most visible case (bias=True on qkv_proj), but any architecture with biased linear layers is affected. Fixes: vllm-project#39663 Related: vllm-project#37334, vllm-project#38746 Signed-off-by: Alan Kessler <alankessler@gmail.com>
mystous
pushed a commit
to mystous/vllm_hybrid
that referenced
this pull request
May 10, 2026
…ample for dpep (vllm-project#37334) Signed-off-by: ahao-anyscale <ahao@anyscale.com>
my-other-github-account
pushed a commit
to my-other-github-account/vllm
that referenced
this pull request
May 15, 2026
…ample for dpep (vllm-project#37334) Signed-off-by: ahao-anyscale <ahao@anyscale.com>
my-other-github-account
pushed a commit
to my-other-github-account/vllm
that referenced
this pull request
May 15, 2026
…ample for dpep (vllm-project#37334) Signed-off-by: ahao-anyscale <ahao@anyscale.com>
mvanhorn
pushed a commit
to mvanhorn/vllm
that referenced
this pull request
Jun 4, 2026
…ample for dpep (vllm-project#37334) Signed-off-by: ahao-anyscale <ahao@anyscale.com> Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
plasticchris
pushed a commit
to plasticchris/vllm
that referenced
this pull request
Jul 20, 2026
…ample for dpep (vllm-project#37334) Signed-off-by: ahao-anyscale <ahao@anyscale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
get_layer_size()included SKIP_TENSORS buffers (e.g. _expert_map) when computing load_numel_total, but these buffers are never offloaded to meta device and never loaded via weight_loader. This caused load_numel to never reach load_numel_total for FusedMoE layers, preventing _layerwise_process from ever firing. As a result, all incoming expert weights accumulated indefinitely in info.loaded_weights until the GPU ran out of memory.During initialize_layerwise_reload, each layer's load_numel_total is set by get_layer_size(), which summed the numel of all parameters and buffers. For SharedFusedMoE layers, this included _expert_map (a 128-element int32 buffer) and other EP-related buffers listed in SKIP_TENSORS. These buffers are intentionally skipped by restore_layer_on_meta — they stay on GPU and are never loaded through weight_loader. This created a permanent gap (128 elements) between load_numel (150,994,944) and load_numel_total (150,995,072), so the completion condition load_numel >= load_numel_total was never satisfied. The layer was never materialized/freed, and cached weights grew until OOM.
Test Plan
New weight sync example for dpep using Qwen/Qwen3-30B-A3B
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.