Conversation
Backport Qwen4Exp (vllm-project#53896) onto the v0.28.0 Spark GB10 branch so RadixArk/Qwen3.8-Flash-Next-NVFP4 can load. Select the FP8 PLE n-gram method under ModelOpt NVFP4 when *.ple.* is excluded (vllm-project#54765). Add docker/gb10/run-qwen38.sh for 2-node TP2 (BF16 KV, no PLE CPU offload). Co-authored-by: Cursor Grok 4.6 Signed-off-by: Ricardo <ricardo@asternos.com>
Docker creates an empty host directory when MODEL_HOST is missing, so vLLM treats /models/Qwen3.8-Flash-Next-NVFP4 as an invalid local repo. Serve RadixArk/Qwen3.8-Flash-Next-NVFP4 from the hub/cache unless the host directory already has config.json. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Ricardo <ricardo@asternos.com>
The vllm-project#53896 model_runner brought later-main APIs that do not exist on v0.28.0 (JitWarmupRegistry, profile_cudagraph_memory helper, BatchSharder). Spark TP2 workers then died on import. Keep the 0.28.0 runner and only add Qwen4 CircularBufferSpec / Mamba profile handling. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Ricardo <ricardo@asternos.com>
Co-authored-by: Cursor Grok 4.6 Signed-off-by: Ricardo <ricardo@asternos.com>
ncclCommInitRank CUDA OOM on rank 1 happens before weights load. The same NCCL flags work for DSv4, so a leftover dspark container is the usual cause on 128 GB unified memory. Stop those servers, print a GPU preflight, and default to 8k context / 4 seqs so PLE can stay on GPU. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Ricardo <ricardo@asternos.com>
is_fusion_moe_shared_experts_enabled() returns None when AITER is absent. Qwen3Next stored that on each MoE layer, then Qwen4Exp summed the flags and crashed with int + NoneType during model init. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Ricardo <ricardo@asternos.com>
Qwen4Exp PLE calls short_conv_state_shape(num_spec=...) after weight load. v0.28.0 only grew GDN/Mamba2 conv state for speculative tokens, so hybrid block-size alignment crashed with an unexpected keyword. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Ricardo <ricardo@asternos.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Get
RadixArk/Qwen3.8-Flash-Next-NVFP4loading and serving with TP=2 on the existing v0.28.0 Spark GB10 stack.Recent Spark bring-up fixes
GPUModelRunner(JitWarmupRegistry).Noneas disabled on NVIDIA.num_specinMambaStateShapeCalculator.short_conv_state_shapeso Qwen4Exp PLE hybrid block-size alignment works after weight load.Test Plan
On two Sparks: pull both nodes and restart both containers.
Test Result
test_short_conv_num_spec.py: 1 passed.Spark: weights loaded (62.31 GiB / ~459s) then crashed on
short_conv_state_shape(num_spec=...). That is fixed here.Model evaluation: not run.
AI assistance
AI assistance was used. A human submitter must review every changed line and run the Spark TP2 serve before treating this as production-ready.