[New model][DSv4] Serve DeepSeek-V4-Flash-Vision-Exp on SM12x Spark - #6
Merged
Merged
Conversation
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com> Co-authored-by: asterayx <asterayx@users.noreply.github.com>
Keep DSpark: image sentinels are in-vocab reserved ids. Proxy lm_head and MTP hidden states through the VL wrapper. Add run-vision.sh for the official Vision-Exp checkpoint. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Cursor Agent <cursoragent@cursor.com>
hf_config_override stamps n_predict from num_nextn_predict_layers. Vision-Exp ships 3 MTP layers; the MTP reuse check then rejects the proven Spark DSpark k=5 (5 % 3 != 0). DSpark drafts k tokens in one pass, so drop that leftover n_predict on the V4 draft. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Set dspark_draft_topk=None so MagicMock does not enter the Qwen3 top-k validator. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Cursor Agent <cursoragent@cursor.com>
After weight load, DSpark calls set_eagle3_aux_hidden_state_layers on the target. The VL wrapper is not SupportsEagle3, so Spark dies with "Model does not support EAGLE3 interface". Inherit the protocol; defaults unwrap via get_language_model() to the text backbone that already has EagleModelMixin. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Cursor Agent <cursoragent@cursor.com>
SupportsEagle3 is a runtime_checkable Protocol with a ClassVar, so issubclass() is not allowed. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Cursor Agent <cursoragent@cursor.com>
The convertor maps vision_n_layers>0 to DeepseekV4ForConditionalGeneration. After DSpark remaps the draft to DSparkDraftModel, update_arch_() ran the convertor again and rebuilt a second language_model stack. SWA cache names then collided: language_model.model.layers.0.attn.swa_cache. Skip the VL rewrite for DSpark/MTP draft archs and mark the V4 DSpark draft as _dsv4_vl_inner. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Vision-Exp Python passes bias_vl into _moe_C.topk_softplus_sqrt, but the Spark precompiled kernel only accepts 10 args. Detect the schema, use the old call for text/hash, and overwrite image sentinel rows in Python when the kernel has no bias_vl. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Cursor Agent <cursoragent@cursor.com>
The 10-arg GB10 fallback overwrote image-sentinel rows in Python, then bailed out with image_mask.any(). That host sync is illegal during CUDA-graph capture and kills Vision-Exp serve right after dummy warmup. Always apply the torch.where overwrite instead. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Cursor Agent <cursoragent@cursor.com>
FlashInfer SM120 DSV4 dual-cache prefill is only instantiated for SWA topk=128. Vision-Exp allocates window+vision_max_n_token (128+384=512) even on text dummy rows, so the 8192-token autotune dummy launched topk=512 + C4A extra and aborted. Slice text SWA to 128 and keep C4A. Image-widened rows drop C4A and use the single-cache 512 cubin. 0731 (width 128) is unchanged. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Vision-Exp ships num_nextn_predict_layers=3. Forcing k=5 matched 0731 but long text CoT acceptance fell from ~68% to ~27% (positions 4/5 near zero). Serve Vision with k=3 (next_n=4, a native SM12x decode width). Capture 6 seqs x 4 = 24. 0731 run.sh stays on k=5 / 36. Co-authored-by: Cursor Grok 4.6 Signed-off-by: Cursor Agent <cursoragent@cursor.com>
${SPECULATIVE_CONFIG:-{...}} ended at the first `}` so vLLM saw
...probabilistic"}} and refused --speculative-config. Assign JSON
defaults with quoted literals instead. Same for CUGRAPH_CFG.
Co-authored-by: Cursor Grok 4.6
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
asterayx
marked this pull request as ready for review
September 1, 2026 08:12
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
cursor Bot
pushed a commit
that referenced
this pull request
Sep 1, 2026
…stack. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: asterayx <asterayx@users.noreply.github.com>
cursor Bot
pushed a commit
that referenced
this pull request
Sep 1, 2026
docker/gb10/run.sh and run-vision.sh expose /metrics on host port 30001 (rank 0 only). Default scrape target and docs now match that stack. Co-authored-by: Cursor Grok 4.6 <cursoragent@cursor.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Serve official
deepseek-ai/DeepSeek-V4-Flash-Vision-Expon the 2× DGX Spark (GB10 / SM121) stack that is already green for text-only DeepSeek-V4-Flash-0731.This is not a duplicate of #1 / #2 / #3.
Upstream source: cherry-pick of vllm-project/vllm#54566.
Spark first-boot fixes on this branch:
k=5when Vision-Expn_predict=3(0731 still uses this).SupportsEagle3._moe_C.topk_softplus_sqrtis 10-arg. Detect the schema, keep the 10-arg hash path, overwrite image sentinel rows in Python.image_mask.any()(host sync) during CUDA-graph capture.k=3, matchingnum_nextn_predict_layers=3.k=5started at ~68% draft accept then fell to ~27% on the same text algorithm CoT that 0731 handled (positions 4/5 ~0.11/0.03).k=3isnext_n=4, a native SM12x decode width (no 5→6 pad). Capture 6 seqs × 4 = 24../docker/gb10/run.sh(0731) stays onk=5/ 36.Test Plan
Test Result
Capture-size (k=5 set unchanged, k=3 → 4/8/12/16/24), dual-prefill align, and n_predict override tests passed. Spark e2e not run here (no SM121).
AI assistance
AI assistance was used (Cursor Grok 4.6). A human submitter must review every changed line and run the Spark serve before treating this as production-ready.