[New model][Multimodal] Add DeepSeek-V4-Flash-Vision-Exp support - #54566
Conversation
|
Documentation preview: https://vllm--54566.org.readthedocs.build/en/54566/ |
|
Nice to see this — I'd independently built a smaller version of the same thing before you opened this (#54561), and landed on the same four LM couplings (OOV sentinels, the One verification you may find useful. I differentially checked your preprocessing against the reference implementation shipped in the checkpoint (
19/19 exact match, tensors included. So the resize solver, token-count formula and N-layout permutation all reproduce the reference. Offer: I can boot-verify this on SM120. I have 2× RTX PRO 6000 Blackwell (96 GB, TP=2, PCIe, no NVLink), the checkpoint local, and a working text-only baseline of this model on the per-model image to diff against. Happy to run whatever you want — boot, image correctness, token-count check, long-context, DSpark interaction — and post logs. Turnaround is usually well under an hour. Two cross-check numbers from my own runs, if they're useful as expected values: the repo's example images produce 313 and 109 image tokens respectively ( Two notes that may matter for SM120 specifically:
The one thing still open from my side is the position-dependent expansion: |
|
Happy to test this on GH200 with 144GB of VRAM (with a bit of offloading). I currently run 0731 and I'm happy to test-run this anytime. |
|
Built this branch from source and ran it on SM120. Three findings, in the order I hit them — the third is the interesting one and is SM120-specific. Environment. 2× RTX PRO 6000 Blackwell (96 GB, TP=2, PCIe, no NVLink), 1. The arch convertor doesn't reach the class that actually gets built — vision is silently disabled.
tuple(getattr(model_config.hf_config, "architectures", None) or []),and With mm disabled, Adding 2. Hash-layer gate bias has no destination. The Vision-Exp checkpoint adds 3. SM120: With 1 and 2 patched locally, weights load and it dies in warmup decode: via I did not patch past this one, since a Happy to re-run any candidate fix on this hardware — turnaround is well under an hour, and I can diff against a working text-only baseline of the same checkpoint. Also happy to test on the official per-model image if a build with this lands there. One unrelated note in case it saves you a cycle: |
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
|
Re-tested at
Finding 3 still blocks, unchanged, now the only thing between this and a working boot on SM120: vLLM-side frames, for the record: It fires during warmup decode (before any request), so it should reproduce on any SM120 boot of this branch. I didn't chase the exact non-contiguous view — Environment unchanged: 2× RTX PRO 6000 Blackwell, TP=2, flashinfer-python 0.6.18, |
|
Correction on finding 3 — I misattributed it, sorry. It is not caused by this PR. I ran the control I should have run before posting: your branch at So my "widening prefill rows to What the difference actually looks like, on identical hardware/flags/checkpoint:
So it is a main-vs-fork and/or FlashInfer-version issue on SM120, orthogonal to vision. Two candidates I can't yet separate: (a) FlashInfer 0.6.18 vs 0.6.13 changing the sparse-MLA SM120 entry point's layout expectations, or (b) main lacking whatever #41834 ("Add SM12x support for DeepSeek V4 Flash with essential fix", still open) carries — which is presumably why that per-model image exists in the first place. I'm testing your branch with FlashInfer pinned to the fork's 0.6.13+cu132 now and will report which of the two it is. If it turns out to be (b), then SM120 simply can't run DSv4 on main today and this PR is untestable here until #41834 lands — which would be worth knowing before anyone reads my earlier comment as a blocker on your work. Also worth correcting from my first comment: I said main "already carries the SM120 csrc files, so support largely landed." The presence of those files clearly doesn't mean the path works, given the above. |
|
Answer to the open question from my last comment: it is main, not FlashInfer. I rebuilt your branch with FlashInfer pinned to the exact build my working image uses (
So Practical consequence: this PR is not testable end-to-end on SM120 until the SM12x enablement lands on main. Nothing here is a blocker on your work, and my finding 3 should be disregarded entirely — findings 1 and 2 were the real ones and you have already fixed both. Happy to re-run the moment either this branch is rebased onto something carrying the SM12x fix, or a per-model image is cut with these commits — the build environment is set up and turnaround is minutes. And if it is useful for triaging #41834, I can bisect what specifically main is missing for the SM120 decode path; just say the word. Update — stacked on #41834: vision works end-to-end on SM120I did the thing I offered above rather than wait: grafted this PR's three commits onto #41834 and built it ( It works. Synthetic image (red circle top-left, blue square top-right, green triangle bottom-centre), 305 prompt tokens, 3.3 s:
6/6 on colour, shape and position, and it correctly spotted that the triangle is inverted. So this PR is functionally correct on SM120; it just needed the SM12x enablement underneath, exactly as the table above predicted. Zero 1. A real interaction bug between this PR and #41834This one matters to whoever merges these two, and it is silent — no crash, just wrong routing. #41834 hardens the hash router because const bool token_in_range = token_id >= 0 && token_id < hash_table_rows;
const bool is_pad_row = (is_padding != nullptr && is_padding[warp]) || !token_in_range;This PR identifies image tokens by Verified rather than reasoned, on one build, changing only that predicate:
The failure is Not hypothetical for this checkpoint: the specialised kernel is gated on Fix I used — memory-safety intent preserved, since image rows never gather const bool is_image = bias_vl != nullptr && token_id >= vocab_size;
const bool token_in_range = token_id >= 0 && token_id < hash_table_rows;
const bool is_pad_row = (is_padding != nullptr && is_padding[warp]) ||
(!token_in_range && !is_image);2. Not your bug — a note for anyone testing this on #41834The multimodal core was refactored after 2026-08-13. #41834's base still does 3. FlashInfer SM120 packed prefill rejects the vision configurationWith the vision checkpoint, startup dies in the FI kernel: Controlled: the text-only Happy to push the merged branch if it is useful for whoever lands these together. |
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
|
/ci run |
|
✅ Triggered Buildkite CI #86786 for commit |
|
Data point from two DGX Spark (GB10, sm_121, aarch64) nodes on the
With
|
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
|
/ci retry |
|
✅ Triggered Buildkite CI #86853 for commit |
|
/ci run |
|
✅ Triggered Buildkite CI #86869 for commit |
…m-project#54566) Signed-off-by: Isotr0py <Isotr0py@outlook.com>
… bias_vl and image_sentinel_lo in HPU MoE router factory (#1780) This PR consolidates 1 hourly-CI fix against vllm@`a56654d6de060495ff2db3b1d9ff0b187084d1a9`. ## Bug 1: Accept bias_vl and image_sentinel_lo in HPU MoE router factory - **State machine id**: fused_moe_router_bias_vl_kwarg_mismatch - **Commit**: 03d42df ### Root cause upstream vllm#54566 added bias_vl and image_sentinel_lo to create_fused_moe_router, and FusedMoEFactory forwards both unconditionally, so the HPU override raised TypeError on every MoE model build. ### Culprit Regression introduced by [PR #54566](vllm-project/vllm#54566). ### Fix mirror the upstream signature and forward both kwargs to FusedTopKBiasRouter, which owns the vision-bias routing path. Signed-off-by: Paweł Olejniczak <pawelx.olejniczak@intel.com>
|
SM120 field report: serving in production on main + one FlashInfer patch Follow-up to @tacos8me's SM120 blocker report (the The one remaining SM120 blocker is FlashInfer's dual-cache prefill dispatch: the first real image request fails with (num_heads follows TP — H=32/H=64 variants reported in flashinfer #4850). Text-only serving is unaffected. The fix is tracked in flashinfer #4802 (runtime-topk refactor); until it lands we run a two-arm local dispatch backport on stock 0.6.18. One trap worth flagging for other SM120 users: if startup fails with "requires a FlashInfer DSV4 sparse MLA decode specialization for (num_q_heads=16, top_k=128)", check Production shape (in case the recipe wants an SM120/PCIe data point — currently only GB200 NVL4 is listed): 4× RTX PRO 6000 Blackwell Server (PCIe, no NVLink), TP4 + EP, Numbers: DSpark k=3 mean accepted length 2.87 tok/step (62.4% acceptance; image tokens in the prompt don't degrade it); TPOT p50 110 ms at conc-32 — ≈ parity with our text-only 0731 deployment at k=6 (103 ms baseline); TTFT p50 2.2 s at 96-conc 4K prompts; 60K-token prompt + image prefills in ~21 s; zero errors across mixed text/image stress. Image-size matrix (upscale path / 384-token cap / 8:1 crop) all behave per spec. |
### What this PR does / why we need it Adds `DeepSeek-V4-Flash-Vision-Exp` support to vLLM Ascend `main`, paired with the vLLM v0.27.x API line. Roadmap and remaining validation tracking: #15462 Upstream behavior reference: vllm-project/vllm#54566 The paired vLLM release does not contain the DeepSeek-V4 vision processor and vision tower from the upstream PR. This change therefore carries the required v0.27-compatible implementation in vLLM Ascend without importing unmerged vLLM modules. ### Implementation - registers `DeepseekV4ForConditionalGeneration` and routes vision checkpoints to an Ascend multimodal wrapper; - provides the image processor, ViT, aligner, sentinel embeddings, placeholder expansion, and multimodal weight mapping required by vLLM v0.27; - preserves raw input token IDs through prefill for DeepSeek-V4 MoE routing; - implements hybrid `bias_vl` routing: text tokens retain hash/correction-bias routing while image sentinel tokens use dynamic routing with `bias_vl`; - builds paged original-KV sparse indices that make each complete image span bidirectionally visible while keeping causal SWA outside image spans; - passes original-KV image indices and compressed-KV top-k indices to the Ascend sparse-attention operator independently and simultaneously; - supports ModelSlim W8A8 checkpoints whose language quantization description remains in the causal-LM `model.*` / `lm_head.*` namespace; - keeps config-convertor registration safe in spawned DP engine processes. - enables DeepSeek-V4 Vision with a text-only DSpark drafter: image inputs are encoded by the target during prefill, while DSpark proposes subsequent text decode tokens; restores the draft architecture after VL config conversion, exposes the EAGLE3 hidden-state interface, and ignores draft-only vision router bias tensors without changing text routing. ### Runtime validation completed Validated in a 16-NPU Ascend container with: - vLLM: `6e448d0ea9bf3d88d898b65449ca6dc2aec170ac` (v0.27.1 API line) - checkpoint: DeepSeek-V4-Flash-Vision-Exp W8A8, with all BF16 `bias_vl` tensors included in the checkpoint overlay - release topology: TP4 / DP4 / EP16, block size 32 Results: - targeted processor, ModelSlim, MoE router, and DSA tests pass (90 tests in the combined suite; 59 tests in the focused follow-up suite); - the Ascend sparse-attention operator accepts widened original indices and simultaneous original/compressed indices with finite BF16 output; - all 79 checkpoint shards load successfully with the multimodal wrapper; - eager OpenAI-compatible serving passes text-only, single-image, ordered two-image, deterministic repeated-image, and invalid-image cases; - the single-image OCR result correctly reads `Hello, AI world!`, and the ordered two-image result preserves both image order and text; - ACLGraph mode passes torch compilation, PIECEWISE mixed prefill/decode capture, FULL decode capture, and runtime graph replay on all four DP leaders; - greedy text and single-image outputs are identical between eager and graph modes; - invalid image data returns an explicit HTTP 400 response. - DSpark eager and FULL_DECODE_ONLY target-graph serving pass text, single-image, two-image, and 149-token long-decode cases with byte-identical outputs; - eager and graph runs each accepted 153 of 180 draft tokens (85.0%); four concurrent graph-mode long decodes all passed, reaching 656/750 cumulative accepted/draft tokens (87.47%). - three 149-token requests pinned concurrently to one DP rank also pass byte-identically inside the captured 24-token graph range, with 375/414 accepted/draft tokens (90.58%). ### OCRBench V1 accuracy A complete 1,000-sample OCRBench V1 evaluation completed successfully with no request errors and scored **826/1000 (82.6)**. | Category | Score | Maximum | | --- | ---: | ---: | | Regular Text Recognition | 47 | 50 | | Irregular Text Recognition | 44 | 50 | | Artistic Text Recognition | 47 | 50 | | Handwriting Recognition | 31 | 50 | | Digit String Recognition | 35 | 50 | | Non-Semantic Text Recognition | 46 | 50 | | Scene Text-centric VQA | 178 | 200 | | Doc-oriented VQA | 176 | 200 | | Key Information Extraction | **171** | 200 | | Handwritten Mathematical Expression Recognition | **51** | 100 | | **Total** | **826** | **1000** | Evaluation configuration: concurrency 64, temperature 1.0, top_p 0.95, max_tokens 8192, seed 7, thinking enabled, reasoning_effort=high. For TP4 graph mode, capture batch sizes must be multiples of TP size. The validated graph launch therefore uses `--max-num-seqs 4`; `--max-num-seqs 1` is valid for eager smoke but not for this graph topology. ### User-facing change The model architecture can now be loaded and served on Ascend using the vLLM-compatible multimodal API. The validated W8A8 launch topology is TP4 / DP4 / EP16 in eager or ACLGraph mode. This PR is ready for implementation review. Marking it Ready for Review does not yet claim full production qualification or inclusion in the supported model matrix. ### Remaining release validation - processor, ViT/aligner, router, hidden-state, and logits numerical comparison against the upstream NVIDIA/HF reference; - full BF16 checkpoint validation and BF16/W8A8 accuracy comparison; - maximum 384-token image spans, non-aligned boundaries, heterogeneous images, mixed batches, and longer-context cases; - deployment alternatives such as TP8 / DP2; - fixed image-task accuracy suite and text-only regression suite; - startup, HBM, TTFT, TPOT, throughput, and regression measurements; - supported-model documentation and a public launch example. - automatic prefix caching passes with the DeepSeek-V4-specific contract: launch with `--block-size 32` and use prompts longer than 4K tokens. Three identical 5,014-token requests pinned to one DP rank reused 8,192 of 15,042 prompt tokens; DSpark remained active with 15/18 accepted/draft tokens. - the multimodal APC stack also passes: three identical 6,139-token long-prefix plus image requests all returned exact OCR text, reused 8,192 prompt tokens, recorded two multimodal-processor cache hits, and kept DSpark active at 15/18 accepted/draft tokens. - multimodal SSE streaming on the graph service reconstructs exact OCR text, reports `finish_reason=stop`, terminates with `[DONE]`, and exercises DSpark with 5/6 accepted/draft tokens. First-phase exclusions remain DSA context parallel and FlashComm1. DSpark is validated only as target-side multimodal prefill followed by text-only speculative decode; the drafter does not consume image embeddings or propose inside image spans. Signed-off-by: GDzhu01 <116337067+GDzhu01@users.noreply.github.com> - vLLM main: vllm-project/vllm@ba07e4a Signed-off-by: GDzhu01 <116337067+GDzhu01@users.noreply.github.com> Co-authored-by: MengLong Chen <71744434+dragondream-chen@users.noreply.github.com> Co-authored-by: RenYuKai <184603735+pgzddxx@users.noreply.github.com>
…ject#15457) ### What this PR does / why we need it Adds `DeepSeek-V4-Flash-Vision-Exp` support to vLLM Ascend `main`, paired with the vLLM v0.27.x API line. Roadmap and remaining validation tracking: vllm-project#15462 Upstream behavior reference: vllm-project/vllm#54566 The paired vLLM release does not contain the DeepSeek-V4 vision processor and vision tower from the upstream PR. This change therefore carries the required v0.27-compatible implementation in vLLM Ascend without importing unmerged vLLM modules. ### Implementation - registers `DeepseekV4ForConditionalGeneration` and routes vision checkpoints to an Ascend multimodal wrapper; - provides the image processor, ViT, aligner, sentinel embeddings, placeholder expansion, and multimodal weight mapping required by vLLM v0.27; - preserves raw input token IDs through prefill for DeepSeek-V4 MoE routing; - implements hybrid `bias_vl` routing: text tokens retain hash/correction-bias routing while image sentinel tokens use dynamic routing with `bias_vl`; - builds paged original-KV sparse indices that make each complete image span bidirectionally visible while keeping causal SWA outside image spans; - passes original-KV image indices and compressed-KV top-k indices to the Ascend sparse-attention operator independently and simultaneously; - supports ModelSlim W8A8 checkpoints whose language quantization description remains in the causal-LM `model.*` / `lm_head.*` namespace; - keeps config-convertor registration safe in spawned DP engine processes. - enables DeepSeek-V4 Vision with a text-only DSpark drafter: image inputs are encoded by the target during prefill, while DSpark proposes subsequent text decode tokens; restores the draft architecture after VL config conversion, exposes the EAGLE3 hidden-state interface, and ignores draft-only vision router bias tensors without changing text routing. ### Runtime validation completed Validated in a 16-NPU Ascend container with: - vLLM: `6e448d0ea9bf3d88d898b65449ca6dc2aec170ac` (v0.27.1 API line) - checkpoint: DeepSeek-V4-Flash-Vision-Exp W8A8, with all BF16 `bias_vl` tensors included in the checkpoint overlay - release topology: TP4 / DP4 / EP16, block size 32 Results: - targeted processor, ModelSlim, MoE router, and DSA tests pass (90 tests in the combined suite; 59 tests in the focused follow-up suite); - the Ascend sparse-attention operator accepts widened original indices and simultaneous original/compressed indices with finite BF16 output; - all 79 checkpoint shards load successfully with the multimodal wrapper; - eager OpenAI-compatible serving passes text-only, single-image, ordered two-image, deterministic repeated-image, and invalid-image cases; - the single-image OCR result correctly reads `Hello, AI world!`, and the ordered two-image result preserves both image order and text; - ACLGraph mode passes torch compilation, PIECEWISE mixed prefill/decode capture, FULL decode capture, and runtime graph replay on all four DP leaders; - greedy text and single-image outputs are identical between eager and graph modes; - invalid image data returns an explicit HTTP 400 response. - DSpark eager and FULL_DECODE_ONLY target-graph serving pass text, single-image, two-image, and 149-token long-decode cases with byte-identical outputs; - eager and graph runs each accepted 153 of 180 draft tokens (85.0%); four concurrent graph-mode long decodes all passed, reaching 656/750 cumulative accepted/draft tokens (87.47%). - three 149-token requests pinned concurrently to one DP rank also pass byte-identically inside the captured 24-token graph range, with 375/414 accepted/draft tokens (90.58%). ### OCRBench V1 accuracy A complete 1,000-sample OCRBench V1 evaluation completed successfully with no request errors and scored **826/1000 (82.6)**. | Category | Score | Maximum | | --- | ---: | ---: | | Regular Text Recognition | 47 | 50 | | Irregular Text Recognition | 44 | 50 | | Artistic Text Recognition | 47 | 50 | | Handwriting Recognition | 31 | 50 | | Digit String Recognition | 35 | 50 | | Non-Semantic Text Recognition | 46 | 50 | | Scene Text-centric VQA | 178 | 200 | | Doc-oriented VQA | 176 | 200 | | Key Information Extraction | **171** | 200 | | Handwritten Mathematical Expression Recognition | **51** | 100 | | **Total** | **826** | **1000** | Evaluation configuration: concurrency 64, temperature 1.0, top_p 0.95, max_tokens 8192, seed 7, thinking enabled, reasoning_effort=high. For TP4 graph mode, capture batch sizes must be multiples of TP size. The validated graph launch therefore uses `--max-num-seqs 4`; `--max-num-seqs 1` is valid for eager smoke but not for this graph topology. ### User-facing change The model architecture can now be loaded and served on Ascend using the vLLM-compatible multimodal API. The validated W8A8 launch topology is TP4 / DP4 / EP16 in eager or ACLGraph mode. This PR is ready for implementation review. Marking it Ready for Review does not yet claim full production qualification or inclusion in the supported model matrix. ### Remaining release validation - processor, ViT/aligner, router, hidden-state, and logits numerical comparison against the upstream NVIDIA/HF reference; - full BF16 checkpoint validation and BF16/W8A8 accuracy comparison; - maximum 384-token image spans, non-aligned boundaries, heterogeneous images, mixed batches, and longer-context cases; - deployment alternatives such as TP8 / DP2; - fixed image-task accuracy suite and text-only regression suite; - startup, HBM, TTFT, TPOT, throughput, and regression measurements; - supported-model documentation and a public launch example. - automatic prefix caching passes with the DeepSeek-V4-specific contract: launch with `--block-size 32` and use prompts longer than 4K tokens. Three identical 5,014-token requests pinned to one DP rank reused 8,192 of 15,042 prompt tokens; DSpark remained active with 15/18 accepted/draft tokens. - the multimodal APC stack also passes: three identical 6,139-token long-prefix plus image requests all returned exact OCR text, reused 8,192 prompt tokens, recorded two multimodal-processor cache hits, and kept DSpark active at 15/18 accepted/draft tokens. - multimodal SSE streaming on the graph service reconstructs exact OCR text, reports `finish_reason=stop`, terminates with `[DONE]`, and exercises DSpark with 5/6 accepted/draft tokens. First-phase exclusions remain DSA context parallel and FlashComm1. DSpark is validated only as target-side multimodal prefill followed by text-only speculative decode; the drafter does not consume image embeddings or propose inside image spans. Signed-off-by: GDzhu01 <116337067+GDzhu01@users.noreply.github.com> - vLLM main: vllm-project/vllm@ba07e4a Signed-off-by: GDzhu01 <116337067+GDzhu01@users.noreply.github.com> Co-authored-by: MengLong Chen <71744434+dragondream-chen@users.noreply.github.com> Co-authored-by: RenYuKai <184603735+pgzddxx@users.noreply.github.com>
…m-project#54566) Signed-off-by: Isotr0py <Isotr0py@outlook.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>
…ject#15457) ### What this PR does / why we need it Adds `DeepSeek-V4-Flash-Vision-Exp` support to vLLM Ascend `main`, paired with the vLLM v0.27.x API line. Roadmap and remaining validation tracking: vllm-project#15462 Upstream behavior reference: vllm-project/vllm#54566 The paired vLLM release does not contain the DeepSeek-V4 vision processor and vision tower from the upstream PR. This change therefore carries the required v0.27-compatible implementation in vLLM Ascend without importing unmerged vLLM modules. ### Implementation - registers `DeepseekV4ForConditionalGeneration` and routes vision checkpoints to an Ascend multimodal wrapper; - provides the image processor, ViT, aligner, sentinel embeddings, placeholder expansion, and multimodal weight mapping required by vLLM v0.27; - preserves raw input token IDs through prefill for DeepSeek-V4 MoE routing; - implements hybrid `bias_vl` routing: text tokens retain hash/correction-bias routing while image sentinel tokens use dynamic routing with `bias_vl`; - builds paged original-KV sparse indices that make each complete image span bidirectionally visible while keeping causal SWA outside image spans; - passes original-KV image indices and compressed-KV top-k indices to the Ascend sparse-attention operator independently and simultaneously; - supports ModelSlim W8A8 checkpoints whose language quantization description remains in the causal-LM `model.*` / `lm_head.*` namespace; - keeps config-convertor registration safe in spawned DP engine processes. - enables DeepSeek-V4 Vision with a text-only DSpark drafter: image inputs are encoded by the target during prefill, while DSpark proposes subsequent text decode tokens; restores the draft architecture after VL config conversion, exposes the EAGLE3 hidden-state interface, and ignores draft-only vision router bias tensors without changing text routing. ### Runtime validation completed Validated in a 16-NPU Ascend container with: - vLLM: `6e448d0ea9bf3d88d898b65449ca6dc2aec170ac` (v0.27.1 API line) - checkpoint: DeepSeek-V4-Flash-Vision-Exp W8A8, with all BF16 `bias_vl` tensors included in the checkpoint overlay - release topology: TP4 / DP4 / EP16, block size 32 Results: - targeted processor, ModelSlim, MoE router, and DSA tests pass (90 tests in the combined suite; 59 tests in the focused follow-up suite); - the Ascend sparse-attention operator accepts widened original indices and simultaneous original/compressed indices with finite BF16 output; - all 79 checkpoint shards load successfully with the multimodal wrapper; - eager OpenAI-compatible serving passes text-only, single-image, ordered two-image, deterministic repeated-image, and invalid-image cases; - the single-image OCR result correctly reads `Hello, AI world!`, and the ordered two-image result preserves both image order and text; - ACLGraph mode passes torch compilation, PIECEWISE mixed prefill/decode capture, FULL decode capture, and runtime graph replay on all four DP leaders; - greedy text and single-image outputs are identical between eager and graph modes; - invalid image data returns an explicit HTTP 400 response. - DSpark eager and FULL_DECODE_ONLY target-graph serving pass text, single-image, two-image, and 149-token long-decode cases with byte-identical outputs; - eager and graph runs each accepted 153 of 180 draft tokens (85.0%); four concurrent graph-mode long decodes all passed, reaching 656/750 cumulative accepted/draft tokens (87.47%). - three 149-token requests pinned concurrently to one DP rank also pass byte-identically inside the captured 24-token graph range, with 375/414 accepted/draft tokens (90.58%). ### OCRBench V1 accuracy A complete 1,000-sample OCRBench V1 evaluation completed successfully with no request errors and scored **826/1000 (82.6)**. | Category | Score | Maximum | | --- | ---: | ---: | | Regular Text Recognition | 47 | 50 | | Irregular Text Recognition | 44 | 50 | | Artistic Text Recognition | 47 | 50 | | Handwriting Recognition | 31 | 50 | | Digit String Recognition | 35 | 50 | | Non-Semantic Text Recognition | 46 | 50 | | Scene Text-centric VQA | 178 | 200 | | Doc-oriented VQA | 176 | 200 | | Key Information Extraction | **171** | 200 | | Handwritten Mathematical Expression Recognition | **51** | 100 | | **Total** | **826** | **1000** | Evaluation configuration: concurrency 64, temperature 1.0, top_p 0.95, max_tokens 8192, seed 7, thinking enabled, reasoning_effort=high. For TP4 graph mode, capture batch sizes must be multiples of TP size. The validated graph launch therefore uses `--max-num-seqs 4`; `--max-num-seqs 1` is valid for eager smoke but not for this graph topology. ### User-facing change The model architecture can now be loaded and served on Ascend using the vLLM-compatible multimodal API. The validated W8A8 launch topology is TP4 / DP4 / EP16 in eager or ACLGraph mode. This PR is ready for implementation review. Marking it Ready for Review does not yet claim full production qualification or inclusion in the supported model matrix. ### Remaining release validation - processor, ViT/aligner, router, hidden-state, and logits numerical comparison against the upstream NVIDIA/HF reference; - full BF16 checkpoint validation and BF16/W8A8 accuracy comparison; - maximum 384-token image spans, non-aligned boundaries, heterogeneous images, mixed batches, and longer-context cases; - deployment alternatives such as TP8 / DP2; - fixed image-task accuracy suite and text-only regression suite; - startup, HBM, TTFT, TPOT, throughput, and regression measurements; - supported-model documentation and a public launch example. - automatic prefix caching passes with the DeepSeek-V4-specific contract: launch with `--block-size 32` and use prompts longer than 4K tokens. Three identical 5,014-token requests pinned to one DP rank reused 8,192 of 15,042 prompt tokens; DSpark remained active with 15/18 accepted/draft tokens. - the multimodal APC stack also passes: three identical 6,139-token long-prefix plus image requests all returned exact OCR text, reused 8,192 prompt tokens, recorded two multimodal-processor cache hits, and kept DSpark active at 15/18 accepted/draft tokens. - multimodal SSE streaming on the graph service reconstructs exact OCR text, reports `finish_reason=stop`, terminates with `[DONE]`, and exercises DSpark with 5/6 accepted/draft tokens. First-phase exclusions remain DSA context parallel and FlashComm1. DSpark is validated only as target-side multimodal prefill followed by text-only speculative decode; the drafter does not consume image embeddings or propose inside image spans. Signed-off-by: GDzhu01 <116337067+GDzhu01@users.noreply.github.com> - vLLM main: vllm-project/vllm@ba07e4a Signed-off-by: GDzhu01 <116337067+GDzhu01@users.noreply.github.com> Co-authored-by: MengLong Chen <71744434+dragondream-chen@users.noreply.github.com> Co-authored-by: RenYuKai <184603735+pgzddxx@users.noreply.github.com>
Purpose
We have provided the docker image for this model, please refer to https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
Test Plan
Setup: 4x NVIDIA GB200 (TP=4, expert parallel), fp8 KV cache, block size 256,
Test Result
OCRBench (full 1000 samples)
DSpark (MTP) acceptance — aggregated over the full OCRBench run
Notes:
prompt do not degrade draft acceptance (the drafter reads image content via
the target's hidden states).
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.