Skip to content

[New model][Multimodal] Add DeepSeek-V4-Flash-Vision-Exp support - #54566

Merged
WoosukKwon merged 16 commits into
mainfrom
dsv4-vision
Sep 2, 2026
Merged

WoosukKwon merged 16 commits into
mainfrom
dsv4-vision

Conversation

@Isotr0py

@Isotr0py Isotr0py commented Aug 31, 2026

Copy link
Copy Markdown
Member

Purpose

We have provided the docker image for this model, please refer to https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp

Test Plan

Setup: 4x NVIDIA GB200 (TP=4, expert parallel), fp8 KV cache, block size 256,

export VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS="trtllm_fp4_block_scale_moe,flashinfer::trtllm_fp4_block_scale_moe"

vllm serve deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
    --tensor-parallel-size 4 \
    --enable-expert-parallel \
    --kv-cache-dtype fp8 \
    --block-size 256 \
    --max-model-len 32768 \
    --speculative-config '{"method":"dspark","model":"deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","num_speculative_tokens":3,"draft_sample_method":"probabilistic","enable_adaptive_verification":true}' \
    --allowed-local-media-path /mnt/lustre \
    --port 8271

Test Result

OCRBench (full 1000 samples)

Metric Value
Final Score 835 / 1000 (83.5%)
Request errors 0
Eval duration 312.6 s (~3.2 samples/s)
Category Correct / Total Accuracy
Regular Text Recognition 47 / 50 94.0%
Irregular Text Recognition 44 / 50 88.0%
Artistic Text Recognition 47 / 50 94.0%
Digit String Recognition 30 / 50 60.0%
Non-Semantic Text Recognition 45 / 50 90.0%
Handwriting Recognition 29 / 50 58.0%
Text Recognition (subtotal) 242 / 300 80.7%
Scene Text-centric VQA 179 / 200 89.5%
Doc-oriented VQA 180 / 200 90.0%
Key Information Extraction 185 / 200 92.5%
Handwritten Mathematical Expression Recognition 49 / 100 49.0%

DSpark (MTP) acceptance — aggregated over the full OCRBench run

Metric Value
Mean acceptance length (tokens/forward) 2.99 (interval range 2.28–3.40)
Overall draft acceptance rate 66.3% (266,788 accepted / 402,660 drafted)
Per-position acceptance rate (mean) 83.9% / 66.5% / 50.7% (draft pos 1/2/3)

Notes:

  • Acceptance is measured over mixed text+image traffic; image tokens in the
    prompt do not degrade draft acceptance (the drafter reads image content via
    the target's hidden states).

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Signed-off-by: Isotr0py <Isotr0py@outlook.com>
@Isotr0py Isotr0py changed the title [New model][Multimodal] Add DeepSeek-V4-Flash-Vision-Exp support [New model][Multimodal] Add DeepSeek-V4-Flash-Vision-Exp support Aug 31, 2026
@mergify

mergify Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Documentation preview: https://vllm--54566.org.readthedocs.build/en/54566/

@mergify mergify Bot added documentation Improvements or additions to documentation deepseek Related to DeepSeek models DSv4 nvidia labels Aug 31, 2026
@tacos8me

Copy link
Copy Markdown
Contributor

Nice to see this — I'd independently built a smaller version of the same thing before you opened this (#54561), and landed on the same four LM couplings (OOV sentinels, the tid2eid out-of-bounds guard, bias_vl expert selection, and the input-id validator) and the same shape for the visibility fix: widen prefill index rows to window_size + max_image_tokens, thread per-token left/right bounds into the index kernel, keep max_image_tokens == 0 on the text path. Encouraging convergence. Deriving the spans from mm_ranges rather than re-scanning for sentinel ids is nicer than what I had.

One verification you may find useful. I differentially checked your preprocessing against the reference implementation shipped in the checkpoint (inference/image_processor.py), by extracting the pure functions from this PR and running them side by side:

  • grid_tokens, solve_resize_ratio, safe_resize over 7 aspect ratios including the degenerate branches (max_w_float < 1.0, max_h_float < 2.0, e.g. 64×4096 and 2000×100)
  • build_image_block over 5 grid shapes, including odd n_llm_h (pad row) and the position-dependent compress_pad = 3 - start_pos % 4

19/19 exact match, tensors included. So the resize solver, token-count formula and N-layout permutation all reproduce the reference.

Offer: I can boot-verify this on SM120. I have 2× RTX PRO 6000 Blackwell (96 GB, TP=2, PCIe, no NVLink), the checkpoint local, and a working text-only baseline of this model on the per-model image to diff against. Happy to run whatever you want — boot, image correctness, token-count check, long-context, DSpark interaction — and post logs. Turnaround is usually well under an hour.

Two cross-check numbers from my own runs, if they're useful as expected values: the repo's example images produce 313 and 109 image tokens respectively (carrots.jpeg 1024×701 → n_llm 14×21; corn.jpeg 450×308 → n_llm 8×12), each including the 0–3 leading alignment pad.

Two notes that may matter for SM120 specifically:

  1. num_speculative_tokens must divide n_predict. This checkpoint ships num_nextn_predict_layers: 3 where DeepSeek-V4-Flash-0731 ships 1, so configs carrying over 5 from the text model fail validation with must be divisible by n_predict=3.
  2. The two model-side couplings have to be branch-free — inside the compile region, if mask.any() fails capture with "Could not guard on data-dependent expression" and nonzero() gives a data-dependent shape. Looks like you've already handled this, but it cost me a build cycle to find.

The one thing still open from my side is the position-dependent expansion: _get_prompt_updates can't express 3 - start_pos % 4 because its callables only get item_idx, so both of us do the expansion in _call_hf_processor and bypass the per-item cache. A small core extension exposing the running offset would let this use the standard path — happy to write it if that's a direction you'd take.

Signed-off-by: Isotr0py <Isotr0py@outlook.com>
@ebfio

ebfio commented Aug 31, 2026

Copy link
Copy Markdown

Happy to test this on GH200 with 144GB of VRAM (with a bit of offloading). I currently run 0731 and I'm happy to test-run this anytime.

@tacos8me

Copy link
Copy Markdown
Contributor

Built this branch from source and ran it on SM120. Three findings, in the order I hit them — the third is the interesting one and is SM120-specific.

Environment. 2× RTX PRO 6000 Blackwell (96 GB, TP=2, PCIe, no NVLink), edafe3dbe built with TORCH_CUDA_ARCH_LIST=12.0, torch 2.12.0+cu132 / CUDA 13.2, flashinfer-python 0.6.18, deepseek-ai/DeepSeek-V4-Flash-Vision-Exp local. Serve command:

vllm serve /models/ckpt --tensor-parallel-size 2 --max-model-len 131072 \
  --gpu-memory-utilization 0.92 --kv-cache-dtype fp8 --block-size 256 \
  --attention-backend FLASHINFER_MLA_SPARSE_DSV4 \
  --limit-mm-per-prompt '{"image":2,"video":0}' \
  --tokenizer-mode deepseek_v4 --trust-remote-code --no-enable-flashinfer-autotune

1. The arch convertor doesn't reach the class that actually gets built — vision is silently disabled.

DeepseekV4ModelArchConfigConvertor.get_architectures() rewrites ModelConfig.architectures (I confirmed it resolves to DeepseekV4ForConditionalGeneration, and the log prints Resolved architecture: DeepseekV4ForConditionalGeneration). But get_model_architecture() keys off the raw config — model_executor/model_loader/utils.py:247:

tuple(getattr(model_config.hf_config, "architectures", None) or []),

and hf_config.architectures is still ['DeepseekV4ForCausalLM']. So MULTIMODAL_REGISTRY._get_model_cls resolves the text class:

ValueError: Model class DeepseekV4ForCausalLM has no registered multimodal processor
WARNING [registry.py:124] Model /models/ckpt is treated as multimodal but has no
        registered multimodal processor; running in text-only mode.
INFO    [model.py:890] Disabled mm_prefix attention mode because multimodal inputs
        are configuration-disabled.

With mm disabled, _mark_tower_model skips the tower/aligner, and load then dies:

ValueError: There is no module or parameter named 'aligner' in DeepseekV4ForCausalLM

Adding --hf-overrides '{"architectures":["DeepseekV4ForConditionalGeneration"]}' clears the warning and gets past it, so the convertor probably needs to rewrite hf_config.architectures too (or get_model_architecture should prefer model_config.architectures).

2. Hash-layer gate bias has no destination.

KeyError: 'layers.0.ffn.gate.e_score_correction_bias'

The Vision-Exp checkpoint adds layers.{0,1,2}.ffn.gate.bias (absent from DeepSeek-V4-Flash-0731), which hf_to_vllm_mapper's ".ffn.gate.bias" -> ".ffn.gate.e_score_correction_bias" suffix rule renames onto a parameter that is None on the hash-routed layers (num_hash_layers: 3, "hash MoE doesn't use e_score_correction_bias"). The reference forward never reads it, so skipping it looks right. Two call sites need it — nvidia/model.py:1526 and :1587; patching only the first still fails at the second.

3. SM120: eidx must be contiguous in the sparse-MLA decode kernel.

With 1 and 2 patched locally, weights load and it dies in warmup decode:

File flashinfer/mla/_sparse_mla_sm120.py:402, in _paged_attention
  module.sparse_mla_sm120_paged_attention(...)
File csrc/sparse_mla_sm120.cu:225, in SparseMlaSm120PagedAttention
tvm.error.InternalError: Check failed: (eidx.IsContiguous()) is false: eidx must be contiguous

via flashinfer_trtllm_batch_decode_sparse_mla_dsv4_trtllm_batch_decode_sparse_mla_dsv4_sm120. The SM120 kernel requires a contiguous eidx, and widening prefill rows to window_size + max_image_tokens looks like it leaves the decode slice non-contiguous. Same class as #53574 ("pass contiguous C128A decode topk indices on SM120"), and sparse_mla.py already carries a TODO: support adaptive-width decode on SM120 (needs the FlashInfer SM120 kernel to accept a real row stride for eidx) — this may be that TODO coming due for the vision path.

I did not patch past this one, since a .contiguous() in the wrong place would just hide whichever layout assumption is actually wrong.


Happy to re-run any candidate fix on this hardware — turnaround is well under an hour, and I can diff against a working text-only baseline of the same checkpoint. Also happy to test on the official per-model image if a build with this lands there.

One unrelated note in case it saves you a cycle: num_speculative_tokens must divide n_predict, and this checkpoint ships num_nextn_predict_layers: 3 where DeepSeek-V4-Flash-0731 ships 1, so DSpark configs carried over from the text model fail validation with must be divisible by n_predict=3.

Signed-off-by: Isotr0py <Isotr0py@outlook.com>
@tacos8me

Copy link
Copy Markdown
Contributor

Re-tested at a1be19846 on the same SM120 box. Findings 1 and 2 are fixed — confirmed:

  • No --hf-overrides needed any more. The no registered multimodal processor warning and the Disabled mm_prefix attention mode line are both gone, so the VL class is now the one that actually gets built.
  • The hash-layer gate bias loads; no KeyError: layers.0.ffn.gate.e_score_correction_bias. Weights load clean through all 48 shards.

Finding 3 still blocks, unchanged, now the only thing between this and a working boot on SM120:

tvm.error.InternalError: Check failed: (eidx.IsContiguous()) is false: eidx must be contiguous

vLLM-side frames, for the record:

vllm/models/deepseek_v4/nvidia/model.py:1237        in forward
vllm/models/deepseek_v4/attention.py:385            in forward
vllm/models/deepseek_v4/attention.py:518            in _prepare_and_attn
vllm/models/deepseek_v4/attention.py:640            in _sparse_indexer_and_attn
vllm/models/deepseek_v4/nvidia/flashinfer_sparse.py:689  in forward_mqa
vllm/models/deepseek_v4/nvidia/flashinfer_sparse.py:635  in _forward_sparse_impl
vllm/models/deepseek_v4/nvidia/flashinfer_sparse.py:767  in _forward_decode
  -> flashinfer_trtllm_batch_decode_sparse_mla_dsv4(sparse_indices=swa_indices, ...)
     flashinfer/mla/_core.py:1169  _trtllm_batch_decode_sparse_mla_dsv4_sm120
     flashinfer/mla/_sparse_mla_sm120.py:402  _paged_attention
     csrc/sparse_mla_sm120.cu:225

It fires during warmup decode (before any request), so it should reproduce on any SM120 boot of this branch. I didn't chase the exact non-contiguous view — self.decode_swa_indices is allocated contiguous at sparse_swa.py:458, so it is something between there and the kernel, and you'll spot it much faster than I will from outside. I deliberately didn't paper over it with a .contiguous(), since that would hide whichever layout assumption is actually wrong rather than fix it.

Environment unchanged: 2× RTX PRO 6000 Blackwell, TP=2, flashinfer-python 0.6.18, --attention-backend FLASHINFER_MLA_SPARSE_DSV4 --kv-cache-dtype fp8 --block-size 256 --max-model-len 131072, no speculative config. Ping me on any candidate fix and I'll re-run — the build environment is already set up, so a turnaround is minutes rather than hours.

@tacos8me

Copy link
Copy Markdown
Contributor

Correction on finding 3 — I misattributed it, sorry. It is not caused by this PR.

I ran the control I should have run before posting: your branch at a1be19846 serving the text-only DeepSeek-V4-Flash-0731 checkpoint, same box, same flags, no vision anywhere (vision_n_layers absent, so max_image_tokens == 0 and every vision path is inert). It fails identically:

tvm.error.InternalError: Check failed: (eidx.IsContiguous()) is false: eidx must be contiguous
csrc/sparse_mla_sm120.cu:225  via  _trtllm_batch_decode_sparse_mla_dsv4_sm120

So my "widening prefill rows to window_size + max_image_tokens leaves the decode slice non-contiguous" guess is wrong — please don't spend time on it. Findings 1 and 2 were real and your fixes for them check out (confirmed in my previous comment).

What the difference actually looks like, on identical hardware/flags/checkpoint:

serves DSv4 on SM120 vLLM flashinfer
the per-model image I normally run yes 0.11.2.dev279+eldritch…cu132 fork 0.6.13+cu132
my build of this branch no (eidx) main-based 0.29.0.dev0 0.6.18 (pulled by pip during the build)

So it is a main-vs-fork and/or FlashInfer-version issue on SM120, orthogonal to vision. Two candidates I can't yet separate: (a) FlashInfer 0.6.18 vs 0.6.13 changing the sparse-MLA SM120 entry point's layout expectations, or (b) main lacking whatever #41834 ("Add SM12x support for DeepSeek V4 Flash with essential fix", still open) carries — which is presumably why that per-model image exists in the first place.

I'm testing your branch with FlashInfer pinned to the fork's 0.6.13+cu132 now and will report which of the two it is. If it turns out to be (b), then SM120 simply can't run DSv4 on main today and this PR is untestable here until #41834 lands — which would be worth knowing before anyone reads my earlier comment as a blocker on your work.

Also worth correcting from my first comment: I said main "already carries the SM120 csrc files, so support largely landed." The presence of those files clearly doesn't mean the path works, given the above.

@tacos8me

tacos8me commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Answer to the open question from my last comment: it is main, not FlashInfer.

I rebuilt your branch with FlashInfer pinned to the exact build my working image uses (0.6.13+cu132, copied wholesale from that image rather than resolved from the index) and re-ran the text-only control. Identical failure. Three cells, same box, same flags, same DeepSeek-V4-Flash-0731 checkpoint, no vision anywhere:

vLLM FlashInfer result
0.11.2.dev279+eldritch…cu132 (per-model image) 0.6.13+cu132 serves fine
main-based 0.29.0.dev0 (this branch) 0.6.18 eidx must be contiguous
main-based 0.29.0.dev0 (this branch) 0.6.13+cu132 eidx must be contiguous

So sparse_mla_sm120 decode is broken on vLLM main for DSv4 on SM120 regardless of FlashInfer version, independent of this PR. Presumably that is exactly what #41834 ("Add SM12x support for DeepSeek V4 Flash with essential fix", +32k, still open) carries and main does not — which would also explain why the DSv4 SM120 deployments in the wild are all on that per-model image.

Practical consequence: this PR is not testable end-to-end on SM120 until the SM12x enablement lands on main. Nothing here is a blocker on your work, and my finding 3 should be disregarded entirely — findings 1 and 2 were the real ones and you have already fixed both.

Happy to re-run the moment either this branch is rebased onto something carrying the SM12x fix, or a per-model image is cut with these commits — the build environment is set up and turnaround is minutes. And if it is useful for triaging #41834, I can bisect what specifically main is missing for the SM120 decode path; just say the word.


Update — stacked on #41834: vision works end-to-end on SM120

I did the thing I offered above rather than wait: grafted this PR's three commits onto #41834 and built it (TORCH_CUDA_ARCH_LIST=12.0, wheel 0.29.0.dev0+cu132). Hardware 2× RTX PRO 6000 Blackwell (SM120), TP2, DeepSeek-V4-Flash-Vision-Exp.

It works. Synthetic image (red circle top-left, blue square top-right, green triangle bottom-centre), 305 prompt tokens, 3.3 s:

Shape 1: Circle. Color: Red. Position: Top left. — Shape 2: Square. Color: Blue. Position: Top right. — Shape 3: Triangle. Color: Green. Position: Bottom center. … it's an inverted triangle

6/6 on colour, shape and position, and it correctly spotted that the triangle is inverted. So this PR is functionally correct on SM120; it just needed the SM12x enablement underneath, exactly as the table above predicted. Zero eidx must be contiguous in either log — final confirmation that my finding 3 was wrong and that main, not this PR, was the blocker.

1. A real interaction bug between this PR and #41834

This one matters to whoever merges these two, and it is silent — no crash, just wrong routing.

#41834 hardens the hash router because token_id is untrusted data ("an unbounded token_id gathers outside the table … an illegal access that takes down every rank"):

const bool token_in_range = token_id >= 0 && token_id < hash_table_rows;
const bool is_pad_row = (is_padding != nullptr && is_padding[warp]) || !token_in_range;

This PR identifies image tokens by token_id >= vocab_size. Both PRs allocate tid2eid as (config.vocab_size, config.num_experts_per_tok), so hash_table_rows == vocab_size — and every image sentinel is therefore classified as a padding row, so the bias_vl path never runs.

Verified rather than reasoned, on one build, changing only that predicate:

kernel tests/kernels/moe/test_topk_softplus_sqrt.py
naive stack (both changes, as git leaves them) 2 failedtest_fused_topk_softplus_sqrt_hash_bias_vl, both dtypes
sentinel exempted from the padding fallback 1669 passed (whole file)

The failure is 132/384 mismatched elements = 34.4%, and _make_mixed_input_ids makes every third token a sentinel (22 × 6 = 132) — i.e. precisely the image rows, nothing else.

Not hypothetical for this checkpoint: the specialised kernel is gated on use_hash && topk == 6 && renormalize && (num_experts == 256 || num_experts == 384), and DeepSeek-V4-Flash-Vision-Exp is n_routed_experts: 256, num_experts_per_tok: 6, num_hash_layers: 3 — so 3 of its 43 layers hit it. Only dsv4HashTopkSoftplusSqrt is exposed; the generic kernel already skips the gather for image rows via hash_row = USE_HASH && !use_vl_bias.

Fix I used — memory-safety intent preserved, since image rows never gather tid2eid:

const bool is_image = bias_vl != nullptr && token_id >= vocab_size;
const bool token_in_range = token_id >= 0 && token_id < hash_table_rows;
const bool is_pad_row = (is_padding != nullptr && is_padding[warp]) ||
                        (!token_in_range && !is_image);

2. Not your bug — a note for anyone testing this on #41834

The multimodal core was refactored after 2026-08-13. #41834's base still does processed_data.pop("input_ids") in _apply_hf_processor_text_mm; that line is gone in the base this PR targets, and your processor correctly returns BatchFeature({}) for the no-image case. On the stack that surfaces as KeyError: 'input_ids' during profile_run. This is an artefact of the older base, not a defect here — I only mention it because it blocks anyone trying the same combination. A four-line _call_hf_processor override supplying input_ids is enough.

3. FlashInfer SM120 packed prefill rejects the vision configuration

With the vision checkpoint, startup dies in the FI kernel:

Unsupported sparse-MLA prefill configuration:
model=DSV4 num_heads=32 topk=512 page_block_size=64 topk_extra=512 extra_page_block_size=64

Controlled: the text-only DeepSeek-V4-Flash-0731 checkpoint on the identical build with the same flags starts fine, so this is specific to the vision configuration rather than to the stack. Setting VLLM_DEEPSEEK_V4_FLASHINFER_SM120_PREFILL=0 (FlashMLA/Triton prefill, which #41834 gates for exactly this reason) makes it serve, and that is how the run above was produced. I did not chase which parameter is rejected; the plausible mechanism is the max_image_tokens widening of the topk/SWA column region (384 for this checkpoint), but I have not verified that, so treat it as a hypothesis.

Happy to push the merged branch if it is useful for whoever lands these together.

Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
@mergify

mergify Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Sep 1, 2026
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
@Isotr0py
Isotr0py marked this pull request as ready for review September 1, 2026 05:30
@Isotr0py

Isotr0py commented Sep 2, 2026

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #86786 for commit 3e3c938ebb83.

@Defilan

Defilan commented Sep 2, 2026

Copy link
Copy Markdown

Data point from two DGX Spark (GB10, sm_121, aarch64) nodes on the deepseekv4-flash-vision-arm64-cu130 image built from this branch (0.28.1rc1.dev137+g5ab628dd1, FlashInfer 0.6.18+cu130), DeepSeek-V4-Flash-Vision-Exp, TP2 + EP over RoCE, --kv-cache-dtype fp8 --block-size 256.

  1. SM12x needs two FlashInfer dual-cache prefill arms, not one. The stock wheel fails every prefill with Unsupported sparse-MLA prefill configuration: model=DSV4 num_heads=32 topk=512 page_block_size=64 topk_extra=512 extra_page_block_size=64; fix(sm120): support DeepSeek V4 Vision dual-cache prefill flashinfer-ai/flashinfer#4850 fixes that shape, after which the compress-128 layers fail with ... topk_extra=128 extra_page_block_size=2. Details and the extra arm are in a comment on that PR. With both arms the model serves at 2,079 tok/s prefill / 25.3 tok/s decode single-stream (39.1 with DSpark k=3) and vision works.

  2. enable_adaptive_verification: true from the model card's --speculative-config is rejected on this backend:

ValueError: Adaptive verification trims verification requests on device, which the DeepseekV4IndexerBackend attention backend does not support. Pass enable_adaptive_verification=false ...

With false, DSpark k=3 works (acceptance length 2.6 to 2.8 of 3); k=6 is slower (30.7 tok/s) because acceptance length does not grow. If the recipe is meant to cover Blackwell consumer / GB10 parts, the card's example may want that flag off or conditional.

  1. Minor: the mtp. weights are dropped for the Vision class in the weights mapper, but the folded DSpark draft still loads via method: dspark with model pointing at the same path, so speculative decoding does work on this image.

Signed-off-by: Isotr0py <Isotr0py@outlook.com>
@Isotr0py

Isotr0py commented Sep 2, 2026

Copy link
Copy Markdown
Member Author

/ci retry

@mergify mergify Bot added the multi-modality Related to multi-modality (#4194) label Sep 2, 2026
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #86853 for commit b3ebbd511c2b, running 9 failed step(s) from Buildkite CI #86786.

@Isotr0py

Isotr0py commented Sep 2, 2026

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #86869 for commit 047c353c99a6.

@WoosukKwon
WoosukKwon disabled auto-merge September 2, 2026 18:09
@WoosukKwon
WoosukKwon merged commit 1356635 into main Sep 2, 2026
207 of 210 checks passed
@github-project-automation github-project-automation Bot moved this from Ready to Done in NVIDIA Sep 2, 2026
mylibrar pushed a commit to tanyuqian/vllm that referenced this pull request Sep 3, 2026
iboiko-habana pushed a commit to vllm-project/vllm-gaudi that referenced this pull request Sep 3, 2026
… bias_vl and image_sentinel_lo in HPU MoE router factory (#1780)

This PR consolidates 1 hourly-CI fix against
vllm@`a56654d6de060495ff2db3b1d9ff0b187084d1a9`.
## Bug 1: Accept bias_vl and image_sentinel_lo in HPU MoE router factory

- **State machine id**: fused_moe_router_bias_vl_kwarg_mismatch
- **Commit**: 03d42df

### Root cause
upstream vllm#54566 added bias_vl and image_sentinel_lo to
create_fused_moe_router, and FusedMoEFactory forwards both
unconditionally, so the HPU override raised TypeError on every MoE model
build.

### Culprit
Regression introduced by [PR
#54566](vllm-project/vllm#54566).

### Fix
mirror the upstream signature and forward both kwargs to
FusedTopKBiasRouter, which owns the vision-bias routing path.

Signed-off-by: Paweł Olejniczak <pawelx.olejniczak@intel.com>
@Hefulalala

Hefulalala commented Sep 3, 2026

Copy link
Copy Markdown

SM120 field report: serving in production on main + one FlashInfer patch

Follow-up to @tacos8me's SM120 blocker report (the eidx must be contiguous finding): that decode failure is gone on current main — we're running this model in production on 0.28.1rc1.dev349+gee0a4c46a (Sep 3) with no #41834 code, no eidx errors, warmup and CUDA-graph capture clean.

The one remaining SM120 blocker is FlashInfer's dual-cache prefill dispatch: the first real image request fails with

Unsupported sparse-MLA prefill configuration: model=DSV4 num_heads=16 topk=512 topk_extra=512 extra_page_block_size=64

(num_heads follows TP — H=32/H=64 variants reported in flashinfer #4850). Text-only serving is unaffected. The fix is tracked in flashinfer #4802 (runtime-topk refactor); until it lands we run a two-arm local dispatch backport on stock 0.6.18.

One trap worth flagging for other SM120 users: if startup fails with "requires a FlashInfer DSV4 sparse MLA decode specialization for (num_q_heads=16, top_k=128)", check nvcc on PATH first. has_flashinfer() returns False without nvcc when no prebuilt cubin is present, and the resulting error misattributes the cause to the dispatch table. This cost us a debugging cycle on a box where /usr/local/cuda/bin wasn't in the unit's PATH.

Production shape (in case the recipe wants an SM120/PCIe data point — currently only GB200 NVL4 is listed): 4× RTX PRO 6000 Blackwell Server (PCIe, no NVLink), TP4 + EP, --kv-cache-dtype fp8 --block-size 256 --gpu-memory-utilization 0.90 --max-model-len 524288 (also validated at 1048576), DSpark num_speculative_tokens=3 + SimpleCPUOffloadConnector 128 GiB.

Numbers: DSpark k=3 mean accepted length 2.87 tok/step (62.4% acceptance; image tokens in the prompt don't degrade it); TPOT p50 110 ms at conc-32 — ≈ parity with our text-only 0731 deployment at k=6 (103 ms baseline); TTFT p50 2.2 s at 96-conc 4K prompts; 60K-token prompt + image prefills in ~21 s; zero errors across mixed text/image stress. Image-size matrix (upscale path / 384-token cap / 8:1 crop) all behave per spec.

MaxwellBrohm added a commit to MaxwellBrohm/llm-catalog-archive that referenced this pull request Sep 3, 2026
weijinqian0 pushed a commit to vllm-project/vllm-ascend that referenced this pull request Sep 4, 2026
### What this PR does / why we need it

Adds `DeepSeek-V4-Flash-Vision-Exp` support to vLLM Ascend `main`,
paired with
the vLLM v0.27.x API line.

Roadmap and remaining validation tracking: #15462

Upstream behavior reference: vllm-project/vllm#54566

The paired vLLM release does not contain the DeepSeek-V4 vision
processor and
vision tower from the upstream PR. This change therefore carries the
required
v0.27-compatible implementation in vLLM Ascend without importing
unmerged
vLLM modules.

### Implementation

- registers `DeepseekV4ForConditionalGeneration` and routes vision
  checkpoints to an Ascend multimodal wrapper;
- provides the image processor, ViT, aligner, sentinel embeddings,
placeholder
  expansion, and multimodal weight mapping required by vLLM v0.27;
- preserves raw input token IDs through prefill for DeepSeek-V4 MoE
routing;
- implements hybrid `bias_vl` routing: text tokens retain
hash/correction-bias
routing while image sentinel tokens use dynamic routing with `bias_vl`;
- builds paged original-KV sparse indices that make each complete image
span
  bidirectionally visible while keeping causal SWA outside image spans;
- passes original-KV image indices and compressed-KV top-k indices to
the
  Ascend sparse-attention operator independently and simultaneously;
- supports ModelSlim W8A8 checkpoints whose language quantization
description
  remains in the causal-LM `model.*` / `lm_head.*` namespace;
- keeps config-convertor registration safe in spawned DP engine
processes.
- enables DeepSeek-V4 Vision with a text-only DSpark drafter: image
inputs are
encoded by the target during prefill, while DSpark proposes subsequent
text
decode tokens; restores the draft architecture after VL config
conversion,
exposes the EAGLE3 hidden-state interface, and ignores draft-only vision
  router bias tensors without changing text routing.

### Runtime validation completed

Validated in a 16-NPU Ascend container with:

- vLLM: `6e448d0ea9bf3d88d898b65449ca6dc2aec170ac` (v0.27.1 API line)
- checkpoint: DeepSeek-V4-Flash-Vision-Exp W8A8, with all BF16 `bias_vl`
  tensors included in the checkpoint overlay
- release topology: TP4 / DP4 / EP16, block size 32

Results:

- targeted processor, ModelSlim, MoE router, and DSA tests pass (90
tests in
  the combined suite; 59 tests in the focused follow-up suite);
- the Ascend sparse-attention operator accepts widened original indices
and
  simultaneous original/compressed indices with finite BF16 output;
- all 79 checkpoint shards load successfully with the multimodal
wrapper;
- eager OpenAI-compatible serving passes text-only, single-image,
ordered
  two-image, deterministic repeated-image, and invalid-image cases;
- the single-image OCR result correctly reads `Hello, AI world!`, and
the
  ordered two-image result preserves both image order and text;
- ACLGraph mode passes torch compilation, PIECEWISE mixed prefill/decode
  capture, FULL decode capture, and runtime graph replay on all four DP
  leaders;
- greedy text and single-image outputs are identical between eager and
graph
  modes;
- invalid image data returns an explicit HTTP 400 response.
- DSpark eager and FULL_DECODE_ONLY target-graph serving pass text,
single-image,
two-image, and 149-token long-decode cases with byte-identical outputs;
- eager and graph runs each accepted 153 of 180 draft tokens (85.0%);
four
concurrent graph-mode long decodes all passed, reaching 656/750
cumulative
  accepted/draft tokens (87.47%).
- three 149-token requests pinned concurrently to one DP rank also pass
byte-identically inside the captured 24-token graph range, with 375/414
  accepted/draft tokens (90.58%).

### OCRBench V1 accuracy

A complete 1,000-sample OCRBench V1 evaluation completed successfully
with no request errors and scored **826/1000 (82.6)**.

| Category | Score | Maximum |
| --- | ---: | ---: |
| Regular Text Recognition | 47 | 50 |
| Irregular Text Recognition | 44 | 50 |
| Artistic Text Recognition | 47 | 50 |
| Handwriting Recognition | 31 | 50 |
| Digit String Recognition | 35 | 50 |
| Non-Semantic Text Recognition | 46 | 50 |
| Scene Text-centric VQA | 178 | 200 |
| Doc-oriented VQA | 176 | 200 |
| Key Information Extraction | **171** | 200 |
| Handwritten Mathematical Expression Recognition | **51** | 100 |
| **Total** | **826** | **1000** |

Evaluation configuration: concurrency 64, temperature 1.0, top_p 0.95,
max_tokens 8192, seed 7, thinking enabled, reasoning_effort=high.
For TP4 graph mode, capture batch sizes must be multiples of TP size.
The
validated graph launch therefore uses `--max-num-seqs 4`;
`--max-num-seqs 1`
is valid for eager smoke but not for this graph topology.

### User-facing change

The model architecture can now be loaded and served on Ascend using the
vLLM-compatible multimodal API. The validated W8A8 launch topology is
TP4 /
DP4 / EP16 in eager or ACLGraph mode.

This PR is ready for implementation review. Marking it Ready for Review
does
not yet claim full production qualification or inclusion in the
supported
model matrix.

### Remaining release validation

- processor, ViT/aligner, router, hidden-state, and logits numerical
comparison
  against the upstream NVIDIA/HF reference;
- full BF16 checkpoint validation and BF16/W8A8 accuracy comparison;
- maximum 384-token image spans, non-aligned boundaries, heterogeneous
images,
  mixed batches, and longer-context cases;
- deployment alternatives such as TP8 / DP2;
- fixed image-task accuracy suite and text-only regression suite;
- startup, HBM, TTFT, TPOT, throughput, and regression measurements;
- supported-model documentation and a public launch example.
- automatic prefix caching passes with the DeepSeek-V4-specific
contract:
launch with `--block-size 32` and use prompts longer than 4K tokens.
Three
identical 5,014-token requests pinned to one DP rank reused 8,192 of
15,042
prompt tokens; DSpark remained active with 15/18 accepted/draft tokens.
- the multimodal APC stack also passes: three identical 6,139-token
long-prefix plus image requests all returned exact OCR text, reused
8,192
prompt tokens, recorded two multimodal-processor cache hits, and kept
DSpark
  active at 15/18 accepted/draft tokens.
- multimodal SSE streaming on the graph service reconstructs exact OCR
text,
reports `finish_reason=stop`, terminates with `[DONE]`, and exercises
DSpark
  with 5/6 accepted/draft tokens.

First-phase exclusions remain DSA context parallel and FlashComm1.
DSpark is
validated only as target-side multimodal prefill followed by text-only
speculative decode; the drafter does not consume image embeddings or
propose
inside image spans.

Signed-off-by: GDzhu01 <116337067+GDzhu01@users.noreply.github.com>


- vLLM main:
vllm-project/vllm@ba07e4a

Signed-off-by: GDzhu01 <116337067+GDzhu01@users.noreply.github.com>
Co-authored-by: MengLong Chen <71744434+dragondream-chen@users.noreply.github.com>
Co-authored-by: RenYuKai <184603735+pgzddxx@users.noreply.github.com>
Lethobenthos20 pushed a commit to Lethobenthos20/vllm-ascend that referenced this pull request Sep 4, 2026
…ject#15457)

### What this PR does / why we need it

Adds `DeepSeek-V4-Flash-Vision-Exp` support to vLLM Ascend `main`,
paired with
the vLLM v0.27.x API line.

Roadmap and remaining validation tracking: vllm-project#15462

Upstream behavior reference: vllm-project/vllm#54566

The paired vLLM release does not contain the DeepSeek-V4 vision
processor and
vision tower from the upstream PR. This change therefore carries the
required
v0.27-compatible implementation in vLLM Ascend without importing
unmerged
vLLM modules.

### Implementation

- registers `DeepseekV4ForConditionalGeneration` and routes vision
  checkpoints to an Ascend multimodal wrapper;
- provides the image processor, ViT, aligner, sentinel embeddings,
placeholder
  expansion, and multimodal weight mapping required by vLLM v0.27;
- preserves raw input token IDs through prefill for DeepSeek-V4 MoE
routing;
- implements hybrid `bias_vl` routing: text tokens retain
hash/correction-bias
routing while image sentinel tokens use dynamic routing with `bias_vl`;
- builds paged original-KV sparse indices that make each complete image
span
  bidirectionally visible while keeping causal SWA outside image spans;
- passes original-KV image indices and compressed-KV top-k indices to
the
  Ascend sparse-attention operator independently and simultaneously;
- supports ModelSlim W8A8 checkpoints whose language quantization
description
  remains in the causal-LM `model.*` / `lm_head.*` namespace;
- keeps config-convertor registration safe in spawned DP engine
processes.
- enables DeepSeek-V4 Vision with a text-only DSpark drafter: image
inputs are
encoded by the target during prefill, while DSpark proposes subsequent
text
decode tokens; restores the draft architecture after VL config
conversion,
exposes the EAGLE3 hidden-state interface, and ignores draft-only vision
  router bias tensors without changing text routing.

### Runtime validation completed

Validated in a 16-NPU Ascend container with:

- vLLM: `6e448d0ea9bf3d88d898b65449ca6dc2aec170ac` (v0.27.1 API line)
- checkpoint: DeepSeek-V4-Flash-Vision-Exp W8A8, with all BF16 `bias_vl`
  tensors included in the checkpoint overlay
- release topology: TP4 / DP4 / EP16, block size 32

Results:

- targeted processor, ModelSlim, MoE router, and DSA tests pass (90
tests in
  the combined suite; 59 tests in the focused follow-up suite);
- the Ascend sparse-attention operator accepts widened original indices
and
  simultaneous original/compressed indices with finite BF16 output;
- all 79 checkpoint shards load successfully with the multimodal
wrapper;
- eager OpenAI-compatible serving passes text-only, single-image,
ordered
  two-image, deterministic repeated-image, and invalid-image cases;
- the single-image OCR result correctly reads `Hello, AI world!`, and
the
  ordered two-image result preserves both image order and text;
- ACLGraph mode passes torch compilation, PIECEWISE mixed prefill/decode
  capture, FULL decode capture, and runtime graph replay on all four DP
  leaders;
- greedy text and single-image outputs are identical between eager and
graph
  modes;
- invalid image data returns an explicit HTTP 400 response.
- DSpark eager and FULL_DECODE_ONLY target-graph serving pass text,
single-image,
two-image, and 149-token long-decode cases with byte-identical outputs;
- eager and graph runs each accepted 153 of 180 draft tokens (85.0%);
four
concurrent graph-mode long decodes all passed, reaching 656/750
cumulative
  accepted/draft tokens (87.47%).
- three 149-token requests pinned concurrently to one DP rank also pass
byte-identically inside the captured 24-token graph range, with 375/414
  accepted/draft tokens (90.58%).

### OCRBench V1 accuracy

A complete 1,000-sample OCRBench V1 evaluation completed successfully
with no request errors and scored **826/1000 (82.6)**.

| Category | Score | Maximum |
| --- | ---: | ---: |
| Regular Text Recognition | 47 | 50 |
| Irregular Text Recognition | 44 | 50 |
| Artistic Text Recognition | 47 | 50 |
| Handwriting Recognition | 31 | 50 |
| Digit String Recognition | 35 | 50 |
| Non-Semantic Text Recognition | 46 | 50 |
| Scene Text-centric VQA | 178 | 200 |
| Doc-oriented VQA | 176 | 200 |
| Key Information Extraction | **171** | 200 |
| Handwritten Mathematical Expression Recognition | **51** | 100 |
| **Total** | **826** | **1000** |

Evaluation configuration: concurrency 64, temperature 1.0, top_p 0.95,
max_tokens 8192, seed 7, thinking enabled, reasoning_effort=high.
For TP4 graph mode, capture batch sizes must be multiples of TP size.
The
validated graph launch therefore uses `--max-num-seqs 4`;
`--max-num-seqs 1`
is valid for eager smoke but not for this graph topology.

### User-facing change

The model architecture can now be loaded and served on Ascend using the
vLLM-compatible multimodal API. The validated W8A8 launch topology is
TP4 /
DP4 / EP16 in eager or ACLGraph mode.

This PR is ready for implementation review. Marking it Ready for Review
does
not yet claim full production qualification or inclusion in the
supported
model matrix.

### Remaining release validation

- processor, ViT/aligner, router, hidden-state, and logits numerical
comparison
  against the upstream NVIDIA/HF reference;
- full BF16 checkpoint validation and BF16/W8A8 accuracy comparison;
- maximum 384-token image spans, non-aligned boundaries, heterogeneous
images,
  mixed batches, and longer-context cases;
- deployment alternatives such as TP8 / DP2;
- fixed image-task accuracy suite and text-only regression suite;
- startup, HBM, TTFT, TPOT, throughput, and regression measurements;
- supported-model documentation and a public launch example.
- automatic prefix caching passes with the DeepSeek-V4-specific
contract:
launch with `--block-size 32` and use prompts longer than 4K tokens.
Three
identical 5,014-token requests pinned to one DP rank reused 8,192 of
15,042
prompt tokens; DSpark remained active with 15/18 accepted/draft tokens.
- the multimodal APC stack also passes: three identical 6,139-token
long-prefix plus image requests all returned exact OCR text, reused
8,192
prompt tokens, recorded two multimodal-processor cache hits, and kept
DSpark
  active at 15/18 accepted/draft tokens.
- multimodal SSE streaming on the graph service reconstructs exact OCR
text,
reports `finish_reason=stop`, terminates with `[DONE]`, and exercises
DSpark
  with 5/6 accepted/draft tokens.

First-phase exclusions remain DSA context parallel and FlashComm1.
DSpark is
validated only as target-side multimodal prefill followed by text-only
speculative decode; the drafter does not consume image embeddings or
propose
inside image spans.

Signed-off-by: GDzhu01 <116337067+GDzhu01@users.noreply.github.com>


- vLLM main:
vllm-project/vllm@ba07e4a

Signed-off-by: GDzhu01 <116337067+GDzhu01@users.noreply.github.com>
Co-authored-by: MengLong Chen <71744434+dragondream-chen@users.noreply.github.com>
Co-authored-by: RenYuKai <184603735+pgzddxx@users.noreply.github.com>
voipmonitor pushed a commit to voipmonitor/vllm that referenced this pull request Sep 5, 2026
…m-project#54566)

Signed-off-by: Isotr0py <Isotr0py@outlook.com>

Signed-off-by: Martin Vit <martin@voipmonitor.org>
sunny-rain-63 pushed a commit to sunny-rain-63/vllm-ascend that referenced this pull request Sep 12, 2026
…ject#15457)

### What this PR does / why we need it

Adds `DeepSeek-V4-Flash-Vision-Exp` support to vLLM Ascend `main`,
paired with
the vLLM v0.27.x API line.

Roadmap and remaining validation tracking: vllm-project#15462

Upstream behavior reference: vllm-project/vllm#54566

The paired vLLM release does not contain the DeepSeek-V4 vision
processor and
vision tower from the upstream PR. This change therefore carries the
required
v0.27-compatible implementation in vLLM Ascend without importing
unmerged
vLLM modules.

### Implementation

- registers `DeepseekV4ForConditionalGeneration` and routes vision
  checkpoints to an Ascend multimodal wrapper;
- provides the image processor, ViT, aligner, sentinel embeddings,
placeholder
  expansion, and multimodal weight mapping required by vLLM v0.27;
- preserves raw input token IDs through prefill for DeepSeek-V4 MoE
routing;
- implements hybrid `bias_vl` routing: text tokens retain
hash/correction-bias
routing while image sentinel tokens use dynamic routing with `bias_vl`;
- builds paged original-KV sparse indices that make each complete image
span
  bidirectionally visible while keeping causal SWA outside image spans;
- passes original-KV image indices and compressed-KV top-k indices to
the
  Ascend sparse-attention operator independently and simultaneously;
- supports ModelSlim W8A8 checkpoints whose language quantization
description
  remains in the causal-LM `model.*` / `lm_head.*` namespace;
- keeps config-convertor registration safe in spawned DP engine
processes.
- enables DeepSeek-V4 Vision with a text-only DSpark drafter: image
inputs are
encoded by the target during prefill, while DSpark proposes subsequent
text
decode tokens; restores the draft architecture after VL config
conversion,
exposes the EAGLE3 hidden-state interface, and ignores draft-only vision
  router bias tensors without changing text routing.

### Runtime validation completed

Validated in a 16-NPU Ascend container with:

- vLLM: `6e448d0ea9bf3d88d898b65449ca6dc2aec170ac` (v0.27.1 API line)
- checkpoint: DeepSeek-V4-Flash-Vision-Exp W8A8, with all BF16 `bias_vl`
  tensors included in the checkpoint overlay
- release topology: TP4 / DP4 / EP16, block size 32

Results:

- targeted processor, ModelSlim, MoE router, and DSA tests pass (90
tests in
  the combined suite; 59 tests in the focused follow-up suite);
- the Ascend sparse-attention operator accepts widened original indices
and
  simultaneous original/compressed indices with finite BF16 output;
- all 79 checkpoint shards load successfully with the multimodal
wrapper;
- eager OpenAI-compatible serving passes text-only, single-image,
ordered
  two-image, deterministic repeated-image, and invalid-image cases;
- the single-image OCR result correctly reads `Hello, AI world!`, and
the
  ordered two-image result preserves both image order and text;
- ACLGraph mode passes torch compilation, PIECEWISE mixed prefill/decode
  capture, FULL decode capture, and runtime graph replay on all four DP
  leaders;
- greedy text and single-image outputs are identical between eager and
graph
  modes;
- invalid image data returns an explicit HTTP 400 response.
- DSpark eager and FULL_DECODE_ONLY target-graph serving pass text,
single-image,
two-image, and 149-token long-decode cases with byte-identical outputs;
- eager and graph runs each accepted 153 of 180 draft tokens (85.0%);
four
concurrent graph-mode long decodes all passed, reaching 656/750
cumulative
  accepted/draft tokens (87.47%).
- three 149-token requests pinned concurrently to one DP rank also pass
byte-identically inside the captured 24-token graph range, with 375/414
  accepted/draft tokens (90.58%).

### OCRBench V1 accuracy

A complete 1,000-sample OCRBench V1 evaluation completed successfully
with no request errors and scored **826/1000 (82.6)**.

| Category | Score | Maximum |
| --- | ---: | ---: |
| Regular Text Recognition | 47 | 50 |
| Irregular Text Recognition | 44 | 50 |
| Artistic Text Recognition | 47 | 50 |
| Handwriting Recognition | 31 | 50 |
| Digit String Recognition | 35 | 50 |
| Non-Semantic Text Recognition | 46 | 50 |
| Scene Text-centric VQA | 178 | 200 |
| Doc-oriented VQA | 176 | 200 |
| Key Information Extraction | **171** | 200 |
| Handwritten Mathematical Expression Recognition | **51** | 100 |
| **Total** | **826** | **1000** |

Evaluation configuration: concurrency 64, temperature 1.0, top_p 0.95,
max_tokens 8192, seed 7, thinking enabled, reasoning_effort=high.
For TP4 graph mode, capture batch sizes must be multiples of TP size.
The
validated graph launch therefore uses `--max-num-seqs 4`;
`--max-num-seqs 1`
is valid for eager smoke but not for this graph topology.

### User-facing change

The model architecture can now be loaded and served on Ascend using the
vLLM-compatible multimodal API. The validated W8A8 launch topology is
TP4 /
DP4 / EP16 in eager or ACLGraph mode.

This PR is ready for implementation review. Marking it Ready for Review
does
not yet claim full production qualification or inclusion in the
supported
model matrix.

### Remaining release validation

- processor, ViT/aligner, router, hidden-state, and logits numerical
comparison
  against the upstream NVIDIA/HF reference;
- full BF16 checkpoint validation and BF16/W8A8 accuracy comparison;
- maximum 384-token image spans, non-aligned boundaries, heterogeneous
images,
  mixed batches, and longer-context cases;
- deployment alternatives such as TP8 / DP2;
- fixed image-task accuracy suite and text-only regression suite;
- startup, HBM, TTFT, TPOT, throughput, and regression measurements;
- supported-model documentation and a public launch example.
- automatic prefix caching passes with the DeepSeek-V4-specific
contract:
launch with `--block-size 32` and use prompts longer than 4K tokens.
Three
identical 5,014-token requests pinned to one DP rank reused 8,192 of
15,042
prompt tokens; DSpark remained active with 15/18 accepted/draft tokens.
- the multimodal APC stack also passes: three identical 6,139-token
long-prefix plus image requests all returned exact OCR text, reused
8,192
prompt tokens, recorded two multimodal-processor cache hits, and kept
DSpark
  active at 15/18 accepted/draft tokens.
- multimodal SSE streaming on the graph service reconstructs exact OCR
text,
reports `finish_reason=stop`, terminates with `[DONE]`, and exercises
DSpark
  with 5/6 accepted/draft tokens.

First-phase exclusions remain DSA context parallel and FlashComm1.
DSpark is
validated only as target-side multimodal prefill followed by text-only
speculative decode; the drafter does not consume image embeddings or
propose
inside image spans.

Signed-off-by: GDzhu01 <116337067+GDzhu01@users.noreply.github.com>


- vLLM main:
vllm-project/vllm@ba07e4a

Signed-off-by: GDzhu01 <116337067+GDzhu01@users.noreply.github.com>
Co-authored-by: MengLong Chen <71744434+dragondream-chen@users.noreply.github.com>
Co-authored-by: RenYuKai <184603735+pgzddxx@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build deepseek Related to DeepSeek models documentation Improvements or additions to documentation DSv4 intel-gpu Related to Intel GPU multi-modality Related to multi-modality (#4194) nvidia ready ONLY add when PR is ready to merge/full CI is needed

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

8 participants