[Feature][Model] Support DeepSeek V4 Flash Vision on Ascend

[GDzhu01]

What this PR does / why we need it

Adds DeepSeek-V4-Flash-Vision-Exp support to vLLM Ascend main, paired with
the vLLM v0.27.x API line.

Roadmap and remaining validation tracking: #15462

Upstream behavior reference: vllm-project/vllm#54566

The paired vLLM release does not contain the DeepSeek-V4 vision processor and
vision tower from the upstream PR. This change therefore carries the required
v0.27-compatible implementation in vLLM Ascend without importing unmerged
vLLM modules.

Implementation

• registers DeepseekV4ForConditionalGeneration and routes vision
  checkpoints to an Ascend multimodal wrapper;
• provides the image processor, ViT, aligner, sentinel embeddings, placeholder
  expansion, and multimodal weight mapping required by vLLM v0.27;
• preserves raw input token IDs through prefill for DeepSeek-V4 MoE routing;
• implements hybrid bias_vl routing: text tokens retain hash/correction-bias
  routing while image sentinel tokens use dynamic routing with bias_vl;
• builds paged original-KV sparse indices that make each complete image span
  bidirectionally visible while keeping causal SWA outside image spans;
• passes original-KV image indices and compressed-KV top-k indices to the
  Ascend sparse-attention operator independently and simultaneously;
• supports ModelSlim W8A8 checkpoints whose language quantization description
  remains in the causal-LM model.* / lm_head.* namespace;
• keeps config-convertor registration safe in spawned DP engine processes.
• enables DeepSeek-V4 Vision with a text-only DSpark drafter: image inputs are
  encoded by the target during prefill, while DSpark proposes subsequent text
  decode tokens; restores the draft architecture after VL config conversion,
  exposes the EAGLE3 hidden-state interface, and ignores draft-only vision
  router bias tensors without changing text routing.

Runtime validation completed

Validated in a 16-NPU Ascend container with:

• vLLM: 6e448d0ea9bf3d88d898b65449ca6dc2aec170ac (v0.27.1 API line)
• checkpoint: DeepSeek-V4-Flash-Vision-Exp W8A8, with all BF16 bias_vl
  tensors included in the checkpoint overlay
• release topology: TP4 / DP4 / EP16, block size 32

Results:

• targeted processor, ModelSlim, MoE router, and DSA tests pass (90 tests in
  the combined suite; 59 tests in the focused follow-up suite);
• the Ascend sparse-attention operator accepts widened original indices and
  simultaneous original/compressed indices with finite BF16 output;
• all 79 checkpoint shards load successfully with the multimodal wrapper;
• eager OpenAI-compatible serving passes text-only, single-image, ordered
  two-image, deterministic repeated-image, and invalid-image cases;
• the single-image OCR result correctly reads Hello, AI world!, and the
  ordered two-image result preserves both image order and text;
• ACLGraph mode passes torch compilation, PIECEWISE mixed prefill/decode
  capture, FULL decode capture, and runtime graph replay on all four DP
  leaders;
• greedy text and single-image outputs are identical between eager and graph
  modes;
• invalid image data returns an explicit HTTP 400 response.
• DSpark eager and FULL_DECODE_ONLY target-graph serving pass text, single-image,
  two-image, and 149-token long-decode cases with byte-identical outputs;
• eager and graph runs each accepted 153 of 180 draft tokens (85.0%); four
  concurrent graph-mode long decodes all passed, reaching 656/750 cumulative
  accepted/draft tokens (87.47%).
• three 149-token requests pinned concurrently to one DP rank also pass
  byte-identically inside the captured 24-token graph range, with 375/414
  accepted/draft tokens (90.58%).

OCRBench V1 accuracy

A complete 1,000-sample OCRBench V1 evaluation completed successfully with no request errors and scored 826/1000 (82.6).

Category	Score	Maximum
Regular Text Recognition	47	50
Irregular Text Recognition	44	50
Artistic Text Recognition	47	50
Handwriting Recognition	31	50
Digit String Recognition	35	50
Non-Semantic Text Recognition	46	50
Scene Text-centric VQA	178	200
Doc-oriented VQA	176	200
Key Information Extraction	171	200
Handwritten Mathematical Expression Recognition	51	100
Total	826	1000

Evaluation configuration: concurrency 64, temperature 1.0, top_p 0.95, max_tokens 8192, seed 7, thinking enabled, reasoning_effort=high.
For TP4 graph mode, capture batch sizes must be multiples of TP size. The
validated graph launch therefore uses --max-num-seqs 4; --max-num-seqs 1
is valid for eager smoke but not for this graph topology.

User-facing change

The model architecture can now be loaded and served on Ascend using the
vLLM-compatible multimodal API. The validated W8A8 launch topology is TP4 /
DP4 / EP16 in eager or ACLGraph mode.

This PR is ready for implementation review. Marking it Ready for Review does
not yet claim full production qualification or inclusion in the supported
model matrix.

Remaining release validation

• processor, ViT/aligner, router, hidden-state, and logits numerical comparison
  against the upstream NVIDIA/HF reference;
• full BF16 checkpoint validation and BF16/W8A8 accuracy comparison;
• maximum 384-token image spans, non-aligned boundaries, heterogeneous images,
  mixed batches, and longer-context cases;
• deployment alternatives such as TP8 / DP2;
• fixed image-task accuracy suite and text-only regression suite;
• startup, HBM, TTFT, TPOT, throughput, and regression measurements;
• supported-model documentation and a public launch example.
• automatic prefix caching passes with the DeepSeek-V4-specific contract:
  launch with --block-size 32 and use prompts longer than 4K tokens. Three
  identical 5,014-token requests pinned to one DP rank reused 8,192 of 15,042
  prompt tokens; DSpark remained active with 15/18 accepted/draft tokens.
• the multimodal APC stack also passes: three identical 6,139-token
  long-prefix plus image requests all returned exact OCR text, reused 8,192
  prompt tokens, recorded two multimodal-processor cache hits, and kept DSpark
  active at 15/18 accepted/draft tokens.
• multimodal SSE streaming on the graph service reconstructs exact OCR text,
  reports finish_reason=stop, terminates with [DONE], and exercises DSpark
  with 5/6 accepted/draft tokens.

First-phase exclusions remain DSA context parallel and FlashComm1. DSpark is
validated only as target-side multimodal prefill followed by text-only
speculative decode; the drafter does not consume image embeddings or propose
inside image spans.

Signed-off-by: GDzhu01 116337067+GDzhu01@users.noreply.github.com

• vLLM main: vllm-project/vllm@ba07e4a

[github-actions]

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

• A PR should do only one thing, smaller PRs enable faster reviews.
• Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
• Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces preliminary support for the DeepSeek-V4 Flash Vision model on the Ascend backend. It establishes the necessary architectural wrappers, integrates vision-specific components like ViT and aligners, and implements custom MoE routing and attention indexing to handle multimodal inputs effectively. The changes are currently in a draft state, pending final dependency updates and the availability of quantized checkpoints for validation.

Highlights

• Model Architecture: Added the DeepseekV4ForConditionalGeneration wrapper to support multimodal DeepSeek-V4 models on the Ascend backend.
• Vision Integration: Integrated ViT and aligner components using shared vLLM interfaces to enable image processing.
• MoE Routing: Implemented bias_vl routing for hybrid MoE layers, allowing for dynamic expert selection on image sentinel tokens.
• Attention Optimization: Added paged sparse-SWA indices to ensure bidirectional visibility of image spans during prefill while maintaining causal SWA.
• Testing: Added comprehensive unit tests for MoE routing logic and bidirectional SWA index construction.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

Suggested PR Title:

[Attention][Ops][Feature] Support DeepSeek-V4-Flash-Vision-Exp on Ascend

Suggested PR Summary:

### What this PR does / why we need it?
This PR adds support for the DeepSeek-V4-Flash-Vision-Exp model on Ascend. It implements bidirectional sliding window attention (SWA) indices for vision prefill, introduces the `AscendDeepseekV4ForConditionalGeneration` model wrapper, and integrates the vision-only routing bias (`bias_vl`) into the DeepSeek-V4 MoE gate and router.

Feedback:
- In `tests/ut/models/test_deepseek_v4_moe.py`, the test assertion should verify `bias_vl` instead of `e_score_correction_bias` in the mocked `FusedMoEFactory` call.
- In `vllm_ascend/models/deepseek_v4/model.py`, the hardcoded magic number `129257` should be replaced with the imported `IMAGE_SENTINEL_BASE_ID` constant.

### Does this PR introduce _any_ user-facing change?
Yes, it registers and supports the `DeepseekV4ForConditionalGeneration` model class for conditional generation on Ascend.

### How was this patch tested?
The changes are covered by new unit tests in `tests/ut/attention/test_dsa_v1.py` and `tests/ut/models/test_deepseek_v4_moe.py`.

[gemini-code-assist]

The test assertion is incorrect. It checks if moe.gate.bias_vl is passed as the e_score_correction_bias argument, but it should be passed as the bias_vl argument to FusedMoEFactory. This means the test is not correctly verifying the behavior of the code.

Suggested change

	assert fused_moe.call_args.kwargs["e_score_correction_bias"] is moe.gate.bias_vl
	assert fused_moe.call_args.kwargs["bias_vl"] is moe.gate.bias_vl

[gemini-code-assist]

The magic number 129257 is hardcoded for image_sentinel_lo. This value represents IMAGE_SENTINEL_BASE_ID for the DeepSeek-V4 vision model. To improve code clarity and maintainability, it should be replaced with the named constant.

Please add the following import at the top of the file:

from vllm.models.deepseek_v4.common.mm_preprocess import IMAGE_SENTINEL_BASE_ID

And then use the constant in the FusedMoEFactory call.

Suggested change

	image_sentinel_lo=129257,
	image_sentinel_lo=IMAGE_SENTINEL_BASE_ID,

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[GDzhu01]

/cancel
[Bot]: successfully force-cancelled the following runs:

https://github.com/vllm-project/vllm-ascend/actions/runs/33831336723

[vllm-ascend-ci]

/rerun
[Bot]: rerun completed.

Rerun (failed jobs only):

• E2E