[amy-why-3459]

Contributor intent (required human answers)

1. What outcome do you want to achieve?

Lower MiniCPM-o 4.5 Code2Wav CFM time on Ascend by landing the local NPU DiT fused body.

2. What is the highest-leverage variable for that outcome?

DiT body time inside Whole-Euler graph replay. At the serving chunk shape the body is kernel-launch bound.

3. How will this work move that variable?

Run the shared fused DiT schedule on NPU with packed QKV, ACLNN AddLayerNorm, and in-place KV writes.

4. Whose customer or user needs does this address, and what are those needs?

People serving MiniCPM-o 4.5 on Ascend, who need faster speech from Code2Wav.

Purpose

Enable the CFM fused DiT body on NPU. CUDA keeps tiled TF32 attention. NPU uses the same block schedule with packed QKV, in-place KV, and npu_add_layer_norm. cfm_fused_body defaults on for NPU and is set in minicpmo_4_5.yaml. Slot pool stays CUDA-only. An explicit cfm_fused_body: false keeps the ragged body.

Uncaptured shapes stay eager. Eager fused-body forwards were slower than the ragged body in the measurement below, so this helps the graph-replay path that serving uses (enable_cfm_graph: true).

Test Plan

• tests/model_executor/models/minicpmo_4_5/test_npu_dit_fused_body.py
• tests/model_executor/models/minicpmo_4_5/test_code2wav_batching.py::test_cfm_fused_body_defaults_on_for_npu
• tests/platforms/npu/test_minicpmo_attention_in_place.py

vLLM Version: 0.31.0

vLLM-Omni Commit: 9637dd458 vs parent a1c334877

Test Result

DiT body microbenchmark on one idle Ascend 910B1. Real structure: 16 layers, hidden 512, 8 heads. Chunk is 50 mel frames (25 codec tokens). Row count is twice the request count because of classifier-free guidance. Eager is 20 timed calls and graph replay is 30, both device-synchronized. The baseline is parent a1c334877. The fused path is this commit. Each audio chunk runs 10 Euler steps, so multiply the table by 10 for one chunk of CFM body time.

Requests	KV cache	Baseline graph	Fused graph	Change
1	0	15.73 ms	10.25 ms	-35%
1	200	15.22 ms	9.87 ms	-35%
1	512	15.98 ms	10.20 ms	-36%
4	200	18.34 ms	12.84 ms	-30%
8	200	21.42 ms	15.12 ms	-29%
16	100	25.17 ms	18.72 ms	-26%

Eager fused forwards were slower (about 37.5 ms to 44.9 ms at 1 request). The body-only table above is not the serving result. Seed-TTS end-to-end RTF is in the next section.

On a smaller non-zero estimator, AddLayerNorm max abs error was 9.5e-7 and the full body max abs error was 4.8e-7. The CPU schedule-alignment tests passed. The real CosyVoice final projection is zero-initialized, so the returned mel of that microbenchmark is zeros and was not used as a numeric check.

Seed-TTS end-to-end

Same chat-completions path on one Ascend 910B3 (NPU 0), vLLM 0.31.0, MiniCPM-o 4.5, seed-tts-eval en, warmup 1. Concurrency 1 / 8 / 16 with 4 / 32 / 64 prompts. Failed requests: 0. Lower audio RTF is better.

The unchanged baseline is parent 9bbfb61d50 (server log fused_body=False). Refactor is a5cf9e609 with cfm_fused_body: true in minicpmo_4_5.yaml (server log fused_body=True).

Concurrency	Prompts	Baseline mean RTF	Refactor mean RTF	Change
1	4	0.529	0.477	-9.9%
8	32	2.223	1.402	-36.9%
16	64	4.207	2.510	-40.3%

Median RTF moves from 0.521 / 2.216 / 4.167 to 0.470 / 1.406 / 2.474.

Made with Cursor

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/ar_runtime.md, docs/design/module/model_integration.md.

Module owners: @gcanlin @fake0fan @Gaohan123

Routing: @gcanlin via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files; @Gaohan123 via module of the changed files

@amy-why-3459, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot: superseded

The CI failure noted on 725e4fc273d0 refers to an earlier head; the pull request now points at 4284b7ff4aa8.

[hsliuustc0106]

@amy-why-3459, for 4284b7ff:

• Update the description to match the NPU opt-in default in this revision; it still says the fused body defaults on.
• Link the benchmark driver/results and exact tested revisions for the current NPU path, including a real-weight output comparison. The table names 9637dd458, while the reported numeric check uses a smaller estimator and the real-structure benchmark returns zeros.

[Dong1017]

Validated current head 5850d6970f975a90dbf7f796ea33bcfecdcf4bca on Ascend 910B1. test_fused_body_npugraph_matches_its_eager_path_with_inference_weights reported a pass and reached NPU graph capture/replay before the runner aborted during teardown with corrupted size vs. prev_size.

A full MiniCPM-o 4.5 duplex launch also cannot initialize in this server image because the model-registry subprocess aborts while the source reports a vLLM-Omni 0.1.dev / vLLM 0.31 major-minor mismatch. I cannot attribute either failure to this PR, so I have no code-level finding.

[psv666]

Reproduced the fused-body claim at real shapes on 8×Ascend 910B1, single card (torch_npu 2.10.0.post4, CANN 9.1.0, vllm-omni @ 9a7b439). I mirrored test_whole_euler_npugraph_matches_eager_and_owns_outputs but with the checkpoint config (DiT(depth=16, hidden=512, num_heads=8, head_dim=64, in=320, out=80, mlp_ratio=4)), the serving chunk shape (25 codec frames × token_mel_ratio 2 = 50 mel frames), n_timesteps=10, batch 1:

config	per-solve time
eager, unfused body	543.9 ms
eager, fused body	533.0 ms (1.02x)
Whole-Euler graph, unfused	143.6 ms
Whole-Euler graph, fused	101.1 ms (1.42x)

Graph replay was active in both graph arms (captures: 1, hits: 13), and eager fused vs unfused outputs were bit-identical (max abs diff 0.0) at this shape. This matches the stated mechanism: the fused body only pays off once kernel launches are batched inside the Whole-Euler replay.

One discrepancy between the description and the diff: the body says "cfm_fused_body defaults on for NPU", but the final diff leaves NPU opt-in — minicpmo_4_5.yaml only rewrites the comment, and that comment now explicitly says the NPU fused body remains opt-in until full-checkpoint audio quality is validated. Either choice is defensible, but the description should match the shipped default (or link the follow-up that flips it).

[amy-why-3459]

@hsliuustc0106 Could you please help review and merge this PR? The Seed-TTS end-to-end results in the description show mean audio RTF improving from 2.223 to 1.402 at concurrency 8 and from 4.207 to 2.510 at concurrency 16: approximately 1.59x and 1.68x in inverse-RTF speed (over 50% improvement), with zero failed requests. These correspond to RTF reductions of 36.9% and 40.3%. Please help move this forward once the remaining review requirements and CI checks are satisfied. Thanks!

[vllm-omni-review-bot]

Omni ReviewBot: superseded

The CI failure noted on a0d3ce28c568 refers to an earlier head; the pull request now points at fa284258378c.

[vllm-omni-review-bot]

Omni ReviewBot: superseded

The CI failure noted on 5cdf9421792b refers to an earlier head; the pull request now points at 4bf7ca910624.