Repository navigation
[Model] Complete MiniCPM-o 4.5 FP16 fused CFM dispatch - #9231
natureofnature wants to merge 3 commits into
Conversation
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
natureofnature
changed the title
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
This PR appears to belong to: docs/design/module/ar_runtime.md, docs/design/module/model_integration.md. Module owners: @tzhouam @fake0fan @Gaohan123 Routing: @tzhouam via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files; @Gaohan123 via module of the changed files @natureofnature, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer. Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment. |
amy-why-3459
left a comment
amy-why-3459
left a comment
There was a problem hiding this comment.
Thanks, the dispatch change is small and the FP16 path through cfm_attention already existed in the kernel (FP16 dots with FP32 accumulation), so routing matching FP16 Q/K/V there looks right to me. The test that asserts the tiled kernel is actually hit under FP16 is a good addition.
One gap on the second half of the change (the TF32-disabled default), see inline.
| self._tf32_mode != "off" | ||
| or read_bool_flag(extra, "token2wav_float16", False) | ||
| or read_bool_flag(extra, "cfm_float16", False) |
There was a problem hiding this comment.
This flips the default to the fused body for token2wav_allow_tf32: false / MINICPMO_CODE2WAV_TF32=off + FP16, but per the PR description the serving runs all inherit token2wav_allow_tf32: true, and the new tests only assert the fused_body flag in the config dict. So the newly enabled combination is never exercised end to end.
That matters mostly for cfm_float16: true alone: there the estimator weights stay FP32 and FP16 comes from autocast (_cfm_only_float16), so whether BatchedToken2Wav really ends up with _ragged_fused_body=True, and whether any op in the fused body that isn't autocast-cast (modulation table, torch._addmm_activation in _mlp, conv taps) now runs as IEEE FP32 without TF32, isn't covered. Could you add one short check with TF32 off + cfm_float16: true (and one with token2wav_float16: true), e.g. extending test_fp16_fused_body_matches_fp16_ragged_body or a quick C16 run, confirming no "cfm_fused_body requested but unsupported here" warning and output parity with the original body?
Omni ReviewBot triage noteAutomated triage of commit
These are automated triage suggestions only — the final decision belongs to the maintainers. |
Omni ReviewBot: three questions on the performance claim@natureofnature this PR reads as a performance or value claim:
Before the full evidence checklist, three short questions:
When you answer, the evidence that settles it is: base and head SHA, hardware, model, workload, warm-up and repeat count, mean or percentiles with their spread, and a correctness/quality-equivalence signal; an end-to-end claim also needs stage attribution. |
Signed-off-by: natureofnature <wzliu@connect.hku.hk>
amy-why-3459
left a comment
amy-why-3459
left a comment
There was a problem hiding this comment.
Thanks for adding test_tf32_disabled_fp16_default_initializes_fused_body, this covers what I asked about earlier: with TF32 off via either token2wav_allow_tf32: false or MINICPMO_CODE2WAV_TF32=off, both cfm_float16 and token2wav_float16 now go through the real MiniCPMO45Code2Wav / BatchedToken2Wav init, assert the fused body is selected without the unsupported warning, check the FP16 estimator / encoder dtypes, and compare fused vs. ragged outputs and caches under the real _cfm_autocast. The fused_calls check plus the TF32 asserts inside the wrapper make sure the fused path actually ran with TF32 disabled. No further concerns from me on the delta since d4b7e72.
Contributor intent
1. What outcome do you want to achieve?
Improve MiniCPM-o 4.5 serving throughput, time to first audio, and RTF while preserving output quality.
2. What is the highest-leverage variable for that outcome?
FP16 CFM execution cost, including fused-body selection and attention execution.
3. How will this work move that variable?
Keep FP16 CFM fusion enabled independently of TF32, and route its non-pooled attention through the existing FP16-capable tiled kernel. Measure the resulting serving throughput, latency and audio quality against post-#9218 main at the same precision.
4. Whose customer or user needs does this address, and what are those needs?
MiniCPM-o 4.5 serving users seeking lower latency and RTF, and higher throughput under concurrent requests.
Purpose
Follow-up to #9218. Two FP16 dispatch gaps remain:
token2wav_allow_tf32: falseorMINICPMO_CODE2WAV_TF32=off, CUDA disables the default fused CFM body even whentoken2wav_float16: trueorcfm_float16: trueselects FP16. The default now accounts for either FP16 flag; an explicitcfm_fused_bodysetting still takes precedence.cfm_attentionalready supports FP16. The existing SM80+ dispatch now accepts matching FP16 Q/K/V as well. No new attention backend or dependency is introduced.The changes are confined to dispatch, regression coverage, and
docs/design/minicpm_o45_mrv2_performance.md. FP32 policy, existing platform fallbacks, CFG and Euler step count are unchanged.Test Result
Base:
0da1d831e07fadccc3ab51ef27b7c21623b693c2(post-#9218 main). Original sweep:d4b7e7260d2583fe21f41336b30063b2ebf07927. Additional validation: current head3b866f6a0cbc0291eac237ae369fafae29dae8e6, which adds tests and leaves the runtime unchanged.Environment: one NVIDIA H800 80 GB; vLLM
0.32.0; PyTorch2.13.0+cu130; CUDA 13; Triton3.7.1; FlashInfer0.7.0.post1installed. MPS was enabled in every arm, with all three stage workers confirmed as clients.Additional validation at the current head
C16 throughput and RTF gains reproduced across three fresh server starts. Quality noninferiority is not established for the PR: the C16 and C32 WER upper bounds exceed the predeclared margin. SIM and corrected UTMOS meet their margins at all three tested concurrencies. The WER confidence intervals include zero, so this does not establish a quality regression either.
C16 performance repeat
Same H800, MPS, official client, 64 timing inputs and deployment settings as the original sweep below. Each arm has three fresh server starts, each with 32 warmup requests, one discarded 64-request run, and four measured 64-request runs. All 2,304 included timed requests succeeded. Values are arithmetic means of 12 official per-run metrics per arm, including the per-run TTFA percentiles.
Compared with main FP16, PR FP16 improves throughput 12.6% and reduces mean RTF 11.0%. TTFA p50 increases 5.2%, while p95 decreases 4.9%. The original sweep's 19.0% C16 p50 increase is smaller in this repeat, but the latency tradeoff remains. Across individual fresh starts, throughput is 28.930–30.524 audio-s/s for main FP16 and 33.570–33.878 for PR FP16.
The same-precision main FP16 versus PR FP16 comparison measures this PR's dispatch change. The FP32 control uses the existing Triton attention with TF32 allowed; main FP16 switches to SDPA, while PR FP16 uses Triton. The FP32-to-FP16 comparisons therefore describe the respective complete precision modes.
C16 means by fresh server start
Held-out quality validation
256 additional English Seed-TTS prompts × C1/C16/C32 × three fresh server starts × three arms = 6,912 generated WAVs. These IDs exclude all 128 inputs previously used for evaluation or warmup. All included requests and WER/SIM/UTMOS scoring calls completed; all saved WAVs are nonempty and finite. The same paired prompt IDs are used throughout. WER is the official mean per-request WER, and SIM is the official WavLM embedding-cosine proxy.
Average the three repeats within each prompt, then bootstrap the 256 paired prompts with 20,000 draws, seed
20261022. The predeclared one-sided 95% upper-bound margins are WER +0.20 percentage points, SIM drop 0.005, and UTMOS drop 0.05. All nine bounds must meet their margins for a comparison to pass. These are engineering screens, not perceptual-equivalence thresholds.Each cell is mean degradation / one-sided 95% upper bound. Positive degradation means higher WER or lower SIM/UTMOS.
Main FP16 versus main FP32 meets all nine bounds. PR FP16 versus main FP16 does not: the WER upper bounds are +0.935 pp at C16 and +0.327 pp at C32. Their two-sided 95% intervals are [−0.435, +1.110] pp and [−0.223, +0.382] pp, respectively. PR FP16 versus main FP32 also remains inconclusive at C16, whose WER upper bound is +0.305 pp. No quality-equivalence or quality-improvement claim is made.
The largest C16 WER contribution comes from the three-word prompt “Not a thing.”, where all three arms receive large WER values and the PR scores higher than main FP16 in two repeats. That prompt is retained in every mean and confidence interval.
UTMOS correction: the pinned
balacoon/utmosJIT internally casts its input to float32 and divides by 32768, consistent with its documented int16 input. The official evaluator passes already-normalized floats, applying the amplitude scaling twice. The unmodified official JSONs are retained, but their UTMOS values are not used for acceptance. All 6,912 saved WAVs were rescored offline with the same official 16 kHz resampler and pinned JIT, passing normalized float audio multiplied by 32768. A probe on four separate smoke-test WAVs agreed with documented int16 input within 0.0035 MOS. WER, SIM, generated audio and performance are unchanged. The thresholds were fixed before generation; this scorer correction is the only scoring-protocol amendment.Real-asset serving with TF32 disabled
No explicit
cfm_fused_bodyoverride is present in the deployment inheritance chain. Each case uses actual model assets and 16 official C4 requests with audio saving and quality-scoring execution:token2wav_allow_tf32: falsetoken2wav_allow_tf32: falseMINICPMO_CODE2WAV_TF32=offThese 48/48 serving checks validate actual default selection and successful audio generation. The full-Flow case is a serving smoke test; the 256-prompt quality campaign above uses CFM-only FP16.
One container shutdown interrupted an earlier FP32 start. Its partial results are retained separately, and the entire start was repeated. No completed prompt or run was removed based on its result.
Original official serving benchmark
Unmodified
vllm bench serve --omni, MRv2 turn serving, 64 fixed English Seed-TTS requests with per-request reference audio. All three arms use CFM-only FP16,cfm_float16: trueandtoken2wav_float16: false; encoder, speaker extraction and HiFT retain their existing precision. Thinker/Talker use BF16. All use 10 Euler steps, stage sequence limits 16/8/8, CFM microbatch 4 and graph budget 64.The CFM attention paths compared are main: SDPA, PR: the existing Triton kernel, and FlashInfer: an experimental FA3 paged-prefill prototype using FlashInfer
0.7.0.post1, based on50027779f66463f1765330bc7cb67659b1bffeb7. The FlashInfer implementation is not included in this PR. The reproduction commands below apply to main and PR; running the FlashInfer arm requires that local prototype.Each concurrency has four measured runs per arm: two runs from each of two fresh server starts. Each server first receives 32 disjoint warmup requests; each tested concurrency gets a discarded 64-request run before the two measured runs. Cells below are arithmetic means of the official per-run metrics, including the per-run TTFA percentiles. Quality evaluation runs separately after timing.
SVG version
Performance panels retain the original C1/C4/C8/C16/C32 sweep. Quality panels use the new 256-prompt C1/C16/C32 validation above, with three fresh server starts per arm, and compare main FP32, main FP16 and PR FP16. Shading shows the range of run means, not confidence intervals. FlashInfer was not rerun on this quality cohort. The WER panel and table reflect that PR-versus-main FP16 noninferiority is not established at C16/C32; SIM and corrected UTMOS meet their margins.
Relative to main, the PR improves C16 throughput 14.2% and reduces mean RTF 12.1%; at C32 the changes are +8.6% and −8.1%, respectively. TTFA is a tradeoff: C8 p50 increases 6.4%, and C16 p50 increases 19.0% (637.74 → 759.01 ms), while p95 improves at both. This is not a uniform latency improvement.
FlashInfer has higher measured throughput than Triton at C4/C8, lower throughput at C16, and nearly identical throughput at C32. Its C16 TTFA p50 is lower than Triton's (653.38 versus 759.01 ms). These are workload-specific tradeoffs; the FlashInfer results are retained for comparison without adding its implementation to this PR.
All 3,840 timed requests across the three arms succeeded. At C8, generated duration averages 242.08 s main / 232.74 s Triton / 231.38 s FlashInfer. The PR's request throughput improves 9.7% over main, compared with 5.6% for audio throughput. All runs are retained. This is a small fixed workload, not a production traffic estimate.
Attention cost attribution
An isolated CFM attention microbenchmark on the same H800 measures the cost removed by the dispatch change. Both calls use FP16, the existing non-pooled KV arena layout, 8 heads × 64 dimensions, and per-row valid-key masks; CFG uses twice the request batch. These are single-attention timings, not whole-stage timings or a measurement of CFM's share of end-to-end runtime.
Timing uses five eager warmups, 16 calls captured per CUDA Graph, one graph warmup, then five CUDA-event measurements of 32 replays each; the table reports the median per-call time. Both implementations pass comparison with FP32 math SDPA at
rtol=atol=2e-3.In the serving A/B, both deployments explicitly select
cfm_fused_body: trueand allow TF32, so the separate TF32-disabled default fix does not change either measured execution path. The same-precision serving comparison therefore isolates the attention dispatch change; no additional performance gain is attributed to the default-selection fix. The throughput and RTF gains, and the C8/C16 TTFA p50 regressions, are reported above.Initial 64-prompt quality screen
Separate official Whisper large-v3 WER and WavLM embedding-cosine SIM evaluation at C16 only, 64 requests per fresh server. WER is the mean per-request error rate. SIM is the benchmark's WavLM proxy, not UniSpeech SIM-o. Main and FlashInfer each have three quality runs; the PR has two. These initial results are retained in the table below; the figure now shows the separate 256-prompt validation above.
Mean WER across starts is 5.411% main / 5.334% Triton / 5.762% FlashInfer; mean SIM is 0.858971 / 0.859507 / 0.859274, respectively. Main itself varies from 4.382% to 6.723% WER. These results do not establish a quality improvement or statistical equivalence. All 512 saved WAVs were nonempty and finite, with zero request, ASR or SIM failures. UTMOS, listening evaluation, V1/duplex E2E and NPU E2E were not evaluated in this initial screen.
Regression tests
3b866f6a0cbc0291eac237ae369fafae29dae8e6on H800 for the three files below. GPU tests call the real tiled kernel and compare fused FP16 output/cache tensors with the original ragged body atrtol=atol=2e-3.cfm_float16andtoken2wav_float16with TF32 disabled through either the stage configuration orMINICPMO_CODE2WAV_TF32=off. They run the real Code2Wav configuration and backend initialization, verify that fusion is enabled without the unsupported warning, and compare DiT outputs and both caches against the original body atrtol=atol=2e-3, with CUDA matmul and cuDNN TF32 both disabled. Only model-asset loading is substituted;_ragged_fused_bodyis not forced on. 4 passed on the PR; 4 failed on an isolated base checkout because fusion was not enabled.estimator.half(); it does not leave the estimator weights FP32 and rely solely on autocast. Encoder parameters remain FP32 forcfm_float16and become FP16 fortoken2wav_float16. These are initialization/DiT checks. Performance measurements use TF32 enabled; the additional real-asset serving checks above cover TF32 disabled.git diff --checkpassed.Test Plan
Reproduction: pinned inputs and deployment
Set the checkout paths to the base and PR revisions above. Both tested checkouts have the same one-line vLLM 0.32.0 API adaptation in
minicpmo_4_5_omni_llm.py:self.llm.get_input_embeddings(input_ids)→self.llm.embed_input_ids(input_ids). It is outside this PR. All 33 official benchmark Python files are identical across arms.Download the pinned models and dataset subset before enabling offline mode:
Start one server at a time. Set
SERVER_CHECKOUTto the main or PR checkout; the same overlay and precision apply to both.The overlay inherits
cfm_fused_body: trueandtoken2wav_allow_tf32: truefrom the MRv2 configs. Consequently, these performance results measure the attention dispatch change; the TF32-disabled default fix is covered by regression tests and the real-asset serving checks above. Stop each server and wait for its workers and MPS instance to exit before starting the next.Official client commands and quality evaluation
Run in a second terminal with the same model/input paths, after
/healthreturns 200. Use the main checkout for the client in every arm. The function only invokes the official CLI; it does not implement a separate benchmark client or metric calculation.Use a fresh
RESULT_DIRfor every server start. The measured main starts usedCONCURRENCIES="1 16","1 4 8 16 32", and"4 8 32", respectively. Both PR starts used all five concurrencies. Thus every main/PR concurrency has two server starts and four timed runs; quality was evaluated once at C16 after each start's timing runs. The three FlashInfer starts used the same concurrency lists as the three main starts, with one separate C16 quality run after each.Run quality evaluation using the same
benchfunction, outside the timed loop:Additional validation: fixed inputs, commands and scoring
Use the same pinned model, Whisper, WavLM, deployment overlay, request body and
benchfunction above. Use head3b866f6a0cbc0291eac237ae369fafae29dae8e6for the PR. Each arm gets three fresh server starts and a separate result directory per start. Start order: main FP16, PR FP16, main FP32, PR FP16, main FP32, main FP16, main FP32, main FP16, PR FP16. Setcfm_float16: falsefor main FP32 andtruefor both FP16 arms; keeptoken2wav_float16: falseand TF32 allowed in all three. Main FP32 uses the existing Triton attention, main FP16 uses SDPA, and PR FP16 uses Triton.Download the 256 additional inputs and pinned UTMOS checkpoint before enabling offline mode:
For each fresh server, run the following with a unique
RESULT_DIR. Quality scoring is disabled throughout timing:Retain the original official JSONs. After serving finishes, correct UTMOS on those same saved WAVs using the official resampler and pinned JIT loader:
Statistical postprocessing is separate from the official client. For each concurrency and metric, align the 256 IDs, average the three repeats within each ID, then subtract reference from candidate. Multiply WER differences by 100 to obtain percentage points; negate SIM/UTMOS differences so positive always means degradation. Draw 20,000 bootstrap samples of 256 paired prompt indices with
numpy.random.default_rng(20261022). The 95th percentile of the bootstrap mean is the one-sided 95% upper bound. Thresholds were fixed before generation: WER +0.20 percentage points, SIM drop 0.005, UTMOS drop 0.05. All nine bounds must meet their margins for a comparison to pass. No prompts are removed.For the TF32-disabled serving checks, remove
cfm_fused_bodyfrom the MRv2 YAML inheritance chain so initialization selects its default. Test main and PR withcfm_float16: true,token2wav_float16: false,token2wav_allow_tf32: false; separately test the PR withtoken2wav_float16: true,cfm_float16: false, andMINICPMO_CODE2WAV_TF32=off. Each gets a fresh server andbench evaluation 16 4 preflight 60with audio saving and quality evaluation enabled. These 16-request checks establish successful serving and actual default selection, not quality noninferiority for full-Flow FP16.