Repository navigation
Conversation
The NPU override block only covered stages 0 and 2, so stage 1 kept the base max_num_seqs of 4 and gated the top tier. Measured on 910B3 (r16 against the r14 baseline): c8 RTF -15.1%, c4 -3.5%, c1 -1.1%. Concurrency only; the numerical path is untouched. Signed-off-by: y-null <y-null@users.noreply.github.com>
Once the 32 captured shapes are exhausted, later shapes permanently degrade to eager. The CFM mel bucketing already in main (cfm_graph_bucket_frames: 16) keeps the captured shape count at 19, which is what makes a pool this size sustainable. Measured on 910B3 (r34 against r22): c4 RTF -17.0%, c8 -23.5%, c1 +5.8%. Signed-off-by: y-null <y-null@users.noreply.github.com>
The stage input processor learns a first-window size: the first payload uses initial_codec_chunk_frames while the steady-state window is unchanged, and a deployment that leaves the knob unset is bit-identical to the previous behaviour. It only takes effect when initial < codec_chunk_frames. Measured on 910B3 (r17 against r16): c1 RTF -7.4%; Seed-TTS WER 1.2179%, unchanged from the baseline. Signed-off-by: y-null <y-null@users.noreply.github.com>
Behind VLLM_OMNI_C2W_SINGLE_ROW (default off), the code2wav stage reads the switch and _iter_decode_batches gains a single-row bypass that skips wave splitting and the min-batch gate. The numerical path is unchanged (the even path is pre-existing code). Measured (A49/A49P against r34): c1 ~0, c4 -3~8%, c8 -4~7%. The item is not kept: the gain sits inside the run-to-run drift band, so a later commit in this branch removes the switch again. Signed-off-by: y-null <y-null@users.noreply.github.com>
Stage-level enable_prefix_caching on stage 0; the top level stays false for duplex safety. The other two parts of the same source item (the HiFT/CFM graph modes and the stage-2 eager override) are already in main and are not carried. Measured on the source line (same hardware class): c1 -22.0%, c4 -3.9%, c8 +1.1%. Signed-off-by: y-null <y-null@users.noreply.github.com>
Stage 0 gains async_scheduling: false plus a speculative_config (method ngram, 14 speculative tokens). async_scheduling has to be false: vLLM's VllmConfig validation rejects the async-scheduling and n-gram-proposer combination. Measured on the source line: RTF -11.8% / -3.7% / -5.9%, TTFT -69%. It is dropped again by the next commit: vLLM-omni refuses prefix caching and speculative decoding on the same stage, and prefix caching is the configuration this port validates. Signed-off-by: y-null <y-null@users.noreply.github.com>
…hain The Talker emits 8 codec frames per scheduler step instead of one. The stop chain that stalled inside the K loop is fixed end to end: * the outer model class forwards the four multi-frame entry points again (supports_multi_frame_decode, take/set_batch_stop_logits, merge_frame_outputs -- without them the multi-frame loop never engaged, and silently so); * the stop head stays the two-wide continue/stop row, and its vocabulary width is repaired for the rejection sampler; * the native rejection sampler is restored for the armed path; * the scheduler gains a K-frame guard. Without the stop fix a request ran to max_tokens (~143 s instead of ~2 s per request); with it the multi-frame decode is a net win (c1 RTF -19.1%, c8 -10.6% on the source line). Verified on 910B3: warm 3-tier 0.4434 / 1.0867 / 2.1644 with 0 ERROR, the multi-frame loop engaged on the first request, audio non-empty. Signed-off-by: y-null <y-null@users.noreply.github.com>
…hing; tidy comments - The deploy config enabled stage-0 prefix caching and n-gram speculative decoding together, which vLLM rejects at startup (OmniPrefixCacheUnmatchError). Keep prefix caching (the validated configuration) and remove the n-gram block. - Replace internal notes (item numbers, measurement references, Chinese prose) with concise upstream-style comments. Signed-off-by: y-null <y-null@users.noreply.github.com>
…ti-frame feedback chain Pins the frame-code feedback (request-state last_code), the decode preprocess priority, the scheduler K-frame guard and the multi-frame gate matrix. Pure CPU; no device required. Signed-off-by: y-null <y-null@users.noreply.github.com>
…ame decode is off The multi-frame work moved codec sampling into a device-side state machine and, in doing so, dropped the host-state contract the default single-frame path relies on: - make_omni_output no longer recorded state step/recent_codes/finished or the force/mask codec-EOS routing, so empty-speech segments, chunk budgets and the duplex turn-end drain window lost their effect; - compute_logits no longer forced or masked EOS, and sample() no longer restored an EOS blanked by vLLM's min_tokens processor; - the offline budget recorded min_tokens as 0 instead of None. Split make_omni_output into a single-frame path (multi-frame off, the default) that keeps the upstream contract, plus the multi-frame path, and restore the codec-EOS routing in compute_logits/sample. The 12 upstream tests in test_talker_batching.py pass again; test_pipeline expectations follow the new stage-1 concurrency cap and Code2Wav graph pool. Signed-off-by: y-null <y-null@users.noreply.github.com>
- ruff check: 21 findings (15 auto-fixed; manual: N814 import alias, F841 unused variable, E501 long line, 4x UP031 percent formatting) - ruff format: 12 files reformatted Signed-off-by: y-null <y-null@users.noreply.github.com>
codec_chunk_frames 25 -> 75 so initial_codec_chunk_frames: 25 takes effect; stage-1 stop_token_ids: [1] for the multi-frame stop head (nothing injects it for stage 1); stage0/stage2 max_num_seqs 8 so the top tier is not gated by a single stage; NPU cudagraph_mode PIECEWISE -> FULL_DECODE_ONLY, which keeps in-graph sampling and multi-frame expansion. Signed-off-by: y-null <y-null@users.noreply.github.com>
…the deploy YAML Make stage 1's speculative_config in the deploy YAML the single source of truth for the multi-frame decode and remove every code-side default and switch around it. * the deploy YAML carries stage 1's speculative_config (8 codec frames per step) and async_scheduling: false explicitly; no code-side injection or default remains. * the environment switches VLLM_OMNI_MINICPMO_TALKER_FRAMES, OMNI_K_STEP and VLLM_OMNI_C2W_SINGLE_ROW are gone, and the 910B SoC gate is dropped: the rejection-verifier limit it guarded against was the missing multi-frame wiring this branch fixes, and 910B3 is validated end to end. * the model-side and scheduler-side gates derive K from the engine config (model_stage == tts plus the n-gram fingerprint), so the two cannot disagree. * the stage-2 single-row even-batch item is reverted: its measured gain sat inside the run-to-run drift band and a default-off switch adds configuration without value. Signed-off-by: y-null <y-null@users.noreply.github.com>
…the deploy YAML Make stage 1's speculative_config in the deploy YAML the single source of truth for the multi-frame decode, and keep this port to the optimizations plus their tests only. * the deploy YAML carries stage 1's speculative_config (8 codec frames per step) and async_scheduling: false explicitly; no code-side default remains. * the environment switches VLLM_OMNI_MINICPMO_TALKER_FRAMES, OMNI_K_STEP, VLLM_OMNI_C2W_SINGLE_ROW and VLLM_OMNI_MINICPMO_NARROW_REPLAY are gone, and the 910B SoC gate is dropped: the rejection-verifier limit it guarded against was the missing multi-frame wiring this branch fixes, and 910B3 is validated end to end. * the model-side and scheduler-side gates derive K from the engine config (model_stage == tts plus the n-gram fingerprint), so the two cannot disagree. * the per-step/per-frame timers, the stop and bookkeeping traces, the guard view and utils/step_prof.py are removed: they were bring-up instrumentation, not optimizations. * the stage-2 single-row even-batch item is reverted: its measured gain sat inside the run-to-run drift band and a default-off switch adds configuration without value. * tests follow the new deploy values (chunk 75, FULL_DECODE_ONLY, the multi-frame stop id, all-stage concurrency 8). Signed-off-by: y-null <y-null@users.noreply.github.com>
`cfm_graph_bucket_frames` (deploy YAML: 16) only applied when the CUDA-only `CFMGraphWrapper` existed, so on NPU the bucket width stayed 0: the CFM capture key space never collapsed (64 shapes) and, once the graph pool filled, later shapes permanently degraded to eager. - batched_token2wav.py: track "graphs requested" separately (`_cfm_graph_enabled`, from the deploy YAML's `enable_cfm_graph`) and key bucketing/padding off "CUDA wrapper present OR graphs requested". The CUDA path is unchanged: the wrapper is built whenever the config is enabled. - platforms/npu/models/minicpmo_4_5_code2wav.py: stop routing calls that carry an attention mask to eager; only the ragged path (`valid_lengths`) stays eager. A bucketing pad mask has a shape fixed by the bucket, so it travels as a graph input (its presence is part of the capture key) like any other tensor. `attn_mask` is forwarded to the graphable body only when present, so graphable bodies written before this feature keep their old signature. Motivation: this is the memory prerequisite for the single-device two-replica profile; without it the pool fills and the second replica does not fit. Bucketing itself measures neutral for RTF -- it is a shape/memory lever. Signed-off-by: y-null <y-null@users.noreply.github.com>
…the deploy YAML Make stage 1's speculative_config in the deploy YAML the single source of truth for the multi-frame decode, and keep this port to the optimizations plus their tests only. * the deploy YAML carries stage 1's speculative_config (8 codec frames per step) and async_scheduling: false explicitly; no code-side default remains. * the environment switches VLLM_OMNI_MINICPMO_TALKER_FRAMES, OMNI_K_STEP, VLLM_OMNI_C2W_SINGLE_ROW and VLLM_OMNI_MINICPMO_NARROW_REPLAY are gone, and the 910B SoC gate is dropped. * the model-side and scheduler-side gates derive K from the engine config (model_stage == tts plus the n-gram fingerprint), so the two cannot disagree. * the per-step/per-frame timers, the stop and bookkeeping traces, the guard view and utils/step_prof.py are removed: they were bring-up instrumentation, not optimizations. * the stage-2 single-row even-batch item is reverted, along with the four orphan comment lines it left in code2wav. * the 910C/A3 Triton warmup guard (ascend_warmup_patch.py and its install sites) is dropped: it is a platform adaptation, not an optimization, and the 910B baseline ran the stock warmup either way. Its only remaining consumer, native_rejection_sampler, now carries the SoC probe and K-step check inline. * tests follow the new deploy values. Signed-off-by: y-null <y-null@users.noreply.github.com>
The bucketing change lets a pad mask travel as a graph input; add the regression test that pins the contract: a call carrying an attention mask still dispatches to the platform graph runner (only the ragged per-request-length path stays eager), the mask is one of the captured inputs, and its presence is part of the capture key (constants == (with_cache, True)). Also drop one stray blank line so the NPU code2wav file stays byte-identical with the validated branch. Signed-off-by: y-null <y-null@users.noreply.github.com>
Removing ascend_warmup_patch.py makes the service fail on 910C/A3 (confirmed by testing): it is a required platform adaptation, not dead weight. Restore the module and its three install sites, and point native_rejection_sampler back at its helpers instead of the inlined copies. This reverts the platform part of 58e513a. Signed-off-by: y-null <y-null@users.noreply.github.com>
…rofile The validated set measures stage-0 n-gram speculative decoding as part of the optimization set, but the profile generated from the base config inherited the base's deliberate omission of it (upstream fix fff0b0f, on the grounds that vLLM rejects prefix caching together with speculative decoding). That ground does not hold here: vLLM 0.29.0 in this container has no such mutual-exclusion check, and the validated branch ran exactly this pair -- prefix caching on with `async_scheduling: false` -- in both arms of today's 14-vs-15 A/B (two full three-tier runs). So the profile enables it with the base's 15 tokens (a 14-vs-15 A/B on 910B3 landed inside the run-to-run drift band: c1 -4.5%, c4 -7.5%, c8 +9.3% -> equivalent), while the base config keeps it off. With this, the profile differs from the base config only by the single-device dual-replica settings plus this speculation block. Signed-off-by: y-null <y-null@users.noreply.github.com>
…t is the tail-draft gate An earlier version of this comment said 14 and 15 were equivalent because the RTF difference sat near the run-to-run drift band. That reading was incomplete: the stage-0 tail-draft path is gated on `num_speculative_tokens == 15` (stage0_tail_draft.applies()), so k=14 switches it off silently. The 14-vs-15 A/B measured 22-37% worse TTFT in the k=14 arm, matching the tail-draft item's own account (TTFT -10%, RTF unchanged); RTF was inconsistent (-4.5% / -7.5% / +9.3%). The width itself is still untested in isolation. Signed-off-by: y-null <y-null@users.noreply.github.com>
…ofile
The omni prefix cache refuses the combination outright: stage_prefix_cache_config()
-> check_prefix_cache_token_accounting() raises OmniPrefixCacheUnmatchError
('omni prefix caching is not supported together with speculative decoding') as soon
as a stage has enable_prefix_caching and a speculative_config, and stage 0 of this
profile has the former. The runner call sites (NPU initialize_kv_cache, GPU
initialize_metadata_builders) are not wrapped in try/except, so the engine would
fail at kv-cache init.
The base config keeps stage 0 free of speculation for the same reason. Reverts
ad6ff944f (which added the block) and d7de6d4ee (its comment).
Signed-off-by: y-null <y-null@users.noreply.github.com>
…h speculation The comment blamed vLLM. It is not vLLM: prefix caching and speculative decoding are allowed together by vLLM itself (no such validation in the installed 0.29.0, and upstream main even has disable_eagle_block_drop for the interaction). The refusal is ours: stage_prefix_cache_config() -> check_prefix_cache_token_accounting() raises OmniPrefixCacheUnmatchError whenever a stage has enable_prefix_caching and any speculative_config -- regardless of async_scheduling and of the speculative method. Comment-only change; the configs are unchanged. Signed-off-by: y-null <y-null@users.noreply.github.com>
… needs
The deploy-config port dropped vllm_omni/platforms/npu/attention/ and its
install sites, and with them the only decode-capture path the 910B family
has. With the K-step configured, base and prof both fail in capture_model()
with a vector-core exception (acl 507035) on 910B3, so the service never
comes up; the same config on the pre-port tree reaches "Stage 1 engine
startup completed", so the gap is the module, not the part.
Restore the module and its three hooks:
* platforms/npu/attention/{__init__,fixed_kv_backend,fixed_kv_decode,fixed_kv_prefill}.py
-- whole. fixed_kv_prefill is imported by fixed_kv_backend, and narrow
replay reads captured_seq_lens/captured_slot_mapping, so nothing in it is
dead weight: it is the fixed-KV item's own delivery.
* NPUOmniPlatform.get_attn_backend_cls: route short-context decoders onto
OmniFixedKVAttentionBackend.
* NPUARModelRunner: the enable_hamming_sparse default, install_into_ascend_backend()
before parent init, and the per-capacity _capture_cudagraphs override.
Signed-off-by: y-null <y-null@users.noreply.github.com>
With the K-step armed, the stock rejection_sampler_triton_warmup faults the
910B vector core (acl 507035) during its dummy-token setup, which is why the
guard in ascend_warmup_patch exists. The pre-port tree could tell from the
environment; this tree asks the engine config instead, and the stage worker
does not have it yet at warmup time -- the engine hands the runner its
vllm_config only after load_model, and a deploy-layer record does not survive
the spawn that starts these processes. The warmup therefore ran and faulted
inside capture_model(), so stage 1 never came up.
* _skipped_names(): on Ascend910B return {"rejection_sampler"} without asking
_kstep_armed(). The torch-native sampler serves the armed path, and the
stock path only pays a first-request JIT of that kernel.
* _kstep_armed(): keep the engine-config answer first, fall back to the deploy
layer, so callers that do have the runtime config keep their precise answer.
* stage_config: record stage 1's declared frame count when a deploy YAML is
parsed and expose talker_frames_per_step() for those out-of-band readers.
Verified on 910B3: the prof deploy config now reaches "Application startup
complete" with stage 2's two replicas, and stage 1 logs the per-capacity
fixed-KV captures.
Signed-off-by: y-null <y-null@users.noreply.github.com>
…kers Stage workers are spawned, so they do not inherit the recorded value; the comment claimed a fork. The warmup guard on that side decides without it (see ascend_warmup_patch._skipped_names). Signed-off-by: y-null <y-null@users.noreply.github.com>
Same facts, shorter: the deploy-layer fallback, the 910B warmup skip and the fixed-KV route each lose a paragraph of restatement. Signed-off-by: y-null <y-null@users.noreply.github.com>
pre-commit failed on ruff-format (7 files) and ruff-check (6 findings). Run ruff format and ruff check --fix over vllm_omni/, and drop the unused prev_code assignment in minicpmo_4_5_omni_tts.py that F841 flagged. Signed-off-by: y-null <y-null@users.noreply.github.com>
Review pass over this branch's own additions. Comments and docstrings still described removed environment switches, internal item numbers, measurement rounds and out-of-tree design notes. They now describe what the code does, in the wording the rest of the repo uses. Dead code removed: * the unused ``MINICPMO_FAST_CODEC_PENALTY`` read (nothing consumed it); * ``fixed_kv_prefill.is_enabled()`` / ``STAGE0_BUCKETS`` / the ``VLLM_OMNI_FIXED_KV_PREFILL`` gate, which nothing called; * the scheduler's per-request ``states`` list, built and never read; * the unused ``_CODEC_EOS_TOKEN_ID`` left behind by the stop-chain fix; * the frame-count clamp whose only reader compares against 1. Behavior changes are limited to the two that make a gate match its reader: the vocab gate now requires the same arming predicate as the multi-frame path itself (``num_spec_tokens > 0``) rather than the wrapper class alone, and ``get_attn_backend_cls`` logs which of the two fixed-KV outcomes it took. The environment variables this branch introduces are registered in the inventory and the configuration docs. Signed-off-by: y-null <y-null@users.noreply.github.com>
``VLLM_OMNI_FIXED_KV_DECODE_MAX_LEN`` decided the same thing the deploy YAML already decides: the fixed-KV path engages when a stage's ``max_model_len`` is small enough to have fixed-capacity graphs. A second knob for that rule can only be set to a value the captured shapes do not support, so the ceiling is a code constant now and the stage's own ``max_model_len`` is what selects the outcome (Talker 4096 engages, a 32768-token Thinker keeps the stock path). No behavior change at the default: the constant is the old default 8192. The inventory, the configuration reference and the snapshot counts are updated with it. Signed-off-by: y-null <y-null@users.noreply.github.com>
``fixed_kv_*`` named one decode-side op argument, and the prefill module does not fix a KV length at all -- it buckets the prefill by token count. What all three do is capture a step whose shapes are static so that a replay needs nothing from the host, so they are named for that: fixed_kv_decode.py -> static_shape_decode.py fixed_kv_prefill.py -> static_shape_prefill.py fixed_kv_backend.py -> static_shape_backend.py The symbols follow the files: OmniStaticShapeAttentionBackend and its impl, OmniStaticShapeMetadataBuilder, the ``_static_shape_*`` helpers and the ``static_shape_*`` attention-metadata attributes. The reference to vllm-ascend's own ``vllm_ascend.attention.fixed_kv_decode`` module stays as it is. No behavior change. Signed-off-by: y-null <y-null@users.noreply.github.com>
FULL_DECODE_ONLY cannot run on the NPU stage-0 engine under vLLM 0.30.0, so the single-card deploy now pins stage 0 to PIECEWISE. The Talker (stage 1) keeps its full decode graph on the single-card deploy (in-graph sampling + multi-frame K expansion); multi-card variants stay PIECEWISE. Align the stage-scoped topology test with the per-stage configuration. Signed-off-by: y_null <y_null@qq.com>
- P2-1: reject logit_bias/allowed_token_ids at admission when the K-step codec stream is armed (OmniSchedulerMixin.add_request -> finish_requests FINISHED_ERROR); the in-model filter has no reader for them, and the talker_codec_sample module docstring no longer claims support - P2-2: gate the empty-row draft rollback on _talker_kstep_armed() (omni_ar_scheduler.py) -- rollback semantics verified for the Talker K-step loop only - P2-3: pack sampled_id/finished/emit into one D2H per frame in the minicpmo_4_5_omni_tts.py K loop (3 device syncs per frame -> 1) - P3-5: native_rejection_sampler._RESTORED latches only on a confirmed non-target SoC; an empty probe name retries instead of locking out the restore for the whole process lifetime - P3-6: single transition source _advance_classified shared by advance_codec_device_state and codec_sample_result (emit/finished computed once per frame); rep_counts histogram preallocated in the device state and zeroed in place per frame - claims-2: minicpmo_4_5.yaml stage0 comment attributes the K-step FULL-only RTF win to stage 1 Validation: py_compile all touched files; pytest tests/core/sched + tests/model_executor/models/minicpmo_4_5/test_talker_kstep_feedback.py = 255 passed (3 logprobs tests fixed by adding _talker_kstep_armed to the SimpleNamespace stub fixture) Signed-off-by: y-null <y-null@users.noreply.github.com>
…_config overlay The profiling profile was a 193-line copy of minicpmo_4_5.yaml and had already drifted from it (prefix caching, code2wav_max_npu_graphs, captured-shape counts). It also armed enable_prefix_caching on the NPU stage-0 engine of a duplex deploy, which validate_duplex_window_install rejects at startup. Replace it with a thin minicpmo_4_5_stage2_replicas.yaml overlay (base_config: minicpmo_4_5.yaml) that spells out only the two-replica topology and its sizing, inheriting the duplex contracts (including enable_prefix_caching=false) from the base config. The rename follows the *_stage1_replicas naming and the file is marked as a non-production profiling entrypoint. Signed-off-by: y_null <y_null@qq.com>
CI's pre-commit run fails because ruff-format would reformat the kstep unsupported-params comprehension in omni_scheduler_mixin and reformat talker_multiframe and test_talker_kstep_feedback. Apply the formatter output verbatim; no functional changes. Signed-off-by: y-null <y-null@users.noreply.github.com>
…ts.py Keep both execution paths: - K-step (HEAD): resolve_codec_sampling_params family, K-step request state, gpu_resident_buffer_keys, _merge_request_codec_params in the K-step make_omni_output loop. - MRv2 (upstream): _CodecWindowPenaltiesState, _install_mrv2_talker_sampler, MiniCPMO45TalkerSampler, _force_eos_on_sampled_ids (index_to_device variant kept, duplicate torch.tensor variant dropped), preprocess_decode_batch / make_omni_output_mrv2 / _HostCodecIds. Graft: pending codec-id consumption (deferred .item() EOS check from preprocess_decode_batch) into _make_omni_output_single_frame, with empty_speech computed after the pending pop so a pending-EOS row still forces EOS. _flush_deferred_cleanup now clears both sides' state dicts. Signed-off-by: y-null <y-null@users.noreply.github.com>
pre-commit (ruff format) flagged two double blank lines introduced by the merge conflict resolution; collapse both to single blank lines. Signed-off-by: y-null <y-null@users.noreply.github.com>
CachedRequestState in vLLM 0.30.0 has no num_output_tokens attribute (only num_tokens as prompt+output); this made every decode step fall back to uniform budget splitting. Use len(req.output_token_ids), the equivalent generated-token count, instead. Signed-off-by: y-null <y-null@users.noreply.github.com>
…o main Vendor the open vllm-project#7929 head so the MiniCPM-o duplex work that builds on its Talker K-step plumbing can be reviewed against the code it depends on. Signed-off-by: BeatSeat <wendavid552@gmail.com>
Outputs: grow CUMULATIVE dim-0 streams of matrices in place A CUMULATIVE stream re-consolidates the stream from the start on every step. Reusing capacity behind the previous view avoids O(T^2) recopying over long duplex epochs while returning value-identical views. Signed-off-by: BeatSeat <wendavid552@gmail.com>
…ge idle wait Framework hooks for pacing, unit continuation, and stage idle wait Add minimal opt-in hooks for duplex pacing, decoupled unit continuation, event-driven orchestrator loop, and stage idle waiting. Default behaviors for existing non-duplex and turn-based pipelines are preserved. Signed-off-by: BeatSeat <wendavid552@gmail.com>
Worker multi-frame decode hook for Talker K-step Expose a model-level multi_frame_decode_hook on GPU runners and waiting-safe dispatch in the AR scheduler. Default single-frame execution paths are unchanged. Signed-off-by: BeatSeat <wendavid552@gmail.com>
Talker K-step multi-frame decode on CUDA Implement CUDA Talker multi-frame decoding using the vllm-project#7929 hook interface, with native sampling controls, repetition penalty cache, and pinned copies. Signed-off-by: BeatSeat <wendavid552@gmail.com>
…A graphs Replay Talker K-step per-frame sampling tail as CUDA graphs Capture the per-frame logits adjustment and sampling tail into private CUDA graphs interleaved between Talker forward replays, eliminating per-frame host launch overhead. Signed-off-by: BeatSeat <wendavid552@gmail.com>
…KV, and CUDA graphs Stage 0 streaming audio encoder batching, in-place KV, and CUDA graphs Batch streaming audio encoding across sessions, maintain KV caches in place, capture audio encoder CUDA graphs with resident cache slots, and add incremental fbank. Signed-off-by: BeatSeat <wendavid552@gmail.com>
Stage 0 fused vision encoder and CUDA graphs Add packed SDPA vision encoding, cross-session frame batching, fused layer norm ops, and vision CUDA graph capture for multi-modal duplex serving. Signed-off-by: BeatSeat <wendavid552@gmail.com>
Code2Wav Whole-Euler slot pool and fused DiT body Implement Whole-Euler slot pool memory management for high-concurrency duplex Code2Wav, fused DiT block execution with tiled TF32 attention, and single-projection speaker embeddings. Signed-off-by: BeatSeat <wendavid552@gmail.com>
…HiFT, and deadline scheduling Code2Wav exact-shape CUDA graphs for flow encoder, HiFT, and deadline scheduling Add exact-shape CUDA graphs for the flow encoder continuation chunks and HiFT vocoder, along with deadline-driven request scheduling to maximize concurrency and minimize latency. Signed-off-by: BeatSeat <wendavid552@gmail.com>
…trim HiFT zero-width mel safeguard and Stage 2 memory trim Return empty HiFT outputs for zero-width mels to prevent conv padding failure, and drop unused upstream flow attention buffers in Step-Audio2 token2wav. Signed-off-by: BeatSeat <wendavid552@gmail.com>
Consolidated single recommended deployment profile Consolidate high-concurrency full duplex tuning into minicpmo_4_5_h100.yaml and update deploy config test expectations. Signed-off-by: BeatSeat <wendavid552@gmail.com>
…aphs into minicpmo_4_5_h100.yaml Consolidate Stage 0 multi-modal vision and streaming audio CUDA graphs, resident slots, incremental fbank, duplex pacing, and codec deadline scheduling into minicpmo_4_5_h100.yaml. Signed-off-by: BeatSeat <wendavid552@gmail.com>
Signed-off-by: BeatSeat <wendavid552@gmail.com>
Signed-off-by: BeatSeat <wendavid552@gmail.com>
hsliuustc0106
added
omni
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces end-to-end full-duplex serving optimizations for MiniCPM-o 4.5 on single-GPU NVIDIA H100 (and SM80+ architectures), achieving real-time duplex serving up to 20 concurrent sessions (d20).
Key Optimizations
1. Stage 0 (Thinker)
2. Stage 1 (Talker)
3. Stage 2 (Code2Wav)
StepReleasePolicy) replacing fixed sleep windows.4. Duplex Orchestration & Pacing
Scope & Non-Regression Guarantees
models/minicpmo_4_5/ops/.minicpmo_4_5_h100.yaml.Signed-off-by: BeatSeat wendavid552@gmail.com