Repository navigation
[Feature][TTS] adaptive chunk ramp (Phase 2 buffer-feedback controller) - #6001
Conversation
hsliuustc0106
added
enhancement
|
This PR touches vllm_omni/deploy/, vllm_omni/model_executor/, tests/model_executor/, vllm_omni/distributed/, vllm_omni/worker/ (7 files). Based on CODEOWNERS coverage of the changed files, the most-related reviewers appear to be: Could one of you take a look when you get a chance? Thanks! |
3403be9 to
3696c6d
Compare
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
This PR appears to belong to: docs/design/module/ar_runtime.md, docs/design/module/input_output_modality_contracts.md, docs/design/module/omni_connector.md. Module owners: @tzhouam @yinpeiqi @fake0fan @Sy0307 @Gaohan123 @amy-why-3459 @alex-jw-brooks @princepride @yuanheng-zhao @xuechendi @natureofnature Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment. |
Wallbreazzz
changed the title
|
cc @linyueqian — this PR implements the Phase 2 adaptive chunk ramp from your design (A+B hybrid per your 2026-07-19 comment on #4855). Would appreciate your review on:
Phase 1 fixed ramp #5152 (merged) is the baseline this builds on. Validated on Ascend 910B NPU: c=2 RTF 0.56 / AUDIO_UNDERRUN 0.02s (vs #4913 ~1028ms gap); c=8 saturated (per-stream RTF>1, structural, out of Phase 2 scope per your saturation caveat). Audio quality verified by listening. I couldn't formally add you as a reviewer (PR-author lacks |
linyueqian
left a comment
linyueqian
left a comment
There was a problem hiding this comment.
Reviewed 3696c6d2. One blocker in the adaptive emit bookkeeping, plus a dead block and two smaller items. Nice to see the phase-2 controller land as a separate module with the phase-1 ramp path left intact.
pre-commit is red on formatting only (E501 at chunk_size_utils.py:242 and qwen3_tts.py:119, plus ruff-format on the same two files), so that part is mechanical.
Test gap worth closing alongside the fix below: TestAdaptiveEmission._emit replaces code_prompt_token_ids wholesale on every call, so the real one-frame-per-decode-step accumulation path, which is exactly where the High lives, is never exercised.
One thing I checked and can rule out: the controller does advance past chunk 0. ramp_chunk_count is incremented unconditionally on a successful put (chunk_transfer_adapter.py:369, omni_connector_model_runner_mixin.py:1040 and :1166), not gated on the ramp branch.
| if adaptive_enabled: | ||
| ctrl = transfer_manager._adaptive_states.get(request_id) | ||
| if ctrl is not None: | ||
| ctrl.record_emit(time.monotonic(), context_length, length, context_length) |
There was a problem hiding this comment.
[High] This marks every accumulated frame as emitted while compute_adaptive_emit only shipped target_size of them.
compute_adaptive_emit returns (True, target_size) when new_frames >= target_size, but accumulated here is length, so accumulated_at_last_emit jumps to the full accumulation. The surplus new_frames - target_size is never sent, and since line 291 ships code_prompt_token_ids[-context_length:] (the newest slice), the emitted chunk is also non-contiguous with the previous one, which breaks the contiguity that Code2Wav's per-request decoder cache assumes.
new_frames can exceed target_size because target_size falls inside an accumulation window: buffer_ms = emitted_ms - elapsed_ms strictly decreases while last_target, and therefore ramp_floor, stays frozen until the next emit, so compute_next_chunk_size is non-increasing between emits. A call-latency spike then drops the target below the frames already queued.
Transcribing the two pure functions with ewma=16ms, last_target=2 (so ramp_floor=4), buffer ~322ms, and one 320ms scheduler stall mid-window:
call now new target action
6 1.374 6 11 hold
7 1.694 7 4 EMIT ctx=4 -> record_emit(accumulated=27)
new_frames=7, emitted=4, so 3 frames (240 ms of audio) are dropped and frame indices 20..22 never reach Code2Wav.
The trigger is any latency spike while new_frames < target_size and new_frames > ramp_floor, which is the high-concurrency regime this PR targets and that qwen3_tts_high_concurrency.yaml enables.
Smallest fix: pass ctrl.accumulated_at_last_emit + context_length instead of length.
| if finished: | ||
| ctrl.log_summary(request_id) | ||
|
|
||
| end_index = min(length, left_context_size_config + context_length) |
There was a problem hiding this comment.
[Medium] This block is dead. Lines 291-292 immediately overwrite window_frames and set left_context_size = 0, and end_index has no other reader in the function.
Either drop it, or it was meant to re-introduce a left-context window for the adaptive path and silently does not. Worth noting it sits outside the adaptive_enabled guard, so had it taken effect it would also change the fixed-ramp and default paths, and left_context_size is shipped in meta and used downstream to trim.
| self.total_gap_s, | ||
| ) | ||
|
|
||
| def reset(self) -> None: |
There was a problem hiding this comment.
[Low] reset() has no callers. The integration constructs a fresh AdaptiveChunkController whenever chunk_index == 0 instead of resetting in place, so the class docstring ("State is per-segment: reset() must be called at segment boundaries") does not describe what actually happens. Either call it at the segment boundary or drop it and adjust the docstring.
linyueqian
left a comment
linyueqian
left a comment
There was a problem hiding this comment.
Round 2. The controller matches the A+B design from #4855: buffer depth as the regulated state, measured EWMA as the model, per-chunk frame counts rather than table selection, per-segment state, host-only arithmetic. The unconditional ramp_floor is a deliberate departure from the buffer_ms > n * frame_time + margin rule I wrote there, and the death-spiral rationale in the PR body is consistent with the W5 warning, so no objection to that call.
One blocking defect, with a one-line fix. On the non-finished emit path the controller marks every accumulated frame as shipped while sending only target_size of them, so codec frames are skipped in the middle of the stream. Code2Wav holds a per-request stateful decoder cache, so this is lost audio plus a desynchronised decoder, not simply a shorter output. Details inline on chunk_size_utils.py.
Why the Ascend validation did not surface it: the loss concentrates in the 12 to 40 ms per-frame band and disappears above roughly 55 ms per frame. 910B sits above that. I swept 714 single-latency-spike scenarios per row against the real emission function, one frame per call:
| measured ms/frame | scenarios losing frames |
|---|---|
| 12 | 1.7% |
| 16 | 5.9% |
| 25 | 94.4% |
| 40 | 1.4% |
| 55 and above | 0% |
Controls running the Phase 1 fixed ramp and the repo default over identical latency traces lose nothing, so this is specific to the adaptive path.
Measured A/B
L20X (masked H200), Qwen3-TTS-12Hz-1.7B-Base, vllm 0.26.0, this head, 3 reps per cell, one warmup discarded, quiet GPU. Identical code, hardware, model and prompt across arms; the feature is opt-in so config is the only variable.
| c | default RTF / TTFA ms / gap s | Phase 1 ramp | adaptive |
|---|---|---|---|
| 1 | 0.454 / 1972 / 0.095 | 0.232 / 671 / 0.000 | 0.265 / 721 / 0.000 |
| 4 | 2.074 / 9627 / 0.117 | 0.628 / 2306 / 0.000 | 0.627 / 2474 / 0.000 |
| 8 | 1.729 / 7206 / 0.148 | 1.079 / 4497 / 0.000 | 1.203 / 5452 / 0.000 |
The controller does zero the underrun relative to the current default, and that reproduces off Ascend. The merged Phase 1 fixed ramp zeroes it too, though, at equal or better RTF and TTFA at every concurrency. Against the baseline this PR actually builds on, the adaptive path shows no measured continuity or latency win on a single machine; at c=8 it costs about 10% RTF and 20% TTFA, and its run-to-run RTF spread is 2x to 6x the fixed ramp's (0.249 vs 0.043).
What the data does support is portability: on 11 ms/frame hardware the controller self-tunes to [2, 4, 25, 25, 3] with no retuning of the hand-picked Ascend [4, 4, 8, 16, 25]. That is the stronger pitch, and I would rewrite the PR body around it. The current 1028 ms -> 0.02 s headline is measured against the #4913 campaign rather than a paired Phase 1 baseline, so it is not an A/B. This is a description change, not a code change, and I would not hold the merge on it.
Note this run sat at 11.1 ms/frame p50, inside the roughly 1.7% band, and saw 0 losses in 194 emits. It neither confirms nor clears the defect below.
To clear the block: the one-line fix, a regression test that exercises the accumulation path, green pre-commit, and a rebase past #5976 (this branch predates the vllm 0.27 rebase by about 25 minutes and cannot currently run against main). Happy to re-review quickly once those land.
| if not finished and new_frames < target_size: | ||
| return False, 0 | ||
|
|
||
| return True, target_size |
There was a problem hiding this comment.
[blocking] Silent codec-frame loss and Code2Wav decoder desync.
This returns target_size, but the caller records every accumulated frame as shipped (qwen3_tts.py:281 passes accumulated=length) and then slices the newest frames (qwen3_tts.py:291, [-context_length:]). When new_frames > target_size the surplus is skipped in the middle of the stream, and because accumulated_at_last_emit has jumped past them the terminal flush cannot recover them.
The trigger is reachable. new_frames grows by exactly one per call (_extract_last_frame returns one frame, appended at qwen3_tts.py:94; the adapter serialises calls on one save thread). target_size is non-increasing inside an accumulation window, because buffer_ms = emitted_ms - elapsed_ms shrinks with wall clock while ramp_floor is frozen until the next record_emit. So a one-unit drop of target_size at the crossing point is enough to overshoot by one frame, which is 80 ms of audio plus a decoder-state discontinuity.
Minimal reproduction against the real talker2code2wav_async_chunk, one frame per call with a controlled clock, steady 16 ms per frame and a single 24 ms step at call 2:
emit at call= 1 accumulated_length= 2 new_frames= 2 shipped= 2 frames 1-2
emit at call= 5 accumulated_length= 6 new_frames= 4 shipped= 4 frames 3-6
emit at call=16 accumulated_length= 17 new_frames= 11 shipped= 10 frames 8-17 (frame 7 dropped)
emit at call=41 accumulated_length= 42 new_frames= 25 shipped= 25 frames 18-42
Chunk N ends at frame 6, chunk N+1 starts at frame 8.
Smallest fix is to ship everything accumulated, exactly as the finished branch above already does:
| return True, target_size | |
| return True, new_frames |
That keeps accumulated=length and the [-context_length:] slice correct and contiguous, at the cost of a chunk occasionally exceeding target_size by the overshoot (typically one frame). Keeping target_size as the shipped count would instead require both accumulated = ctrl.accumulated_at_last_emit + context_length and an index-based slice, which is more surface for the same outcome.
| if adaptive_enabled: | ||
| ctrl = transfer_manager._adaptive_states.get(request_id) | ||
| if ctrl is not None: | ||
| ctrl.record_emit(time.monotonic(), context_length, length, context_length) |
There was a problem hiding this comment.
[blocking] This is the other half of the frame-loss issue described on chunk_size_utils.py:321.
accumulated is passed as length, so accumulated_at_last_emit advances past frames that were never sent. Fixing compute_adaptive_emit to return new_frames makes this call correct as written, so no change is needed here if you take that fix.
|
|
||
|
|
||
| class TestAdaptiveEmission: | ||
| def _emit(self, tm, rid, n_frames, finished=False): |
There was a problem hiding this comment.
[important] This helper replaces code_prompt_token_ids[rid] wholesale on every call, so new_frames jumps straight to the target and the real one-frame-per-decode-step accumulation path is never entered. That is exactly where the defect above lives, which is why all 138 tests pass on this head.
Please add a regression test that appends one frame per call with a controlled clock (patch qwen3_tts.time.monotonic) and asserts that the concatenation of every emitted chunk equals the accumulated frame sequence, in order, with no gaps. Give each frame a distinct value so the assertion can detect a skipped frame rather than just a short total.
| if finished: | ||
| ctrl.log_summary(request_id) | ||
|
|
||
| end_index = min(length, left_context_size_config + context_length) |
There was a problem hiding this comment.
[suggestion] Dead block. end_index, left_context_size and window_frames are computed here and immediately overwritten at lines 291 and 292, and end_index has no other reader.
It also sits outside the adaptive_enabled guard, so the fixed-ramp and default paths pay a list copy of up to codec_left_context_frames + context_length frames on every emit for nothing. Either delete it, or it was meant to reintroduce a left-context window and silently does not.
| if ctrl is not None: | ||
| ctrl.record_emit(time.monotonic(), context_length, length, context_length) | ||
| if finished: | ||
| ctrl.log_summary(request_id) |
There was a problem hiding this comment.
[suggestion] This fires once per segment, not once per request. chunk_transfer_adapter.py:342 passes is_finished=is_segment_finished or is_finished, so a resumable /v1/realtime request emits one [ADAPTIVE] req=... done line per segment, while the docstring on log_summary promises one line per request at request end. Either reword the docstring or gate on the terminal finish.
| self.total_gap_s, | ||
| ) | ||
|
|
||
| def reset(self) -> None: |
There was a problem hiding this comment.
[suggestion] reset() has no caller. Segment boundaries are handled by self._adaptive_states.pop(external_req_id, None) (chunk_transfer_adapter.py:391 and :465), which discards the controller entirely so the next segment builds a fresh one with a fresh first_emit_time.
That behaviour is correct, but it makes the class docstring claim ("reset() must be called at segment boundaries") false. Suggest deleting reset() and that sentence, or wiring it in.
| return True, target_size | ||
|
|
||
|
|
||
| def parse_adaptive_config( |
There was a problem hiding this comment.
[suggestion] steady is documented here as the adaptive max but never read in the body; the max is applied at the call site via max_frames=chunk_size. Suggest dropping the parameter.
| return self._ramp_chunk_count | ||
|
|
||
| @property | ||
| def adaptive_states(self) -> dict[str, Any]: |
There was a problem hiding this comment.
[suggestion] This property has no reader. The processor reaches for transfer_manager._adaptive_states directly, unlike ramp_chunk_count, which goes through its property. Worth picking one convention: either read through this accessor in qwen3_tts.py, or drop it.
| if self.frame_time_ewma_ms == 0: | ||
| self.frame_time_ewma_ms = measured_ms | ||
| else: | ||
| self.frame_time_ewma_ms = self.EWMA_ALPHA * measured_ms + (1 - self.EWMA_ALPHA) * self.frame_time_ewma_ms |
There was a problem hiding this comment.
[suggestion] pre-commit is red on this line. Verified with the repo's pinned ruff 0.14.10: E501 (125 > 120) here and at qwen3_tts.py:119, plus ruff-format reformatting both files.
2a0283a to
91bbed7
Compare
Round 2 review fixes — updated, ready for re-reviewAll issues addressed per @linyueqian's suggestions. Branch rebased onto latest main, pre-commit green, force-pushed. B1 (blocker) — fix changed to @linyueqian's suggested approachPrevious fix was incomplete: I changed # chunk_size_utils.py — compute_adaptive_emit, non-finished path
return True, new_frames # was: return True, target_sizeThis makes
B2 — regression test updated
(10, 0, 4, False, True, 10), # new_frames=10 > target=4 → ship 10
(8, 2, 4, False, True, 6), # new_frames=6 > target=4 → ship 6
(100, 0, 25, False, True, 100), # new_frames=100 > target=25 → ship 100These would fail on the old code (which returned
All issues status
Ascend 910B NPU A/B (c=8, voice_clone, 40 requests)Validated on vllm 0.27.1 + vllm-ascend (PR #14027) + vllm-omni main. Same text, hardware, model across all three arms; feature is opt-in so config is the only variable.
Key findings:
The adaptive controller shows clear continuity and underrun advantages over both the default and the Phase 1 fixed ramp on Ascend 910B at c=8, refuting the L20X single-machine A/B concern. Audio quality verified by listening (complete, clear, no artifacts). @linyueqian — would appreciate a re-review. |
linyueqian
left a comment
linyueqian
left a comment
There was a problem hiding this comment.
Round 3 on 91bbed71. B1 is fixed and I re-ran the round-2 harness against this head to
confirm it: 0 frames lost across 4,872 injected-latency scenarios. Reverting only
chunk_size_utils.py:320 back to return True, target_size in the same harness reproduces
the old profile (12ms 1.3%, 16ms 9.8%, 25ms 85.9%, 40ms 1.4%, >=55ms 0%), so the harness is
still sensitive and the fix is what closes it. S1 through S6 all landed, and the new
TestAdaptiveAccumulationPath does drive the real processor one frame per call, which was the
coverage gap I asked for. I verified the 11 compute_adaptive_emit parametrize cases directly
against the head's function, 11/11. Clearing my changes-requested.
Two new items below, found by following the emit sizes into Code2Wav rather than stopping at
the transfer adapter. Neither blocks, but the first is the missing explanation for the CUDA
deficit in the L20X table in the PR body.
High: adaptive chunk sizes bypass the Code2Wav decode CUDA graphs
The decode graph table is derived from the ramp config, not from what the controller emits.
segmented_graph_wrapper.py:113 builds transitions[target] = previous from
codec_chunk_ramp when set, otherwise from initial_codec_chunk_frames plus repeated
codec_chunk_frames steps up to sliding_window. At decode time
modeling_qwen3_tts_tokenizer_v2.py:1267 selects a graph by matching the request's cumulative
suffix-frame count against the sources of that dict, exactly, with no bucketing:
target = (prefix_length + codec_chunk_frames if rolling
else next((t for t, source in transitions_by_mode[mode].items() if source == previous), None))
if target is None or not 0 < new_frames <= codec_chunk_frames:
groups.setdefault(("eager", 0), []).append(index)The controller picks sizes at runtime, so previous never lands on a source until the suffix
cache starts rolling. With initial_codec_chunk_frames: 1 and codec_chunk_frames: 25 the
sources are {1, 26, 51, 72}, while a [2, 4, 25, 25, 3] trajectory walks 2, 6, 31, 56.
Chunk 0 misses too, and that part is new here. qwen3_tts.py:146 now lets dynamic IC override
a configured initial_codec_chunk_frames, and compute_dynamic_initial_chunk_size only
returns powers of two starting at 2, so with the shipped deploy configs chunk 0 is 2/4/8/16
while the wrapper captured for 1. That fails suffix_frames == initial_chunk_frames
(modeling_qwen3_tts_tokenizer_v2.py:1250) and the identical guard in
_decode_xvec_prefix_batch (segmented_graph_wrapper.py:725).
Replaying the head's real AdaptiveChunkController and compute_adaptive_emit against
verbatim reimplementations of both lookups, with codec_chunk_frames: 25,
initial_codec_chunk_frames: 1, sliding_window: 72:
| audio | adaptive trajectory | chunks on a graph | Phase 1 ramp |
|---|---|---|---|
| 4.7 s (59 frames) | [2, 4, 25, 25, 3] |
0/5 | 6/6 |
| 12 s (150 frames) | [2, 4, 25, 25, 25, 25, 25, 19] |
3/8 | 9/9 |
| 24 s (300 frames) | [2, 4, 25, ...] |
9/14 | 15/15 |
A typical short utterance decodes entirely eager. Two cross-checks that this is real: the
simulated [2, 4, 25, 25, 3] is exactly the trajectory I measured live on L20X in round 2,
and the ~10% RTF / ~20% TTFA cost against the fixed ramp at c=8 that I reported then, and
could not attribute, is what losing graph replay across the whole ramp phase looks like.
Setting codec_chunk_ramp alongside adaptive does not help: the wrapper then captures the
ramp's sources and the controller still does not follow them (0/5 and 3/8 on the same cases).
There is no re-capture at decode time and no padding path that can serve a non-matching
cumulative position; the new_frames <= expected_new_frames tolerance at
segmented_graph_wrapper.py:791 only applies once previous already matches a source.
Ascend is unaffected: qwen3_tts_code2wav.py:718 gates the whole cudagraph setup on
device.type == "cuda", so the 910B A/B does not cover this either way. It also means the
feature is landing with its measured wins on the one platform where this cost cannot appear.
Caveat on my evidence: this is a static trace plus a simulation on the real controller, not a
live hit-rate measurement. One command confirms it on any CUDA box:
VLLM_OMNI_QWEN3_CODE2WAV_CUDAGRAPH_STATS=1 with ..._STATS_LOG_EVERY=1 prints hit_rate=
and the suffix_N:no_graph breakdown per arm.
Smallest fix that unblocks the merge: warn at startup when codec_chunk_adaptive is on and the
decoder cudagraph is enabled, and say so in the deploy YAML, where the "Decoder CUDA Graph
sizing" block sits directly below the new keys and currently describes only the ramp and fixed
cases. The real fix is either to derive the capture table from the adaptive reachable set (up
to 71 sources per batch bucket at min_frames: 2, so probably too many) or to snap the
controller's cumulative boundaries onto the captured grid. Worth its own PR either way.
Medium: a partial graph replay at a matched source leaves the decoder cache stale
segmented_graph_wrapper.py:791 admits a chunk with new_frames < expected_new_frames and
zero-pads it, and :839 trims the audio correctly, but the cache advance at :845 is guarded
on new_frames == expected_new_frames. On a partial chunk the request keeps its old
suffix_frames, suffix_quantized and suffix_conv, so the next chunk is decoded without the
partial chunk's frames in its context. The audio for the partial chunk itself is fine; the
continuity of the following chunk is not.
Under the fixed ramp and the default path this only ever fires on the terminal chunk, where
not advancing the cache is harmless, which is presumably why the guard is written that way.
Adaptive is the first caller that can produce a non-terminal sub-delta chunk: it needs the
cumulative count to land exactly on a captured source (26 or 51 with the shipped config) and
the next chunk to be smaller than the captured delta, which is the common case during the ramp.
I have not reproduced this, and it is arguably a pre-existing sharp edge in the wrapper rather
than something this PR wrote. Flagging it because this PR is what makes it reachable. A cheap
guard is to fall back to eager when new_frames != expected_new_frames and the request is not
finished, instead of replaying and skipping the commit.
Low: skip_dynamic_ic also changed the fixed-ramp path
qwen3_tts.py:146. The old gate was ramp is None and not fixed_initial_chunk_size; the new
one drops the ramp is None term. With codec_chunk_ramp set, adaptive off and no fixed IC,
dynamic IC now runs on every request. The ramp branch at :250 never reads
initial_chunk_size, so the value is discarded, but it costs an O(active requests) scan plus a
_cached_ic entry per request on the merged Phase 1 path, and it contradicts the comment two
lines above it. Suggest
skip_dynamic_ic = (ramp is not None or fixed_initial_chunk_size) and not adaptive_enabled.
Low: record_emit's target_size parameter is now dead weight
qwen3_tts.py:287 passes context_length for both n_frames and target_size, and it is the
only production call site, so last_target is now the last emitted size rather than the last
target. Harmless for the ramp floor, but the NOTE: at chunk_size_utils.py:229 about
multiple compute_next_chunk_size calls per emit no longer describes how last_target is set.
Drop the parameter and assign last_target = n_frames.
Nits:
- The adaptive empty-finished sentinel at
qwen3_tts.py:245builds
MetaStruct(finished=...)only, while the other two sentinels in the same function
(:184and:257) also setrequest_idandleft_context_size=0. No impact I can find,
sinceleft_context_sizealready defaults to 0 atqwen3_tts_code2wav.py:285, but the
inconsistency is easy to remove. - The
TestAdaptiveAccumulationPathdocstring still describes the earlier
accumulated_at_last_emit + context_lengthapproach ("Fix: accumulated=6+6=12"), which is
not what shipped. The test body is correct.
One thing I checked and can rule out: compute_adaptive_emit flushing unclamped new_frames
cannot exceed codec_chunk_frames and trip the 0 < new_frames <= max_new_frames check at
modeling_qwen3_tts_tokenizer_v2.py:1120. _extract_last_frame appends exactly one frame per
call, and an emit only happens once new_frames reaches a target_size that is itself clamped
to codec_chunk_frames, so the previous call had at most 24 pending and the emit carries at
most 25. The same shape already exists in Phase 1's compute_ramp_emit.
Validation gaps
Buildkite has not run on this head. Only DCO, build, docs and pre-commit are green, and the
labels are enhancement and tts, so the unit suite is unverified in CI and ready needs
reapplying after the force-push.
91bbed7 to
fee034e
Compare
Round 3 review fixesAddressed the Low items and Nits from @linyueqian's Round 3 review. High (CUDA graph bypass) and Medium (partial replay cache staleness) are non-blocking — High is a follow-up PR, Medium is a pre-existing wrapper sharp edge. Force-pushed Low:
|
linyueqian
left a comment
linyueqian
left a comment
There was a problem hiding this comment.
Re-reviewed the amend 91bbed71..fee034ea. Flagging up front that my round-3 approval was submitted against 91bbed71, and the force-push moved the head, so it no longer covers what is here. The delta is small but it changes the controller, and my round-3 evidence was measured on the old assignment, so it does not carry over unchanged.
Three changes. Two look right. MetaStruct on the terminal empty payload now carries request_id and left_context_size=0 instead of only finished, which is a straightforward fix. And skip_dynamic_ic now includes ramp is not None, which finally makes the code do what the comment beside it already claimed, that a fixed ramp replaces IC and steady entirely. That is a behaviour change on the fixed-ramp path rather than a pure cleanup, so it is worth saying explicitly in the PR body that fixed-ramp deployments now skip dynamic IC where previously they did not.
The third one is inline and is the reason this is a comment rather than a re-approval.
Still open from round 3, and untouched by this delta: the adaptive chunk sizes bypass the Code2Wav decode CUDA graphs, since the graph table is built from codec_chunk_ramp or the fixed IC plus codec_chunk_frames steps, not from what the controller actually emits. The delta does not touch segmented_graph_wrapper.py or modeling_qwen3_tts_tokenizer_v2.py, so that item stands as written.
Validation: static reads only, no code from this head executed. I confirmed both heads share merge-base d6b66fab, so this is a clean amend rather than a rebase, and diffed the two PR footprints against that base rather than taking a two-head diff. I did not re-run the round-2 latency harness against this head, which is the gap that matters most here: the 4,872-scenario result quoted in my round-3 approval was measured with last_target = target_size, and this amend changes that assignment, so that number should not be cited for fee034ea until it is re-run.
| self.last_emit_time = now | ||
| self.frames_emitted_this_segment += n_frames | ||
| self.accumulated_at_last_emit = accumulated | ||
| self.last_target = n_frames |
There was a problem hiding this comment.
[important] Switching last_target from the requested target to the emitted count converts a transient latency spike into a persistent floor increase for the rest of the segment.
ramp_floor = self.last_target + ramp_delta at line 224, and compute_next_chunk_size returns max(ramp_floor, greedy_result) capped by max_frames. In the common case this change is a no-op, because frames accumulate one at a time so new_frames crosses target_size exactly and n_frames == target_size. The two diverge precisely on the path the B1 fix exists for: when a latency spike makes new_frames overshoot, compute_adaptive_emit now ships all of them, so n_frames can be several times target_size.
When that happens the floor jumps to the spike size plus delta and, because ramp_floor only ever grows from last_target, it never comes back down within the segment. One transient stall therefore ratchets the controller toward max_frames early, which is a latency regression on exactly the fast-hardware case the ramp is tuned to keep at low TTFP.
If shipping the true emitted size is the intent, that is defensible, but then the ramp wants a floor that can recover, for example seeding from min(n_frames, target_size) or decaying last_target toward the greedy result when the buffer recovers. Worth a comment either way, since the current line reads as bookkeeping and is actually a control decision.
The practical ask: the 4,872-scenario latency-harness result in my round-3 approval was measured with last_target = target_size. Re-running that harness on this head would settle both the frame-loss property and whether the ramp trajectory shifted.
…ler) Phase 2 streaming-continuity (vllm-project#3535, design vllm-project#4855). Builds on Phase 1 vllm-project#5152. Per-request EWMA-scaled adaptive controller; chunk 0 dynamic IC, chunk 1+ greedy buffer-feedback with unconditional ramp_floor + hysteresis lock. Config: codec_chunk_adaptive, codec_chunk_min_frames, codec_chunk_safety_margin_ms, codec_chunk_ramp_divisor (default 15), codec_chunk_ramp_delta_min (default 2). max = codec_chunk_frames (no separate max -- capping below steady is a death-spiral footgun; aligns with linyueqian's clamp spec). ramp_divisor/delta_min replace hardcoded 15/2; defaults preserve original formula so tests stay green. Invariants: unconditional ramp_floor (no vocoder-cost death-spiral), finished flushes all remaining frames, per-segment state, host-only no device syncs. Downward adaptation deferred until W5 (streaming decoder state). Per-request INFO telemetry mirrors stream_client.py's underrun model: [ADAPTIVE] req=... done: chunks=[...] gaps_ms=[...] total_gap=Xs Validated on Ascend 910B NPU: c=2 RTF 0.56/underrun 0.02s, c=8 saturated (structural, out of scope). Audio quality verified by listening. Signed-off-by: Wallbreazzz <110282866+Wallbreazzz@users.noreply.github.com>
fee034e to
cdb0d16
Compare
Round 4:
|
linyueqian
left a comment
linyueqian
left a comment
There was a problem hiding this comment.
Re-reviewed the amend fee034ea..cdb0d161. The round-4 finding is fixed exactly as asked, with a regression test that pins both halves of the invariant, and a property worth naming makes this immediately approvable: the controller is now behaviourally identical to the state my round-3 harness validated, so that evidence applies to this head without a re-run.
The fix restores target_size as a record_emit parameter and sets last_target = target_size rather than the shipped n_frames, with a comment stating the ratchet hazard precisely: under the B1 fix n_frames can overshoot target_size during a latency spike, and keying the floor on it would lock the ramp toward max_frames prematurely. The call site passes the same target_size the emit decision used, so the floor tracks intent while B1's ship-everything behaviour is preserved.
test_no_frames_skipped_when_target_drops now covers the exact scenario from my finding: a greedy target of 18 collapsing to a floor of 6 on a clock jump, asserting that all 18 accumulated frames ship with the slice contiguous, and that last_target lands at 6 rather than 18. That is both properties, no loss and no ratchet, in one test, and it would have caught either regression.
On the evidence question from round 4: I said the 4,872-scenario latency-harness result must not be cited for a head whose assignment differed from the one measured. This head resolves that the right way. At 91bbed71, where the harness measured 0 frames lost, compute_adaptive_emit returned new_frames and last_target was set to target_size. This head has the same signature and the same assignment; the only deltas beyond it are the MetaStruct terminal fix, the skip_dynamic_ic fixed-ramp correction I reviewed in round 4, the new comment and the new test, none of which touch the controller math. So the round-3 measurement is valid for cdb0d161 as it stands.
Still open and still non-blocking, carried from round 3: adaptive chunk sizes bypass the Code2Wav decode CUDA graphs, since the graph table derives from the ramp config rather than what the controller emits. That remains the likely explanation for the CUDA deficit in the PR body's L20X table and deserves its own follow-up. And the skip_dynamic_ic behaviour change on the fixed-ramp path is still worth one line in the PR description.
Validation: static reads only, no code from this head executed. Both heads share merge-base with main, so the amend is clean; I diffed the controller and the call site, read the new test end to end, and verified the behavioural equivalence claim against the round-3 state by comparing the record_emit signature and assignment at 91bbed71 and here.
linyueqian
added
ready
|
Current head |
…r) (vllm-project#6001) Signed-off-by: Wallbreazzz <110282866+Wallbreazzz@users.noreply.github.com>
MOSS-TTS Local has the same chunk-size cliff as Qwen3-TTS: chunk 0 emits 1 frame (80ms), then jumps directly to 15-frame steady chunks, causing an audible silence gap in streaming output. This PR reuses the architecture-level chunk ramp API (chunk_size_utils.py) already merged for Qwen3-TTS (vllm-project#5152 fixed ramp, vllm-project#6001 adaptive controller) and wires it into MOSS-TTS's talker2codec_raw_async_chunk processor. Changes: - moss_tts.py: Add ramp/adaptive branches to talker2codec_raw_async_chunk. - parse_chunk_ramp / parse_adaptive_config parsed once per transfer_manager - Ramp: uses compute_ramp_emit with cumulative frame tracking - Adaptive: uses AdaptiveChunkController + compute_adaptive_emit - Original IC/steady threshold logic preserved when neither is configured - Track chunk_index and cumulative emitted frames per request for ramp math - serving_speech.py: Set tts_local_seed from deploy config default_seed for MOSS-TTS variants (moss_tts/moss_tts_realtime/moss_tts_local), matching the existing Qwen3-TTS pattern. - moss_tts_local.yaml: Document codec_chunk_ramp / codec_chunk_adaptive config options (commented out by default for backward compat) Reuses chunk_size_utils.py (parse_chunk_ramp, compute_ramp_emit, AdaptiveChunkController, compute_adaptive_emit, parse_adaptive_config) shared with Qwen3-TTS — no new utility code, model-agnostic API. Benchmark (2-NPU, Ascend 910B2C, 5 rounds, R1 discarded, median R2-R5): C=8 (concurrency 8): Streaming continuity: 10% → 100% (+90pp) RTF: 0.19 → 0.36 (still <1.0, real-time) E2EL: 2980ms → 2989ms (neutral) C=1 (single stream): Streaming continuity: 100% → 100% (already good) RTF: 0.07 → 0.07 (neutral) All requests HTTP 200, no errors. Backward compatible (no ramp = original behavior). Signed-off-by: jingchengtian <tjc1995@126.com>
MOSS-TTS Local has the same chunk-size cliff as Qwen3-TTS: chunk 0 emits 1 frame (80ms), then jumps directly to 15-frame steady chunks, causing an audible silence gap in streaming output. This PR reuses the architecture-level chunk ramp API (chunk_size_utils.py) already merged for Qwen3-TTS (vllm-project#5152 fixed ramp, vllm-project#6001 adaptive controller) and wires it into MOSS-TTS's talker2codec_raw_async_chunk processor. Changes: - moss_tts.py: Add ramp/adaptive branches to talker2codec_raw_async_chunk. - parse_chunk_ramp / parse_adaptive_config parsed once per transfer_manager - Ramp: uses compute_ramp_emit with cumulative frame tracking - Adaptive: uses AdaptiveChunkController + compute_adaptive_emit - Original IC/steady threshold logic preserved when neither is configured - Track chunk_index and cumulative emitted frames per request for ramp math - serving_speech.py: Set tts_local_seed from deploy config default_seed for MOSS-TTS variants (moss_tts/moss_tts_realtime/moss_tts_local), matching the existing Qwen3-TTS pattern. - moss_tts_local.yaml: Document codec_chunk_ramp / codec_chunk_adaptive config options (commented out by default for backward compat) Reuses chunk_size_utils.py (parse_chunk_ramp, compute_ramp_emit, AdaptiveChunkController, compute_adaptive_emit, parse_adaptive_config) shared with Qwen3-TTS — no new utility code, model-agnostic API. Benchmark (2-NPU, Ascend 910B2C, 5 rounds, R1 discarded, median R2-R5): C=8 (concurrency 8): Streaming continuity: 10% → 100% (+90pp) RTF: 0.19 → 0.36 (still <1.0, real-time) E2EL: 2980ms → 2989ms (neutral) C=1 (single stream): Streaming continuity: 100% → 100% (already good) RTF: 0.07 → 0.07 (neutral) All requests HTTP 200, no errors. Backward compatible (no ramp = original behavior). Signed-off-by: jingchengtian <tjc1995@126.com>
MOSS-TTS Local has the same chunk-size cliff as Qwen3-TTS: chunk 0 emits 1 frame (80ms), then jumps directly to 15-frame steady chunks, causing an audible silence gap in streaming output. This PR reuses the architecture-level chunk ramp API (chunk_size_utils.py) already merged for Qwen3-TTS (vllm-project#5152 fixed ramp, vllm-project#6001 adaptive controller) and wires it into MOSS-TTS's talker2codec_raw_async_chunk processor. Changes: - moss_tts.py: Add ramp/adaptive branches to talker2codec_raw_async_chunk. - parse_chunk_ramp / parse_adaptive_config parsed once per transfer_manager - Ramp: uses compute_ramp_emit with cumulative frame tracking - Adaptive: uses AdaptiveChunkController + compute_adaptive_emit - Original IC/steady threshold logic preserved when neither is configured - Track chunk_index and cumulative emitted frames per request for ramp math - serving_speech.py: Set tts_local_seed from deploy config default_seed for MOSS-TTS variants (moss_tts/moss_tts_realtime/moss_tts_local), matching the existing Qwen3-TTS pattern. - moss_tts_local.yaml: Document codec_chunk_ramp / codec_chunk_adaptive config options (commented out by default for backward compat) Reuses chunk_size_utils.py (parse_chunk_ramp, compute_ramp_emit, AdaptiveChunkController, compute_adaptive_emit, parse_adaptive_config) shared with Qwen3-TTS — no new utility code, model-agnostic API. Benchmark (2-NPU, Ascend 910B2C, 5 rounds, R1 discarded, median R2-R5): C=8 (concurrency 8): Streaming continuity: 10% → 100% (+90pp) RTF: 0.19 → 0.36 (still <1.0, real-time) E2EL: 2980ms → 2989ms (neutral) C=1 (single stream): Streaming continuity: 100% → 100% (already good) RTF: 0.07 → 0.07 (neutral) All requests HTTP 200, no errors. Backward compatible (no ramp = original behavior). Signed-off-by: jingchengtian <tjc1995@126.com>
MOSS-TTS Local has the same chunk-size cliff as Qwen3-TTS: chunk 0 emits 1 frame (80ms), then jumps directly to 15-frame steady chunks, causing an audible silence gap in streaming output. This PR reuses the architecture-level chunk ramp API (chunk_size_utils.py) already merged for Qwen3-TTS (vllm-project#5152 fixed ramp, vllm-project#6001 adaptive controller) and wires it into MOSS-TTS's talker2codec_raw_async_chunk processor. Changes: - moss_tts.py: Add ramp/adaptive branches to talker2codec_raw_async_chunk. - parse_chunk_ramp / parse_adaptive_config parsed once per transfer_manager - Ramp: uses compute_ramp_emit with cumulative frame tracking - Adaptive: uses AdaptiveChunkController + compute_adaptive_emit - Original IC/steady threshold logic preserved when neither is configured - Track chunk_index and cumulative emitted frames per request for ramp math - serving_speech.py: Set tts_local_seed from deploy config default_seed for MOSS-TTS variants (moss_tts/moss_tts_realtime/moss_tts_local), matching the existing Qwen3-TTS pattern. - moss_tts_local.yaml: Document codec_chunk_ramp / codec_chunk_adaptive config options (commented out by default for backward compat) Reuses chunk_size_utils.py (parse_chunk_ramp, compute_ramp_emit, AdaptiveChunkController, compute_adaptive_emit, parse_adaptive_config) shared with Qwen3-TTS — no new utility code, model-agnostic API. Benchmark (2-NPU, Ascend 910B2C, 5 rounds, R1 discarded, median R2-R5): C=8 (concurrency 8): Streaming continuity: 10% → 100% (+90pp) RTF: 0.19 → 0.36 (still <1.0, real-time) E2EL: 2980ms → 2989ms (neutral) C=1 (single stream): Streaming continuity: 100% → 100% (already good) RTF: 0.07 → 0.07 (neutral) All requests HTTP 200, no errors. Backward compatible (no ramp = original behavior). Signed-off-by: jingchengtian <tjc1995@126.com>
MOSS-TTS Local has the same chunk-size cliff as Qwen3-TTS: chunk 0 emits 1 frame (80ms), then jumps directly to 15-frame steady chunks, causing an audible silence gap in streaming output. This PR reuses the architecture-level chunk ramp API (chunk_size_utils.py) already merged for Qwen3-TTS (vllm-project#5152 fixed ramp, vllm-project#6001 adaptive controller) and wires it into MOSS-TTS's talker2codec_raw_async_chunk processor. Changes: - moss_tts.py: Add ramp/adaptive branches to talker2codec_raw_async_chunk. - parse_chunk_ramp / parse_adaptive_config parsed once per transfer_manager - Ramp: uses compute_ramp_emit with cumulative frame tracking - Adaptive: uses AdaptiveChunkController + compute_adaptive_emit - Original IC/steady threshold logic preserved when neither is configured - Track chunk_index and cumulative emitted frames per request for ramp math - serving_speech.py: Set tts_local_seed from deploy config default_seed for MOSS-TTS variants (moss_tts/moss_tts_realtime/moss_tts_local), matching the existing Qwen3-TTS pattern. - moss_tts_local.yaml: Document codec_chunk_ramp / codec_chunk_adaptive config options (commented out by default for backward compat) Reuses chunk_size_utils.py (parse_chunk_ramp, compute_ramp_emit, AdaptiveChunkController, compute_adaptive_emit, parse_adaptive_config) shared with Qwen3-TTS — no new utility code, model-agnostic API. Benchmark (2-NPU, Ascend 910B2C, 5 rounds, R1 discarded, median R2-R5): C=8 (concurrency 8): Streaming continuity: 10% → 100% (+90pp) RTF: 0.19 → 0.36 (still <1.0, real-time) E2EL: 2980ms → 2989ms (neutral) C=1 (single stream): Streaming continuity: 100% → 100% (already good) RTF: 0.07 → 0.07 (neutral) All requests HTTP 200, no errors. Backward compatible (no ramp = original behavior). Signed-off-by: jingchengtian <tjc1995@126.com>
MOSS-TTS Local has the same chunk-size cliff as Qwen3-TTS: chunk 0 emits 1 frame (80ms), then jumps directly to 15-frame steady chunks, causing an audible silence gap in streaming output at concurrency > 1. This PR reuses the architecture-level chunk ramp API (chunk_size_utils.py) already merged for Qwen3-TTS (vllm-project#5152 fixed ramp, vllm-project#6001 adaptive controller) and wires it into MOSS-TTS's talker2codec_raw_async_chunk processor. Changes: - moss_tts.py: Add ramp/adaptive branches to talker2codec_raw_async_chunk. - parse_chunk_ramp / parse_adaptive_config parsed once per transfer_manager - Ramp: uses compute_ramp_emit with cumulative frame tracking - Adaptive: uses AdaptiveChunkController + compute_adaptive_emit - Original IC/steady threshold logic preserved when neither is configured - ramp_chunk_count is read-only (owned by adapter, not processor) - emit_frames clamped to actual pending to handle async overshoot - _ramp_total_emitted tracks real emitted count, cleaned up on finish - chunk_transfer_adapter.py: Add _ramp_total_emitted to adapter state and clean up in _clear_sender_state_locked and segment-finished paths - serving_speech.py: Set tts_local_seed from deploy config default_seed for MOSS-TTS variants, matching the existing Qwen3-TTS pattern. - moss_tts_local.yaml: Document codec_chunk_ramp / codec_chunk_adaptive config options (commented out by default for backward compat) - test_moss_tts_async_chunk.py: 13 CPU tests covering default/ramp/adaptive paths, finish flush, cleanup, seed fallback, and adapter-owned counter Benchmark (2-NPU, Ascend 910B2C, 5 rounds, R1 discarded, median R2-R5): C=8: Streaming continuity 10% → 100% (+90pp), RTF 0.19 → 0.36 (<1.0) C=1: Streaming continuity 100% → 100% (neutral) All requests HTTP 200, no errors. Backward compatible. Signed-off-by: jingchengtian <tjc1995@126.com>
…r) (vllm-project#6001) Signed-off-by: Wallbreazzz <110282866+Wallbreazzz@users.noreply.github.com>
…S-TTS Rebased onto latest main after vllm-project#8124 (static chunk ramp) and vllm-project#8213 (MOSS Local 1.5 MRV2 serving) were merged. The static ramp portion of the original PR is now redundant — vllm-project#8124 already landed it using parse_chunk_ramp / ramp_chunk_size. This rebased commit keeps only the two features not yet on main: 1. Adaptive chunk controller (codec_chunk_adaptive): - moss_tts.py: Add adaptive branch to talker2codec_raw_async_chunk. When codec_chunk_adaptive is set, AdaptiveChunkController + compute_adaptive_emit from chunk_size_utils.py (introduced by Qwen3-TTS vllm-project#6001) replace the static ramp/IC-steady threshold. Chunk 0 uses IC/steady (same as original); chunk 1+ uses the controller's buffer-feedback target_size. Adaptive takes precedence over static ramp when both are configured. The ramp threshold override is moved after the adaptive branch so it only applies when adaptive is off. _adaptive_states cleanup is owned by the adapter (chunk_transfer_adapter already pops it on segment-finish and abort on main). - moss_tts_local_v1.yaml: Document codec_chunk_adaptive (commented). 2. MOSS-TTS seed fallback (tts_local_seed): - tts_adapters/moss_tts.py: In apply_sampling_overrides, propagate the deploy config's default_sampling_params.seed to tts_local_seed (matching Qwen3-TTS pattern). Shallow-copies stage 0 params so shared defaults stay immutable. Tests: - test_moss_tts_async_chunk.py: 7 new adaptive tests (chunk 0 IC threshold, finish flush, empty-finish sentinel, adaptive-over-ramp precedence, independent requests, states lifecycle). Existing static-ramp tests unchanged. - test_serving_speech.py: MOSS seed-fallback test via _prepare_speech_generation (entrypoints-level, not model_executor). All 23 tests pass (16 existing + 7 new). Signed-off-by: jingchengtian <tjc1995@126.com>
5 participants
Summary
Phase 2 of the streaming-continuity work scoped in #3535 (design on #4855). Builds on the merged Phase 1 fixed-ramp #5152. Replaces the fixed
codec_chunk_ramptable with a per-request EWMA-scaled adaptive controller that self-tunes chunk size across hardware (NPU/NVIDIA) and load, generalizing the dynamic IC machinery that already existed for chunk 0.Why adaptive (vs. Phase 1 fixed ramp)
The Phase 1 fixed ramp (
codec_chunk_ramp: [4, 4, 8, 16, 25]) eliminates the chunk-0→1 cliff on a specific hardware target but requires manual tuning per deploy. On 11 ms/frame hardware (L20X) the Ascend[4, 4, 8, 16, 25]table is suboptimal — the controller self-tunes to[2, 4, 25, 25, 3]with zero config changes. Portability across hardware without manual ramp-table tuning is the core value of this PR.Design (A+B hybrid, per @linyueqian 2026-07-19 on #4855)
buffer_ms = total_audio_ms_emitted - (now - t_first_emit);ramp_delta = max(ramp_delta_min, int(ewma/ramp_divisor))is unconditional (never gated on buffer sign); hysteresis locks tocodec_chunk_framesonce buffer holds one steady chunk of headroom.[min_frames, codec_chunk_frames].Config (connector extra)
codec_chunk_adaptive(opt-in)codec_chunk_min_frames(default 2) — absolute size floor (greedy path)codec_chunk_safety_margin_ms(default 50) — Code2Wav decode+transfer margincodec_chunk_ramp_divisor(default 15) — EWMA→delta divisor; higher=gentler rampcodec_chunk_ramp_delta_min(default 2) — min chunk-size growth per emitcodec_chunk_frames(no separate max config — capping below steady is a death-spiral footgun; aligns with linyueqian's "clamp to [ic_min, codec_chunk_frames]")Invariants
ramp_floorunconditional. Gating onbuffer>0collapses tomin_framesunder load and triggers a chunk-count spiral — each small chunk still incurs per-chunk overhead (transport, batch padding, graph replay setup) even with W5's incremental decode. Verified empirically on pre-W5 code: a gated variant produced[2,2,2,...]and exploded RTF at c=8 NPU.compute_adaptive_emitships ALL accumulated frames (new_frames), not justtarget_size— prevents the[-context_length:]slice from skipping surplus frames when a latency spike drops target below queued frames (B1 fix per @linyueqian's suggestion).last_target = target_size(the controller's intended target), NOTn_frames(the actual emitted count). When a latency spike causesnew_framesto overshoottarget_size, usingn_framesforlast_targetwould ratchetramp_floorto the spike size and lock the controller towardmax_framesprematurely.target_sizeis passed separately fromn_framesinrecord_emitto keep the ramp floor stable through transient overshoot.skip_dynamic_icincludesramp is not None— fixed-ramp deployments now skip dynamic IC where previously they did not (behavior change: the ramp branch never readsinitial_chunk_size, so the previous O(active) scan + cache entry was wasted).finishedflushes ALL remaining frames (no truncation / data loss)._adaptive_states.pop().time.monotonic()+ integer arithmetic, no new device syncs, out of the worker step loop.W5 status (#5202 merged)
W5 (Code2Wav streaming decoder state, #5202) has merged into main. Code2Wav now carries conv/attention state across chunks via incremental decode, eliminating the ~3.9x redundant context re-decode that motivated the original ramp-up-only design. The unconditional
ramp_floorand hysteresis lock remain safe defaults — small chunks still incur per-chunk overhead (transport, batch padding, graph replay setup), and the death-spiral risk under sustained overload persists even with W5 — but the per-frame decode penalty that drove the original ~10x cost ratio is now gone. Downward chunk-size adaptation (shrinking chunks when buffer is healthy, growing when under pressure) is now feasible as a follow-up PR; this PR keeps the conservative ramp-up-only approach.Per-request telemetry (INFO, one line per segment at finish)
Mirrors
stream_client.py's playback-cursor underrun model so server-sidetotal_gapis comparable to the client-reportedAUDIO_UNDERRUN:Note:
finished=is_segment_finished or is_finished(seechunk_transfer_adapter), so a resumable/v1/realtimesession emits one line per segment.Validation
Ascend 910B NPU (vllm 0.27.1 + vllm-ascend PR #14027 + vllm-omni main)
c=8, voice_clone, 40 requests, same text/hardware/model across all three arms; feature is opt-in so config is the only variable.
The adaptive controller shows clear continuity and underrun advantages over both the default and the Phase 1 fixed ramp on Ascend 910B at c=8:
L20X (masked H200), vllm 0.26.0 (from @linyueqian's Round 2 review)
On 11 ms/frame hardware the controller self-tunes to
[2, 4, 25, 25, 3]with no config changes from the Ascend[4, 4, 8, 16, 25]. Portability is the pitch.Audio quality verified by listening (complete, clear, no artifacts).
Status
Ready for re-review. Rebased onto latest main (vllm 0.27). All Round 2 review issues addressed (B1 fixed per @linyueqian's suggestion, B2 regression test, S1-S5 cleanup, S6 PR body). W5 (#5202) already merged — ramp-up-only approach is conservative default; downward adaptation is a follow-up.
Refs: #3535 #4855 #5152 #4913 #5202.