[Feat][Qwen3TTS] reduce TTFA with flexible initial phase

[JuanPZuluaga]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Related to #938.

In this PR, we lower the TTFA of Qwen3TTS by having a smaller number of frames required to start to stream the output. Ideally, this would help to perceive a faster response of the whole system by reducing the TTFA.

The current implementation is not too flexible, as if one wants to reduce the TTFA, we would end up always decoding with very small chunks, which can reduce audio quality.

• what we add is initial_chunk_size which dictates the chunk rate used to generate audio. we can call this phase "warmup phase": initial_chunk_size + full left context
• then, after we have collected enough frames, we move to the standard decoding phase: chunk_size + left_context_size

PS: i decided to not add support of left_context_size=-1 (aka full left context), cause we might get OOM in very long sequences (too large context), though the user can just increase left context, to let's say 10s.

I will add some audio samples later.

Test Plan

Test Result

• benchmark of sending to 10 prompts to the model
• I tested: initial_codec_chunk_frames = 0, 2, 5, 10, 15, 20" ; 0 means no initial_codec_chunk_frames
• I ran the e2e script and added a timer in between chunks and print that:
• ic means, vlaue set for initial_codec_chunk_frames
• first req always takes more time, probably due to compilation?

TTFA (Time To First Audio) in milliseconds

Config	Req 1	Req 2	Req 3	Req 4	Req 5	Req 6	Req 7	Req 8	Req 9	Avg	Min	Max
ic_0	2092	2099	2263	1835	2180	2097	1879	1870	1952	2030	1835	2263
ic_2	253	196	197	178	191	184	216	215	220	205	178	253
ic_5	489	403	401	417	471	383	412	522	436	437	383	522
ic_10	796	790	737	732	906	859	777	767	837	800	732	906
ic_15	1283	1398	1209	1122	1274	1166	1227	1090	1108	1209	1090	1398
ic_20	1641	1673	1632	1461	1324	1309	1285	1327	1274	1436	1274	1673

Total Generation Time (ms)

Config	Req 1	Req 2	Req 3	Req 4	Req 5	Req 6	Req 7	Req 8	Req 9	Avg
ic_0	5395	5809	5806	6054	5894	6013	5458	7293	6296	6002
ic_2	4886	5486	4724	5164	4886	5503	5500	7974	6725	5650
ic_5	4976	5256	5072	5256	5073	5340	5892	7141	6213	5580
ic_10	4720	6344	5287	5815	5488	5817	5416	6778	6310	5775
ic_15	5047	5777	5049	5421	5336	5741	5931	6998	6078	5709
ic_20	4880	6340	5103	4805	4372	4663	4578	6086	4855	5076

Inter-Chunk Time (ms)

Config	Avg	Min	Max	Std	Chunks
ic_0	2025	1733	2271	173	11
ic_2	328	124	2245	532	110
ic_5	714	305	2058	634	47
ic_10	1522	702	2294	596	25
ic_15	1901	1706	2100	111	18
ic_20	1664	1503	2173	217	16

output_1_ic_0.wav
output_1_ic_2.wav
output_1_ic_5.wav
output_1_ic_10.wav
output_1_ic_15.wav
output_1_ic_20.wav

Note that ic_15 and ic_20, would yield similar results.

EDIT: removed the first sample (in the tables) due to overhead in compile.
EDIT2: updated the numbers with a fresh run.

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[amy-why-3459]

Can we reduce TTFP by adjusting chunk_size?

[JuanPZuluaga]

Can we reduce TTFP by adjusting chunk_size?

Indeed, but then:

• lower chunk size, means more calls to the code2wav model. Let’s say we go from 25 (current default) to 5, we will end up having 5x more calls
• we could play with increasing the left context to give more overall context, and account for the lower chunk size; but the pain point would still be there; TTFA would be high unless we go to reduce chunk size to very low values.

With this approach, at least, we know that the TTFA can go below 500ms, without compromising too much the quality and not too much overhead. We could even increase the chunk size to higher values to have better audio quality without increasing TTFA.

[JuanPZuluaga]

please let me know what you think @amy-why-3459 @linyueqian. TTFC can go to below ~300ms

Also, should i add some test?

[amy-why-3459]

Thank you so much for your contribution, it's a great idea. May I ask what your test scenario is? For example, concurrency and input/output length?

[JuanPZuluaga]

Do you mean a real use case? If it's that, ideally, we would like to reduce TTFC on high concurrency loads for voice assistants, where we need very low latency for generating the first audio to the user. For instance, in batched offline decoding scenarios, i wouldn't see much importance to this value.

[Sy0307]

It makes sense to me that there is initial_codec_chunk_frames in the warm-up stage to reduce TTFA, and we need a similar scenario as well.

Additionally, I have a question: is it better to implement initial_codec_chunk_frames as a stage-level configuration or a request-level configuration? Or could both exist, with the request-level config taking higher priority? Does this make sense? Welcome to discuss.

[JuanPZuluaga]

It makes sense to me that there is initial_codec_chunk_frames in the warm-up stage to reduce TTFA, and we need a similar scenario as well.

Additionally, I have a question: is it better to implement initial_codec_chunk_frames as a stage-level configuration or a request-level configuration? Or could both exist, with the request-level config taking higher priority? Does this make sense? Welcome to discuss.

this is a good idea actually, though it means adding yet more params to be set request-level. Let me know what you think, and we can implement it.

[amy-why-3459]

I'd like to ask about the test scenarios in your Test Result. Also, could you please adapt this solution to qwen3-omni and mimo_audio as well?

[hsliuustc0106]

what are the definitions of TTFA/TTFP?

[JuanPZuluaga]

what are the definitions of TTFA/TTFP?

Sorry about that i've mixed both, but they mean the same: Time to First Audio or Time to first Packet. So far, i hear glitches while doing "live streaming" due to the fact that the model cannot keep with real time processing (meaning RTFx is less than one). I expect it to be fast once we get the different models compiled, etc.

[linyueqian]

Thanks for this feature! The TTFA improvement at ic=2 is really impressive (~10x reduction with negligible total latency overhead).

I had a question about the warmup to normal phase transition for longer responses. Correct me if my understanding is wrong:

At ic=2, each warmup chunk delivers 2 × 40ms = 80ms of audio. By the time all warmup chunks are delivered, the player has received a total of 12 × 80ms = 960ms of audio. But from that point, the first normal-phase chunk takes ~1593ms to generate (matching the ic_0 baseline). Since the total accumulated warmup audio (960ms) is less than the normal chunk generation time (1593ms), would the player run out of audio before the first normal chunk arrives? I noticed ic_2 has max inter-chunk = 1614ms and std = 430ms. Is that spike the transition point?

I suspect the benchmark avoids this because the test prompts are short enough (~25 frames, roughly 1 second) that the entire response is served within the initial phase and the normal phase is never entered. Would it be possible to test with a longer prompt (say, 5 to 10 seconds of output) to confirm?

Also a small terminology question: the codebase already uses "warmup" in a few other contexts (model warmup, benchmark warmup requests, cache-dit warmup steps). Would a different name like "startup phase" or "initial-chunk phase" avoid ambiguity? Happy to defer to whatever the team prefers!

[JuanPZuluaga]

Thanks for this feature! The TTFA improvement at ic=2 is really impressive (~10x reduction with negligible total latency overhead).

Indeed! It was very surprising to me, that with only 2 chunks, the performance is so good! I did a signal processing analysis, and the difference is minimal. I'd put the script but it's too long...

I had a question about the warmup to normal phase transition for longer responses. Correct me if my understanding is wrong:

At ic=2, each warmup chunk delivers 2 × 40ms = 80ms of audio. By the time all warmup chunks are delivered, the player has received a total of 12 × 80ms = 960ms of audio. But from that point, the first normal-phase chunk takes ~1593ms to generate (matching the ic_0 baseline). Since the total accumulated warmup audio (960ms) is less than the normal chunk generation time (1593ms), would the player run out of audio before the first normal chunk arrives? I noticed ic_2 has max inter-chunk = 1614ms and std = 430ms. Is that spike the transition point?

you are correct, i can notice the glitches in the first startup phase when the ic is very low (in reality the audio doesn't have glitches, it's only the perceived audio, kind of getting cut, it only happens during the startup phase though). I think with the optimizations that are planned (cuda graph in the speech tokenizer, etc), this issue will be gone. also, perhaps there's some hyper-param that needs to be tuned that i am missing.

In any case, if the current functionality is supported, further future optimizations already planned will allow us to use very low ic.

I suspect the benchmark avoids this because the test prompts are short enough (~25 frames, roughly 1 second) that the entire response is served within the initial phase and the normal phase is never entered. Would it be possible to test with a longer prompt (say, 5 to 10 seconds of output) to confirm?

I will try with longer sequences, but in my erlier experiments, i didn't see any issue TBH.

Also a small terminology question: the codebase already uses "warmup" in a few other contexts (model warmup, benchmark warmup requests, cache-dit warmup steps). Would a different name like "startup phase" or "initial-chunk phase" avoid ambiguity? Happy to defer to whatever the team prefers!

Startup phase sounds good! I'll change this! thanks for the feedback.

[JuanPZuluaga]

I'd like to ask about the test scenarios in your Test Result. Also, could you please adapt this solution to qwen3-omni and mimo_audio as well?

@amy-why-3459 we would need to wait for #1423 in order to add the same functionality in qwen3-omni

[Sy0307]

It makes sense to me that there is initial_codec_chunk_frames in the warm-up stage to reduce TTFA, and we need a similar scenario as well.
Additionally, I have a question: is it better to implement initial_codec_chunk_frames as a stage-level configuration or a request-level configuration? Or could both exist, with the request-level config taking higher priority? Does this make sense? Welcome to discuss.

this is a good idea actually, though it means adding yet more params to be set request-level. Let me know what you think, and we can implement it.

For example, the current configurations could be included as request-level parameters in something like additional_information. Additionally, we could specify the warmup duration in single request. For instance, we might allow the network protocol to dynamically adjust the chunk_size during warmup within a single request based on current network conditions, which would provide better real-time feedback. (By the way, I also agree that the term "warm_up" needs to be revised — it can be somewhat ambiguous.)

[JuanPZuluaga]

It makes sense to me that there is initial_codec_chunk_frames in the warm-up stage to reduce TTFA, and we need a similar scenario as well.
Additionally, I have a question: is it better to implement initial_codec_chunk_frames as a stage-level configuration or a request-level configuration? Or could both exist, with the request-level config taking higher priority? Does this make sense? Welcome to discuss.

this is a good idea actually, though it means adding yet more params to be set request-level. Let me know what you think, and we can implement it.

For example, the current configurations could be included as request-level parameters in something like additional_information. Additionally, we could specify the warmup duration in single request. For instance, we might allow the network protocol to dynamically adjust the chunk_size during warmup within a single request based on current network conditions, which would provide better real-time feedback. (By the way, I also agree that the term "warm_up" needs to be revised — it can be somewhat ambiguous.)

this sounds cool, is it worth doing it? I can add the variable chunk size that goes into additional_information.

@amy-why-3459 @linyueqian i changed warmup to "initial phase" everywhere. Please let me know which are other things to try, so i can try, otherwise, we could check for merge?

[JuanPZuluaga]

Ok, updated with configurable initial_codec_chunk_frames that can be set in the request. the rest is the same. @linyueqian @Sy0307

Test results below

[JuanPZuluaga]

benchmark overriding initial-codec-chunk-frames via the request.

Benchmark on:

NVIDIA RTX 6000 Ada 48GB.

step 1: run model serve:

CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main serve \
    "Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice" \
    --omni --host 127.0.0.1 --port 8000 \
    --stage-configs-path benchmarks/qwen3-tts/vllm_omni/configs/qwen3_tts_bs4.yaml \
    --trust-remote-code

note this is the default config, the one in main already.

step 2: run

python vllm_omni/bench_tts_serve.py \
    --port 8000 \
    --num-prompts 50 \
    --max-concurrency 1 4 10 \
    --config-name "async_chunk" \
    --result-dir results

(my local vllm_omni/bench_tts_serve.py includes initial-codec-chunk-frames param).

[hsliuustc0106]

benchmark overriding initial-codec-chunk-frames via the request.

Benchmark on:

NVIDIA RTX 6000 Ada 48GB.

step 1: run model serve:

CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main serve \
    "Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice" \
    --omni --host 127.0.0.1 --port 8000 \
    --stage-configs-path benchmarks/qwen3-tts/vllm_omni/configs/qwen3_tts_bs4.yaml \
    --trust-remote-code

note this is the default config, the one in main already.

step 2: run

python vllm_omni/bench_tts_serve.py \
    --port 8000 \
    --num-prompts 50 \
    --max-concurrency 1 4 10 \
    --config-name "async_chunk" \
    --result-dir results

(my local vllm_omni/bench_tts_serve.py includes initial-codec-chunk-frames param).

why RTF becomes worse in this scenario?

[JuanPZuluaga]

I think they're very similar @hsliuustc0106. Could you please point out to which experiment specifically?

[hsliuustc0106]

I think they're very similar @hsliuustc0106. Could you please point out to which experiment specifically?

in the first figure, concurrency at 1/4, the rtf looks higher when IC=2

[JuanPZuluaga]

I think they're very similar @hsliuustc0106. Could you please point out to which experiment specifically?

in the first figure, concurrency at 1/4, the rtf looks higher when IC=2

Oh you’re correct. In this setting, probably the small RTF overhead is due to doing 12 forward passes of the code2wav (of ic2, 2x12=24 chunks) instead of 1 forward passes in ic0, which does it once we have cached 25 chunks.

[lishunyang12]

see inline

[lishunyang12]

Where does put_req_chunk get incremented after a chunk is emitted? This function reads it but I don't see the write side in this diff.

[JuanPZuluaga]

I think it is done already in:

vllm-omni/vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py

Lines 231 to 235 in 1612948

	if success:
	self.put_req_chunk[request_id] += 1
	logger.debug(f"[Stage-{stage_id}] Sent {connector_put_key}")

specifically in ChunkTransferAdapter._send_single_request().

[hsliuustc0106]

lgtm

[Sy0307]

lgtm. BTW I believe this should probably become a general method that can be used to reduce TTFA/TTFV for all models with high real-time requirements.

[JuanPZuluaga]

lgtm. BTW I believe this should probably become a general method that can be used to reduce TTFA/TTFV for all models with high real-time requirements.

Thanks! yeah, which models do you think we should target? qwen3-omni is down in line. I can work on it. Also, I think we should implement a smarter way to select this chunk.

[hsliuustc0106]

lgtm. BTW I believe this should probably become a general method that can be used to reduce TTFA/TTFV for all models with high real-time requirements.

Thanks! yeah, which models do you think we should target? qwen3-omni is down in line. I can work on it. Also, I think we should implement a smarter way to select this chunk.

qwen3.5-omni is on the way :)

[hsliuustc0106]

lgtm. BTW I believe this should probably become a general method that can be used to reduce TTFA/TTFV for all models with high real-time requirements.

Thanks! yeah, which models do you think we should target? qwen3-omni is down in line. I can work on it. Also, I think we should implement a smarter way to select this chunk.

we should provide an elegant abstraction for this dynamic IC selection

[JuanPZuluaga]

lgtm. BTW I believe this should probably become a general method that can be used to reduce TTFA/TTFV for all models with high real-time requirements.

Thanks! yeah, which models do you think we should target? qwen3-omni is down in line. I can work on it. Also, I think we should implement a smarter way to select this chunk.

we should provide an elegant abstraction for this dynamic IC selection

I’ll brainstorm a bit, and come with an useful abstraction based on the load in the ‘code2wav’ block.

[mkgs210]

When I use initial_codec_chunk_frames < 25 I always have 1s latency in the middle of every generation
chunk=1 t=0.360s size=7680
...
chunk=11 t=1.894s size=7680
chunk=12 t=2.054s size=7680
chunk=13 t=3.496s size=8689
chunk=14 t=3.604s size=16384
...
chunk=22 t=7.773s size=18182
done total=8.250s bytes=318720

It hapens with concurrency 4 or more on 3090/A100 and 6 or more on 4090
rft is good (>1.5-2)
I use qwen3_tts_bs16.yaml

[linyueqian]

@JuanPZuluaga ptal

When I use initial_codec_chunk_frames < 25 I always have 1s latency in the middle of every generation chunk=1 t=0.360s size=7680 ... chunk=11 t=1.894s size=7680 chunk=12 t=2.054s size=7680 chunk=13 t=3.496s size=8689 chunk=14 t=3.604s size=16384 ... chunk=22 t=7.773s size=18182 done total=8.250s bytes=318720

It hapens with concurrency 4 or more on 3090/A100 and 6 or more on 4090 rft is good (>1.5-2) I use qwen3_tts_bs16.yaml