Skip to content

[Voxtral TTS] Support voxtral tts cfg_alpha sampling params via temperature#2243

Closed
y123456y78 wants to merge 20 commits into
vllm-project:mainfrom
y123456y78:chenyo/voxtral-tts-sampling-params
Closed

[Voxtral TTS] Support voxtral tts cfg_alpha sampling params via temperature#2243
y123456y78 wants to merge 20 commits into
vllm-project:mainfrom
y123456y78:chenyo/voxtral-tts-sampling-params

Conversation

@y123456y78
Copy link
Copy Markdown
Contributor

@y123456y78 y123456y78 commented Mar 26, 2026 •
edited
Loading

Purpose

  • Support voxtral tts cfg_alpha sampling params via temperature
  • Add corresponding unit test
  • Clean up cuda graph & cuda graph test code a bit

Testing Plan

pytest -s -v \
  tests/model_executor/stage_input_processors/test_voxtral_tts_async_chunk.py \
  tests/model_executor/models/voxtral_tts/test_cuda_graph_acoustic_transformer.py \
  tests/model_executor/models/voxtral_tts/test_audio_tokenizer_parsing.py \
  tests/e2e/online_serving/test_voxtral_tts.py \
  tests/model_executor/models/voxtral_tts/test_text_preprocess.py \
  tests/e2e/offline_inference/test_voxtral_tts.py

Result

Uploading image.png…

@y123456y78
Support voxtral tts cfg_alpha sampling params in eager mode
03f3d9b 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78 y123456y78 force-pushed the chenyo/voxtral-tts-sampling-params branch from ea989bd to 03f3d9b Compare March 26, 2026 22:57
y123456y78 added 15 commits March 26, 2026 23:11
@y123456y78
Support cuda graph
dcb0466 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
Update help message
f16176e 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
Supprot cfg_alpha in gradio_demo
ddd4939 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
Update test to support cfg_alpha
ad68fff 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
Merge branch 'main' into chenyo/voxtral-tts-sampling-params
c0cfb59
@y123456y78
add test code
6ad25bd 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
clean up
59f0036 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
clean up
966a821 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
Update test
7f93b24 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
Align variable name for test and actual model
c694424 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
Clean up and add test
8c04ea6 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
Set temperature in end2end.py
b7d62d1 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
Cleanup log
e899b90 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
Clean up duplicate
8140668 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
Fix format
f98a647 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78 y123456y78 changed the title [Voxtral TTS] Support voxtral tts cfg_alpha sampling params [Voxtral TTS] Support voxtral tts cfg_alpha sampling params via temperature Mar 27, 2026
@y123456y78
Fix test
1470b0b 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
Revert code cleanup
bb7c395 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
Fix
80689c3 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@y123456y78
Fix format
40d2163 
Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>
@linyueqian linyueqian mentioned this pull request Mar 30, 2026
Merged
lishunyang12
lishunyang12 reviewed Apr 2, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@lishunyang12 lishunyang12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left a couple comments

Comment thread vllm_omni/model_executor/models/voxtral_tts/voxtral_tts.py
sampling_metadata = kwargs.get("sampling_metadata")
if sampling_metadata is None or sampling_metadata.temperature is None:
raise ValueError(
"VoxtralTTS requires a non-zero 'temperature' sampling parameter (used as cfg_alpha for flow-matching)."
Copy link
Copy Markdown
Collaborator

@lishunyang12 lishunyang12 Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will silently accept temperature=0, which would zero out the conditional velocity entirely (pure unconditional generation). Should probably validate temperature > 0 here, or at least != 0.

Comment thread vllm_omni/model_executor/stage_configs/voxtral_tts.yaml
final_output_type: text
default_sampling_params:
temperature: 0.0
# NOTE: VoxtralTTS repurposes 'temperature' as the CFG alpha
Copy link
Copy Markdown
Collaborator

@lishunyang12 lishunyang12 Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: might be worth adding a user-facing note somewhere (CLI help, docs) that temperature controls CFG strength for voxtral-tts — otherwise people will set temperature=0.7 expecting normal sampling behavior and get confused.

Comment thread vllm_omni/model_executor/models/voxtral_tts/cuda_graph_acoustic_transformer_wrapper.py
padded_size = self._get_padded_size(actual_size)
if padded_size is None or padded_size not in self.graphs:
return self.model.compute_mm_logits(hidden_states)
return self.model.compute_mm_logits(hidden_states, cfg_alpha=cfg_alpha)
Copy link
Copy Markdown
Collaborator

@lishunyang12 lishunyang12 Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 1D -> 2D reshape (unsqueeze(1)) happens inside decode_one_frame, but in the graph path static_cfg_alpha is already (size, 1). This means the eager fallback via compute_mm_logits will unsqueeze, but the graph path skips it. Works today but the shape contract is fragile — a comment on the expected shape at this interface would help.

@y123456y78 y123456y78 marked this pull request as ready for review April 20, 2026 19:50
@chatgpt-codex-connector
Copy link
Copy Markdown

chatgpt-codex-connector Bot commented Apr 20, 2026

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@y123456y78 y123456y78 marked this pull request as draft April 20, 2026 19:50
@linyueqian linyueqian added this to the v0.20.0 milestone Apr 22, 2026
@linyueqian linyueqian added the ready label to trigger buildkite CI label Apr 22, 2026
@lishunyang12
Copy link
Copy Markdown
Collaborator

lishunyang12 commented Apr 23, 2026

Resolve conflicts.

@y123456y78 y123456y78 closed this Apr 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lishunyang12 lishunyang12 lishunyang12 left review comments

@hsliuustc0106 hsliuustc0106 Awaiting requested review from hsliuustc0106 hsliuustc0106 is a code owner

@linyueqian linyueqian Awaiting requested review from linyueqian linyueqian will be requested when the pull request is marked ready for review linyueqian is a code owner

@ZeldaHuang ZeldaHuang Awaiting requested review from ZeldaHuang ZeldaHuang will be requested when the pull request is marked ready for review ZeldaHuang is a code owner

@princepride princepride Awaiting requested review from princepride princepride will be requested when the pull request is marked ready for review princepride is a code owner

@yuanheng-zhao yuanheng-zhao Awaiting requested review from yuanheng-zhao yuanheng-zhao will be requested when the pull request is marked ready for review yuanheng-zhao is a code owner

@tzhouam tzhouam Awaiting requested review from tzhouam tzhouam will be requested when the pull request is marked ready for review tzhouam is a code owner

@gcanlin gcanlin Awaiting requested review from gcanlin gcanlin will be requested when the pull request is marked ready for review gcanlin is a code owner

@yenuo26 yenuo26 Awaiting requested review from yenuo26 yenuo26 will be requested when the pull request is marked ready for review yenuo26 is a code owner

@Gaohan123 Gaohan123 Awaiting requested review from Gaohan123 Gaohan123 will be requested when the pull request is marked ready for review Gaohan123 is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 will be requested when the pull request is marked ready for review ywang96 is a code owner

Assignees

No one assigned

Labels

ready label to trigger buildkite CI

Projects

None yet

Milestone

v0.20.0

Development

Successfully merging this pull request may close these issues.

3 participants

@y123456y78 @lishunyang12 @linyueqian