Skip to content

Support Fish Speech TTS inference on XPU platform - #4856

Merged
hsliuustc0106 merged 7 commits into
vllm-project:mainfrom
Liangyx2:FishSpeechSlowARForConditionalGeneration
Jul 18, 2026
Merged

Support Fish Speech TTS inference on XPU platform#4856
hsliuustc0106 merged 7 commits into
vllm-project:mainfrom
Liangyx2:FishSpeechSlowARForConditionalGeneration

Conversation

@Liangyx2

@Liangyx2 Liangyx2 commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Motivation

  • Fish Speech (FishSpeechSlowARForConditionalGeneration) model cannot run on Intel XPU due to CUDA-specific API calls and memory constraints.
  • The offline inference script lacks --deploy-config support, inconsistent with other model examples in the framework.
  • Streaming mode loses audio data due to DELTA output draining.

Technical details

  1. fish_kvcache_attn.py: Guard torch.cuda.is_current_stream_capturing() with torch.cuda.is_available() check to avoid RuntimeError on XPU where torch.cuda is a dummy module.
  2. fish_qwen3_omni.yaml: Add platforms.xpu overrides reducing max_model_len to 2048 and increasing gpu_memory_utilization to 0.8 for stage-0, fitting within XPU memory constraints when two stages share a single device.
  3. end2end.py: Add --deploy-config argument aligned with other offline inference scripts. Fix streaming mode to accumulate audio chunks across DELTA outputs instead of only saving the final (empty) chunk.

Performance impact

  • Enables Fish Speech S2 Pro end-to-end TTS on single Intel XPU (BMG/ARC).
  • No regression on CUDA path (original defaults unchanged).

Workload mapping

  • Fish Speech S2 Pro (fishaudio/s2-pro) text-to-speech on XPU.

Test Plan

  1. Quick start
    python examples/offline_inference/text_to_speech/fish_speech/end2end.py
    --deploy-config vllm_omni/deploy/fish_qwen3_omni.yaml
    --text "Hello, this is a test of the Fish Speech text to speech system."

  2. Voice cloning
    prompt.wav
    python examples/offline_inference/text_to_speech/fish_speech/end2end.py
    --deploy-config vllm_omni/deploy/fish_qwen3_omni.yaml
    --text "Hello, this is a cloned voice."
    --ref-audio prompt.wav
    --ref-text "testing my voices, why should I not?"

  3. Streaming
    python examples/offline_inference/text_to_speech/fish_speech/end2end.py
    --deploy-config vllm_omni/deploy/fish_qwen3_omni.yaml
    --text "Hello, this is a streaming test."
    --streaming

Test Result

  1. Quick start
    output_0_c4bf9699-8798-4ef2-beec-6073c522d709.wav
    pass_1.log

  2. Voice cloning
    output_0_658b1f39-a1f0-4a97-b3a9-ddeb4ab1984c.wav
    pass_2.log

  3. Streaming
    output_0.wav
    pass_3.log

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@Liangyx2
Liangyx2 force-pushed the FishSpeechSlowARForConditionalGeneration branch from dcab72b to f6a2232 Compare July 3, 2026 06:23
Liangyx2 added 2 commits July 3, 2026 14:25
@Liangyx2
Support Fish Speech TTS inference on XPU platform
f6a2232 
Motivation:
- Fish Speech (FishSpeechSlowARForConditionalGeneration) model cannot run
  on Intel XPU due to CUDA-specific API calls and memory constraints.
- The offline inference script lacks --deploy-config support, inconsistent
  with other model examples in the framework.
- Streaming mode loses audio data due to DELTA output draining.

Technical details:
1. fish_kvcache_attn.py: Guard torch.cuda.is_current_stream_capturing()
   with torch.cuda.is_available() check to avoid RuntimeError on XPU
   where torch.cuda is a dummy module.
2. fish_qwen3_omni.yaml: Add platforms.xpu overrides reducing
   max_model_len to 2048 and increasing gpu_memory_utilization to 0.8
   for stage-0, fitting within XPU memory constraints when two stages
   share a single device.
3. end2end.py: Add --deploy-config argument aligned with other offline
   inference scripts. Fix streaming mode to accumulate audio chunks
   across DELTA outputs instead of only saving the final (empty) chunk.

Performance impact:
- Enables Fish Speech S2 Pro end-to-end TTS on single Intel XPU (BMG/ARC).
- No regression on CUDA path (original defaults unchanged).

Workload mapping:
- Fish Speech S2 Pro (fishaudio/s2-pro) text-to-speech on XPU.

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update supported_models.md
aaf96fa 
Signed-off-by: Yuxiang Liang <yuxiang.liang@intel.com>
@Liangyx2
Liangyx2 requested a review from david6666666 as a code owner July 3, 2026 06:38
@hsliuustc0106

hsliuustc0106 commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

@xuechendi PTAL

@hsliuustc0106 hsliuustc0106 added xpu PR related to XPU tts code related to tts models labels Jul 3, 2026
xuechendi
xuechendi reviewed Jul 6, 2026
View reviewed changes
Comment thread examples/offline_inference/text_to_speech/fish_speech/end2end.py Outdated
xuechendi
xuechendi reviewed Jul 6, 2026
View reviewed changes
Comment thread examples/offline_inference/text_to_speech/fish_speech/end2end.py Outdated
xuechendi
xuechendi reviewed Jul 6, 2026
View reviewed changes
Comment thread examples/offline_inference/text_to_speech/fish_speech/end2end.py Outdated
xuechendi
xuechendi reviewed Jul 6, 2026
View reviewed changes

@xuechendi xuechendi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me, please clean up suggestions for e2e.py example and then we can set as ready

@Liangyx2

Liangyx2 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Overall looks good to me, please clean up suggestions for e2e.py example and then we can set as ready

Done.

@Liangyx2
Refactor Fish Speech end2end.py: extract helper functions to reduce d…
583c0fb 
…uplication

Extract _build_omni_kwargs() and _collect_audio_chunk() helpers to
eliminate duplicated logic between main() and main_streaming() paths.

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
xuechendi
xuechendi reviewed Jul 8, 2026
View reviewed changes
Comment thread vllm_omni/deploy/fish_qwen3_omni.yaml
@xuechendi

xuechendi commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

@Liangyx2 , please rerun example test locally and provide pass log.
Meanwhile, it would be nice to have comment to indicate validated xpu HW info in vllm_omni/deploy/fish_qwen3_omni.yaml

@Liangyx2
Add XPU HW validation note for Fish Speech deploy config
b290426 
Add comment noting XPU platform overrides are verified on Intel Arc B60
(BMG, 24GB, 800 EUs).

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2

Liangyx2 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

@Liangyx2 , please rerun example test locally and provide pass log. Meanwhile, it would be nice to have comment to indicate validated xpu HW info in vllm_omni/deploy/fish_qwen3_omni.yaml

Done. Updated output in Test Result.

xuechendi
xuechendi approved these changes Jul 9, 2026
View reviewed changes

@xuechendi xuechendi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xuechendi xuechendi added the ready label to trigger buildkite CI label Jul 9, 2026
@xuechendi
xuechendi enabled auto-merge (squash) July 9, 2026 16:39
@xuechendi
xuechendi disabled auto-merge July 9, 2026 20:18
@xuechendi

xuechendi commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

@hsliuustc0106 , PR looks good to me, PTAL

@Liangyx2
Trigger CI
1f275d3 
Signed-off-by: Yuxiang <yuxiang.liang@intel.com>
@hsliuustc0106 hsliuustc0106 added the enhancement New feature or request label Jul 17, 2026 — with ChatGPT Codex Connector
@hsliuustc0106 hsliuustc0106 removed the ready label to trigger buildkite CI label Jul 18, 2026
@hsliuustc0106
hsliuustc0106 merged commit 1099424 into vllm-project:main Jul 18, 2026
6 checks passed
y-null pushed a commit to y-null/vllm-omni that referenced this pull request Jul 21, 2026
@Liangyx2 @xuechendi @y-null
Support Fish Speech TTS inference on XPU platform (vllm-project#4856)
506fea4 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
Signed-off-by: Yuxiang Liang <yuxiang.liang@intel.com>
Signed-off-by: Yuxiang <yuxiang.liang@intel.com>
Co-authored-by: Chendi.Xue <chendi.xue@intel.com>
Signed-off-by: y_null <y_null@qq.com>
bowieshi pushed a commit to bowieshi/vllm-omni that referenced this pull request Jul 22, 2026
@Liangyx2 @xuechendi @bowieshi
Support Fish Speech TTS inference on XPU platform (vllm-project#4856)
0aebe0d 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
Signed-off-by: Yuxiang Liang <yuxiang.liang@intel.com>
Signed-off-by: Yuxiang <yuxiang.liang@intel.com>
Co-authored-by: Chendi.Xue <chendi.xue@intel.com>
nagisa-kunhah pushed a commit to nagisa-kunhah/vllm-omni that referenced this pull request Jul 30, 2026
@Liangyx2 @xuechendi @nagisa-kunhah
Support Fish Speech TTS inference on XPU platform (vllm-project#4856)
2a93e29 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
Signed-off-by: Yuxiang Liang <yuxiang.liang@intel.com>
Signed-off-by: Yuxiang <yuxiang.liang@intel.com>
Co-authored-by: Chendi.Xue <chendi.xue@intel.com>
khairulkabir1661 pushed a commit to khairulkabir1661/vllm-omni that referenced this pull request Sep 25, 2026
@Liangyx2 @xuechendi
Support Fish Speech TTS inference on XPU platform (vllm-project#4856)
1325748 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
Signed-off-by: Yuxiang Liang <yuxiang.liang@intel.com>
Signed-off-by: Yuxiang <yuxiang.liang@intel.com>
Co-authored-by: Chendi.Xue <chendi.xue@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xuechendi xuechendi xuechendi approved these changes
@tzhouam tzhouam Awaiting requested review from tzhouam
@lishunyang12 lishunyang12 Awaiting requested review from lishunyang12 lishunyang12 is a code owner
@Gaohan123 Gaohan123 Awaiting requested review from Gaohan123 Gaohan123 is a code owner
@hsliuustc0106 hsliuustc0106 Awaiting requested review from hsliuustc0106 hsliuustc0106 is a code owner
@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner
@david6666666 david6666666 Awaiting requested review from david6666666 david6666666 is a code owner

Assignees

No one assigned

Labels

enhancement New feature or request tts code related to tts models xpu PR related to XPU

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Liangyx2 @hsliuustc0106 @xuechendi