Add benchmark for v1/audio/speech non-streaming

[ekagra-ranjan]

Add support for benchmarking v1/audio/speech which measures Audio perf benchmark numbers. It only supports non-streaming endpoint. Fixes partly the Benchmarking item in #938

Below is an example on running 10 samples from MTBench.
The logs print the char len and the token count of the prompts.

Cmd
Terminal 1:

vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \
    --host 0.0.0.0 \
    --port 8000 \
    --gpu-memory-utilization 0.9 \
    --trust-remote-code \
    --enforce-eager \
    --omni

Terminal 2:

time vllm-omni bench serve \
    --backend openai-audio-speech \
    --endpoint /v1/audio/speech \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --dataset-name hf \
	  --dataset-path philschmid/mt-bench \
	  --skip-chat-template \
	  --num-prompts 10 \
	  --max_concurrency 1 \
    --percentile-metrics e2el,audio_rtf,audio_duration \
    --extra-body '{"voice": "Vivian", "instructions": "Speak with great enthusiasm"}'

Logs

time vllm-omni bench serve \
    --backend openai-audio-speech \
    --endpoint /v1/audio/speech \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --dataset-name hf \
          --dataset-path philschmid/mt-bench \
          --skip-chat-template \
          --num-prompts 10 \
          --max_concurrency 1 \
    --percentile-metrics e2el,audio_rtf,audio_duration \
    --extra-body '{"voice": "Vivian", "instructions": "Speak with great enthusiasm"}'
Namespace(subparser='bench', bench_type='serve', dispatch_function=<function BenchmarkServingSubcommand.cmd at 0x762e1190ce00>, seed=0, num_prompts=10, dataset_name='hf', no_stream=False, dataset_path='philschmid/mt-bench', no_oversample=False, skip_chat_template=True, enable_multimodal_chat=False, disable_shuffle=False, custom_output_len=256, spec_bench_output_len=256, spec_bench_category=None, sonnet_input_len=550, sonnet_output_len=150, sonnet_prefix_len=200, sharegpt_output_len=None, blazedit_min_distance=0.0, blazedit_max_distance=1.0, random_input_len=1024, random_output_len=128, random_range_ratio=0.0, random_prefix_len=0, random_batch_size=1, no_reranker=False, random_mm_base_items_per_request=1, random_mm_num_mm_items_range_ratio=0.0, random_mm_limit_mm_per_prompt={'image': 255, 'video': 1}, random_mm_bucket_config={(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}, hf_subset=None, hf_split=None, hf_name=None, hf_output_len=None, prefix_repetition_prefix_len=256, prefix_repetition_suffix_len=256, prefix_repetition_num_prefixes=10, prefix_repetition_output_len=128, label=None, backend='openai-audio-speech', base_url=None, host='127.0.0.1', port=8000, endpoint='/v1/audio/speech', header=None, max_concurrency=1, model='Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice', input_len=None, output_len=None, tokenizer=None, tokenizer_mode='auto', use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, trust_remote_code=False, disable_tqdm=False, num_warmups=0, profile=False, save_result=False, save_detailed=False, append_result=False, metadata=None, result_dir=None, result_filename=None, ignore_eos=False, percentile_metrics='e2el,audio_rtf,audio_duration', metric_percentiles='99', goodput=None, request_id_prefix='bench-d3702d6b-', top_p=None, top_k=None, min_p=None, temperature=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, served_model_name=None, lora_modules=None, ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None, ready_check_timeout_sec=0, extra_body={'voice': 'Vivian', 'instructions': 'Speak with great enthusiasm'})
WARNING: vllm bench serve no longer sets temperature==0 (greedy) in requests by default. The default will be determined on the server side and can be model/API specific. For the old behavior, include --temperature=0.
Starting initial single prompt test run...
Skipping endpoint ready check.
Starting main benchmark run...
Traffic request rate: inf
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: 1
  0%|                                                                                                                                                                                                                      | 0/10 [00:00<?, ?it/s]input: Evaluate the following movie reviews on a scale of 1 to 5, with 1 being very negative, 3 being neutral, and 5 being very positive:
1. This movie released on Nov. 18, 2019, was phenomenal. The cinematography, the acting, the plot - everything was top-notch.
2. Never before have I been so disappointed with a movie. The plot was predictable and the characters were one-dimensional. In my opinion, this movie is the worst one to have been released in 2022.
3. The movie was okay. There were some parts I  enjoyed, but there were also parts that felt lackluster. This is a movie that was released in Feb 2018 and seems to be quite ordinary.
Return the answer as a JSON array of integers., prompt len: 172, char len: 684
st: 2624998.408188054
length of audio bytes: 3321644
 10%|████████████████████▌                                                                                                                                                                                         | 1/10 [01:00<09:02, 60.30s/it]output: MixRequestFuncOutput(generated_text='', success=True, latency=60.16013785917312, output_tokens=0, ttft=60.16013785917312, itl=[], tpot=0.0, prompt_len=172, error='', start_time=2624998.408188054, audio_ttfp=0.0, audio_duration=69.2, audio_frames=1660800, audio_rtf=0.8693661540342936, text_latency=0.0)
output.audio_duration: 69.2
output.audio_frames: 1660800
output.audio_rtf: 0.8693661540342936
output.ttft: 60.16013785917312
input: Write a descriptive paragraph about a bustling marketplace, incorporating sensory details such as smells, sounds, and visual elements to create an immersive experience for the reader., prompt len: 30, char len: 183
st: 2625058.711702966
length of audio bytes: 721964
 20%|█████████████████████████████████████████▏                                                                                                                                                                    | 2/10 [01:13<04:21, 32.65s/it]output: MixRequestFuncOutput(generated_text='', success=True, latency=13.179728702642024, output_tokens=0, ttft=13.179728702642024, itl=[], tpot=0.0, prompt_len=30, error='', start_time=2625058.711702966, audio_ttfp=0.0, audio_duration=15.04, audio_frames=360960, audio_rtf=0.8763117488458793, text_latency=0.0)
output.audio_duration: 15.04
output.audio_frames: 360960
output.audio_rtf: 0.8763117488458793
output.ttft: 13.179728702642024
input: Picture yourself as a 100-years-old tree in a lush forest, minding your own business, when suddenly, a bunch of deforesters shows up to chop you down. How do you feel when those guys start hacking away at you?, prompt len: 51, char len: 209
st: 2625072.004352851
length of audio bytes: 679724
 30%|█████████████████████████████████████████████████████████████▊                                                                                                                                                | 3/10 [01:26<02:44, 23.45s/it]output: MixRequestFuncOutput(generated_text='', success=True, latency=12.391256986185908, output_tokens=0, ttft=12.391256986185908, itl=[], tpot=0.0, prompt_len=51, error='', start_time=2625072.004352851, audio_ttfp=0.0, audio_duration=14.16, audio_frames=339840, audio_rtf=0.8750887702108692, text_latency=0.0)
output.audio_duration: 14.16
output.audio_frames: 339840
output.audio_rtf: 0.8750887702108692
output.ttft: 12.391256986185908
input: The city of Vega intends to build a bridge that will span the Vegona River, covering a distance of 1.8 kilometers. The proposed location falls within a seismically active area that has experienced several high-magnitude earthquakes. Given these circumstances, what would be the best approach to constructing the bridge?, prompt len: 62, char len: 319
st: 2625084.510547531
length of audio bytes: 956204
 40%|██████████████████████████████████████████████████████████████████████████████████▍                                                                                                                           | 4/10 [01:43<02:06, 21.05s/it]output: MixRequestFuncOutput(generated_text='', success=True, latency=17.261045246385038, output_tokens=0, ttft=17.261045246385038, itl=[], tpot=0.0, prompt_len=62, error='', start_time=2625084.510547531, audio_ttfp=0.0, audio_duration=19.92, audio_frames=478080, audio_rtf=0.8665183356618994, text_latency=0.0)
output.audio_duration: 19.92
output.audio_frames: 478080
output.audio_rtf: 0.8665183356618994
output.ttft: 17.261045246385038
input: Given the following data, identify the company with the highest profit in 2021 and provide its CEO's name:
a) Company X, with CEO Amy Williams, reported $30 billion in revenue and a $3 billion profit in 2021.
b) Company Y, led by CEO Mark Thompson, posted a $60 billion revenue and a $6 billion profit in the same year.
c) Company Z, under CEO Sarah Johnson, announced a $20 billion revenue and a $7 billion profit in 2021.
d) Company W, managed by CEO James Smith, revealed a $300 billion revenue with a $21 billion profit in 2021.
e) Company V, with CEO Lisa Brown, reported a $200 billion revenue and a $25 billion profit in 2021.
f) Company U, under CEO John White, posted a $180 billion revenue and a $20 billion profit in the same year., prompt len: 209, char len: 742
st: 2625101.88715105
length of audio bytes: 3621164
 50%|███████████████████████████████████████████████████████████████████████████████████████████████████████                                                                                                       | 5/10 [02:49<03:06, 37.32s/it]output: MixRequestFuncOutput(generated_text='', success=True, latency=66.02956408215687, output_tokens=0, ttft=66.02956408215687, itl=[], tpot=0.0, prompt_len=209, error='', start_time=2625101.88715105, audio_ttfp=0.0, audio_duration=75.44, audio_frames=1810560, audio_rtf=0.875259333008442, text_latency=0.0)
output.audio_duration: 75.44
output.audio_frames: 1810560
output.audio_rtf: 0.875259333008442
output.ttft: 66.02956408215687
input: Edit the following paragraph to correct any grammatical errors:
She didn't remembre where is her purse, so I thinks its in the car but he's say it's on kitchen table but he are not sure, and then they asked me to looking for it, she's say, "Can you?", and I responds with, "Maybe, but ain't no sure," and he not heard me, and, "What?", he asks, "Did you found it?"., prompt len: 96, char len: 365
st: 2625168.059135783
length of audio bytes: 1382444
 60%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                                                                  | 6/10 [03:15<02:13, 33.25s/it]output: MixRequestFuncOutput(generated_text='', success=True, latency=25.228616593871266, output_tokens=0, ttft=25.228616593871266, itl=[], tpot=0.0, prompt_len=96, error='', start_time=2625168.059135783, audio_ttfp=0.0, audio_duration=28.8, audio_frames=691200, audio_rtf=0.8759936317316411, text_latency=0.0)
output.audio_duration: 28.8
output.audio_frames: 691200
output.audio_rtf: 0.8759936317316411
output.ttft: 25.228616593871266
input: Describe the process and write out the balanced chemical equation for the reaction that occurs when solid calcium carbonate reacts with hydrochloric acid to form aqueous calcium chloride, carbon dioxide, and water. What type of reaction is this, and what observations might indicate that the reaction is taking place?, prompt len: 57, char len: 317
st: 2625193.409024696
length of audio bytes: 1075244
 70%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏                                                             | 7/10 [03:34<01:26, 28.74s/it]output: MixRequestFuncOutput(generated_text='', success=True, latency=19.32455507433042, output_tokens=0, ttft=19.32455507433042, itl=[], tpot=0.0, prompt_len=57, error='', start_time=2625193.409024696, audio_ttfp=0.0, audio_duration=22.4, audio_frames=537600, audio_rtf=0.862703351532608, text_latency=0.0)
output.audio_duration: 22.4
output.audio_frames: 537600
output.audio_rtf: 0.862703351532608
output.ttft: 19.32455507433042
input: How many integers are in the solution of the inequality |x + 5| < 10, prompt len: 20, char len: 68
st: 2625212.851217672
length of audio bytes: 368684
 80%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                                         | 8/10 [03:41<00:43, 21.77s/it]output: MixRequestFuncOutput(generated_text='', success=True, latency=6.736967790871859, output_tokens=0, ttft=6.736967790871859, itl=[], tpot=0.0, prompt_len=20, error='', start_time=2625212.851217672, audio_ttfp=0.0, audio_duration=7.68, audio_frames=184320, audio_rtf=0.8772093477697733, text_latency=0.0)
output.audio_duration: 7.68
output.audio_frames: 184320
output.audio_rtf: 0.8772093477697733
output.ttft: 6.736967790871859
input: Imagine you are participating in a race with a group of people. If you have just overtaken the second person, what's your current position? Where is the person you just overtook?, prompt len: 38, char len: 178
st: 2625219.699216081
length of audio bytes: 637484
 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                    | 9/10 [03:52<00:18, 18.55s/it]output: MixRequestFuncOutput(generated_text='', success=True, latency=11.360684623010457, output_tokens=0, ttft=11.360684623010457, itl=[], tpot=0.0, prompt_len=38, error='', start_time=2625219.699216081, audio_ttfp=0.0, audio_duration=13.28, audio_frames=318720, audio_rtf=0.8554732396845224, text_latency=0.0)
output.audio_duration: 13.28
output.audio_frames: 318720
output.audio_rtf: 0.8554732396845224
output.ttft: 11.360684623010457
input: Embrace the role of Sheldon from "The Big Bang Theory" as we delve into our conversation. Don’t start with phrases like "As Sheldon". Let's kick things off with the following question: "What is your opinion on hand dryers?", prompt len: 50, char len: 223
st: 2625231.178188761
length of audio bytes: 833324
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [04:08<00:00, 17.57s/it]output: MixRequestFuncOutput(generated_text='', success=True, latency=15.253788170870394, output_tokens=0, ttft=15.253788170870394, itl=[], tpot=0.0, prompt_len=50, error='', start_time=2625231.178188761, audio_ttfp=0.0, audio_duration=17.36, audio_frames=416640, audio_rtf=0.8786744338058983, text_latency=0.0)
output.audio_duration: 17.36
output.audio_frames: 416640
output.audio_rtf: 0.8786744338058983
output.ttft: 15.253788170870394
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [04:08<00:00, 24.81s/it]
tip: install termplotlib and gnuplot to plot the metrics
============ Serving Benchmark Result ============
Successful requests:                     10        
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  248.14    
Request throughput (req/s):              0.04      
Peak concurrent requests:                2.00      
----------------End-to-end Latency----------------
Mean E2EL (ms):                          24692.63  
Median E2EL (ms):                        16257.42  
P99 E2EL (ms):                           65501.32  
================== Text Result ===================
Total input tokens:                      785       
Total generated tokens:                  0         
Output token throughput (tok/s):         0.00      
Peak output token throughput (tok/s):    1.00      
Peak concurrent requests:                2.00      
Total Token throughput (tok/s):          3.16      
================== Audio Result ==================
Total audio duration generated(s):       283.28    
Total audio frames generated:            6798720   
Audio throughput(audio duration/s):      1.14      
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.87      
Median AUDIO_RTF:                        0.88      
P99 AUDIO_RTF:                           0.88      
------------------Audio Duration------------------
Mean AUDIO_DURATION (s):                 28.33     
Median AUDIO_DURATION (s):               18.64     
P99 AUDIO_DURATION (s):                  74.88     
==================================================

real    4m24.772s
user    0m20.315s
sys     0m7.999s

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 41f9a2e4c1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Decode speech bytes with the declared response format

This path always decodes with AudioSegment.from_file(io.BytesIO(audio_bytes)), which relies on container/header autodetection, but /v1/audio/speech in this repo supports response_format="pcm" (raw bytes without headers). For PCM responses, decoding will fail, the exception is swallowed, and the request is still marked successful, so audio_duration/audio_rtf are silently reported as zero and the benchmark output is incorrect for that valid format.

Useful? React with 👍 / 👎.

[linyueqian]

Looks good for non-streaming TTS benchmarking. A few minor things inline.

Main suggestion: with #1161 about to land, we should also add a streaming variant (async_request_openai_audio_speech_streaming) so we can measure audio_ttfp, inter-chunk latency, and per-chunk RTF. Happy to discuss whether that belongs here or in a follow-up.

[linyueqian]

If AudioSegment.from_file raises, output.success is still set to True with audio_duration=0, audio_frames=0, audio_rtf=0. This silently skews aggregate metrics. Consider setting output.success = False on parse failure, or at least logging at error level.

[ekagra-ranjan]

makes sense!

[ekagra-ranjan]

@linyueqian - Thank you for your quick reviews! I have addressed the comments.

[linyueqian]

@hsliuustc0106 LGTM, all comments addressed.

[lishunyang12]

A few observations on the benchmark patch.

[lishunyang12]

Would unknown backends be better off falling through to get_samples_old instead of raising? That way new upstream backends wouldn't require a patch change.

[ekagra-ranjan]

makes sense

[lishunyang12]

AudioSegment.from_file will fail on raw PCM (response_format=pcm). Server default is WAV so it's fine today, but if someone passes response_format through extra_body, this silently breaks. Worth documenting or guarding.

[ekagra-ranjan]

I have added code to raise error when format is pcm. I didnt go with processing pcm value because it would need frame_rate, sample_width and channel beforehand so just made it simple since this is benchmark so using wav should be fine

[lishunyang12]

Does _update_payload_common inject keys (stream, max_tokens, logprobs) that the /v1/audio/speech endpoint doesn't expect? The chat-omni path builds its payload manually instead.

[ekagra-ranjan]

It doesnt add those https://github.com/vllm-project/vllm/blob/f5972a872fa3fabd94b7a6c6f031f4b5bcee2b2d/vllm/benchmarks/lib/endpoint_request_func.py#L122-L129

[lishunyang12]

Nit: traceback.format_exc() does the same thing as the sys.exc_info() + format_exception dance in one call.

[hsliuustc0106]

🤖 Code Review: PR #1408 — TTS Benchmark for v1/audio/speech ✅

Verdict: Approve

Summary: Adds a new 'openai-audio-speech' backend to the benchmark infrastructure for non-streaming TTS performance measurement. Measures RTF and throughput via the /v1/audio/speech endpoint. Clean and minimal at ~80 lines of new code, follows the exact same pattern as async_request_openai_chat_omni_completions. Correctly reuses MixRequestFuncOutput, shared helpers, and the existing metrics pipeline. Well done overall with only minor items to address.

Key Concerns:

• ⚠️ Verify get_samples() produces appropriate TTS-only inputs (plain text) when backend is 'openai-audio-speech', rather than multimodal chat inputs from OmniRandomMultiModalDataset.

Detailed Feedback

📄 benchmarks/backend_request_func.py

🟡 MEDIUM (architecture) — Line 43

get_samples() backend allowlist correctly includes 'openai-audio-speech', but then calls OmniRandomMultiModalDataset which generates multimodal chat samples. For TTS, the 'samples' should be plain text strings. Verify the dataset generates appropriate TTS inputs rather than multimodal chat inputs (text + audio + image).

🟢 LOW (quality)

For non-streaming, setting ttft = audio_ttfp = latency is correct and well-commented. The audio_frames calculation handles frame_width == 0 edge case with a warning. Good defensive coding.

📄 tests/

🟢 LOW (tests)

No unit tests for async_request_openai_audio_speech(). The existing async_request_openai_chat_omni_completions also lacks unit tests, so this is consistent with the codebase, but still a gap worth noting.

Suggestions

1. Verify get_samples() produces appropriate TTS-only inputs when backend is 'openai-audio-speech'.
2. Add a usage example in the PR description or a README update showing how to run the TTS benchmark (e.g., vllm bench serve --omni --backend openai-audio-speech ...).
3. Consider adding a 'voice' field to the payload construction so different speakers can be benchmarked.

Part of Issue #938 — Qwen3-TTS Production Ready

---

Review generated by vLLM-Omni PR Review Agent

[linyueqian]

@ekagra-ranjan can you address the reviews above so that we can merge this PR for future benchmarking purporse?

[hsliuustc0106]

@vllm-omni-reviewer

[ekagra-ranjan]

@linyueqian - the comments are now addressed. Pls have a look

[linyueqian]

LGTM for merge. I re-checked the earlier concerns and they’re addressed: audio_ttfp is populated, audio parse failures are marked as failed requests, and response_format=pcm is guarded.

Optional follow-up: skip random-mm multimodal sample construction for openai-audio-speech to avoid unnecessary benchmark overhead.

[linyueqian]

Reviewed the latest updates again; concerns are addressed and this looks good to merge.