Implement custom dataset class for ASR benchmarking

[ymoslem]

Added a new function to process a custom audio dataset for ASR benchmarking.

Purpose

This PR adds support for benchmarking ASR (Automatic Speech Recognition) models using a custom local dataset. It introduces:

• process_audio(): a utility function that normalizes audio inputs from:

  • file paths (via soundfile);
  • HuggingFace-style dicts {"array": ..., "sampling_rate": ...}; or
  • raw (array, sr) tuples.

• CustomAudioDataset: a new dataset class extending CustomDataset that loads audio samples from a JSONL file (e.g., {"prompt": "", "audio": "/path/to/audio.wav"}), processes them via process_audio(), and constructs SampleRequest objects with the audio as multi_modal_data.

• CLI support: custom_audio added as a valid --dataset-name choice, with a corresponding elif branch in get_samples().

Latest updates:

• Added custom_audio to the --dataset-name choices, CustomAudioDataset class and process_audio function:
  • Support ASR models (Whisper tested)
  • Support Multimodal (text + audio) models requiring a chat template (Qwen2-Audio tested)
• Changed custom_mm to custom_image and CustomMMDataset to CustomImageDataset:
  • For now, both custom_mm and custom_image are accepted to keep backward compatibility.

Test Plan 1 (Whisper)

Create a sample JSONL dataset

echo '{"prompt": "", "audio": "/path/to/test.wav"}' > whisper_dataset.jsonl

Start a server

vllm serve openai/whisper-tiny

Run the benchmark

Note: You might need to start another Terminal window, unless you use nohup when starting the server.

vllm bench serve \
  --model openai/whisper-tiny \
  --backend openai-audio \
  --endpoint /v1/audio/transcriptions \
  --dataset-name custom_audio \
  --dataset-path whisper_dataset.jsonl \
  --no-oversample \
  --save-result \
  --save-detailed \
  --result-filename whisper_bench.json

Test Plan 2 (Qwen2-Audio)

Create a sample JSONL dataset.

It is better to have a "prompt" with the required instruction.

echo '{"prompt": "Transcribe the audio.", "audio": "/path/to/test.wav"}' > qwen_dataset.jsonl

Start a server

vllm serve Qwen/Qwen2-Audio-7B-Instruct

Run the benchmark

Note: You might need to start another Terminal window, unless you use nohup when starting the server.

vllm bench serve \
  --model Qwen/Qwen2-Audio-7B-Instruct \
  --backend openai-chat \
  --endpoint /v1/chat/completions \
  --dataset-name custom_audio \
  --dataset-path qwen_dataset.jsonl \
  --no-oversample \
  --enable-multimodal-chat \
  --save-result \
  --save-detailed \
  --result-filename qwen_bench.json

Test Result

By the end of the run:

• You should see "Serving Benchmark Result"
• If you used the --save-result and --save-detailed options, the whisper_bench.json and qwen_bench.json files should include the results and outputs.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[gemini-code-assist]

Code Review

This pull request introduces support for custom audio datasets in the benchmarking suite by adding a CustomAudioDataset class and a process_audio utility. The review feedback highlights a potential module-level failure due to the top-level import of soundfile without an ImportError check. Additionally, the feedback identifies several inconsistencies in the CustomAudioDataset.sample method, including missing support for the skip_chat_template flag, incorrect handling of null tokenizers, and the omission of logic for the output_tokens field.

[ymoslem]

Automatic suggestions reviewed

[ymoslem]

@ywang96 Would you please review. Thanks!

[ymoslem]

Reviewing suggestions

[ymoslem]

Fixed suggestions

[mergify]

Hi @ymoslem, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[mergify]

Hi @ymoslem, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[mergify]

Hi @ymoslem, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[mergify]

Hi @ymoslem, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[mergify]

Hi @ymoslem, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[mergify]

Hi @ymoslem, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[ymoslem]

Added a deprecation warning for custom_mm. Use --dataset-name custom_image instead.

[ymoslem]

Updated deprecation warning for custom_mm

[DarkLight1337]

Can you also update the docs? https://docs.vllm.ai/en/latest/benchmarking/cli/?h=custom_mm#custom-multimodal-dataset

[mergify]

Documentation preview: https://vllm--41576.org.readthedocs.build/en/41576/

[mergify]

Hi @ymoslem, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[mergify]

Hi @ymoslem, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[mergify]

Hi @ymoslem, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[DarkLight1337]

Thanks for your patience!