Whisper model support & /v1/audio/transcriptions endpoint & benchmark

[JustinTong0323]

Motivation

• Follow up model: support OpenAI Whisper #8064. Thanks @MahmoudAshraf97 for the brilliant work.
• This PR follows up the original PR and makes a workaround on the encoder-decoder code path.
• Adds the implementation of the /v1/audio/transcriptions endpoint (currently for whisper only).
• Also adds a benchmark script for the ASR task.

Modifications

1. Support whisper model
2. Add a ASR benchmark

Further TODOs:

• Adapt to sglang's native encoder-decoder code path
• fix precision issue when using --attention-backend flashinfer

Accuracy Tests

Launch Server:

sglang serve --model-path openai/whisper-large-v3

Run benchmark script:

cd sglang/benchmark/asr
python bench_sglang.py --api-type transcription --concurrency 128 --show-predictions

Output:

Loading dataset: D4nt3/esb-datasets-earnings22-validation-tiny-filtered...
Using API type: transcription
Repo card metadata block was not found. Setting CardData to empty.
WARNING:huggingface_hub.repocard:Repo card metadata block was not found. Setting CardData to empty.
Performing warmup...
Processing 511 samples...
------------------------------
Results for openai/whisper-large-v3:
Total Requests: 511
WER: 12.7690
Average Latency: 1.3602s
Median Latency: 1.2090s
95th Latency: 2.9986s
Throughput: 19.02 req/s
Token Throughput: 354.19 tok/s
Total Test Time: 26.8726s
------------------------------

==================== Sample Predictions ====================
Sample 1:
  REF: on the use of taxonomy i you know i think it is it is early days for us to to make any clear indications to the market about the proportion that would fall under that requirement
  PRED: on the eu taxonomy i think it is early days for us to make any clear indications to the market about the proportion that would fall under that requirement
----------------------------------------
Sample 2:
  REF: so within fiscal year 2021 say 120 a 100 depending on what the micro will do and next year it is not necessarily payable in q one is we will look at what the cash flows for 2022 look like
  PRED: so within fiscal year 2021 say $120000 $100000 depending on what the macro will do and next year it is not necessarily payable in q one is we will look at what the cash flows for 2022 look like
----------------------------------------
Sample 3:
  REF: we talked about 4.7 gigawatts
  PRED: we talked about 4.7 gigawatts
----------------------------------------
Sample 4:
  REF: and you know depending on that working capital build we will we will see what that yields
  PRED: and depending on that working capital build we will see what that yields what
----------------------------------------
Sample 5:
  REF: so on on sinopec what we have agreed with sinopec way back then is that free cash flows after paying all capexs are distributed out 30 70%
  PRED: so on sinopec what we have agreed with sinopec way back then is that free cash flows after paying all capexes are distributed out 30% 70%
----------------------------------------
============================================================

Benchmarking and Profiling

As above.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
4. After green CI and required approvals, ask Merge Oncalls to merge.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[JustinTong0323]

/tag-and-rerun-ci

[JustinTong0323]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds comprehensive support for the Whisper model, including a new OpenAI-compatible transcription endpoint, a benchmark script, and the necessary model and processor implementations. The changes are well-structured and cover all the required aspects from the server entrypoint to the model-specific logic.

I've identified a few areas for improvement:

• A potential regression in the is_encoder_decoder_model check that could affect existing models.
• Minor improvements in the new benchmark script for accuracy in metric calculation and code style.
• A small code simplification in the new transcription serving handler.

Overall, this is a great contribution that significantly extends the capabilities of the server. My detailed comments are below.

[yuan-luo]

Just curious why disable cuda graph for encoder-decoder models?

[JustinTong0323]

some compatible issue with the current code path

[JustinTong0323]

/tag-and-rerun-ci

[JustinTong0323]

/rerun-failed-ci

[JustinTong0323]

/rerun-failed-ci

[vshekhawat-hlab]

Why we are having zero k and v. We will hit with junk output with this case, as k, v are not from encoder cache output.

With flashinfer backend, are you getting any useful transcript output?

[JustinTong0323]

/tag-and-rerun-ci

[JustinTong0323]

/rerun-failed-ci