[CPU] Support for Whisper

[aditew01]

Purpose

Enable support for Whisper for CPU backend

Test Plan

• Added tests to tests/models/multimodal/generation/test_whisper.py -> should be enabled in run-cpu-test.sh

• python examples/offline_inference/audio_language.py -m whisper

• Test Result

python examples/offline_inference/audio_language.py -m whisper
...

INFO 12-04 13:43:46 [llm.py:343] Supported tasks: ['transcription']
Adding requests: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:04<00:00,  4.37s/it]
Processed prompts: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:15<00:00, 15.88s/it, est. speed input: 0.06 toks/s, output: 2.90 toks/s]
 The first words I spoke in the original phonograph, a little piece of practical poetry. Mary had a little lamb, its streets were quite as slow, and everywhere that Mary went the lamb was sure to go.

VLLM_CPU_KVCACHE_SPACE=32 pytest -x -v -s  tests/models/multimodal/generation/test_whisper.py -m cpu_model

 The first words I spoke in the original phonograph, a little piece of practical poetry. Mary had a little lamb, its feet were quite as slow, and everywhere that Mary went, the lamb was sure to go.
 And the 0-1 pitch on the way to Edgar Martinez. Swung on the line. Now the left field line for a base hit. Here comes Joy. Here is Junior to third base. They're going to wave him in. The throw to the plate will be late. The Mariners are going to play for the American League Championship. I don't believe it. It just continues. My, oh, my.
 The first words I spoke in the original phonograph, a little piece of practical poetry. Mary had a little lamb, its feet were quite as slow, and everywhere that Mary went, the lamb was sure to go.
 And the 0-1 pitch on the way to Edgar Martinez. Swung on the line. Now the left field line for a base hit. Here comes Joy. Here is Junior to third base. They're going to wave him in. The throw to the plate will be late. The Mariners are going to play for the American League Championship. I don't believe it. It just continues. My, oh, my.
 The first words I spoke in the original phonograph, a little piece of practical poetry. Mary had a little lamb, its feet were quite as slow, and everywhere that Mary went, the lamb was sure to go.
 And the 0-1 pitch on the way to Edgar Martinez. Swung on the line. Now the left field line for a base hit. Here comes Joy. Here is Junior to third base. They're going to wave him in. The throw to the plate will be late. The Mariners are going to play for the American League Championship. I don't believe it. It just continues. My, oh, my.
 The first words I spoke in the original phonograph, a little piece of practical poetry. Mary had a little lamb, its feet were quite as slow, and everywhere that Mary went, the lamb was sure to go.
 And the 0-1 pitch on the way to Edgar Martinez. Swung on the line. Now the left field line for a base hit. Here comes Joy. Here is Junior to third base. They're going to wave him in. The throw to the plate will be late. The Mariners are going to play for the American League Championship. I don't believe it. It just continues. My, oh, my.
**PASSED**

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request adds CPU support for the Whisper model by enabling non-causal attention in the CPU backend. The changes are logical and include adding necessary tests. However, I've identified a critical issue where the causal flag is not correctly handled for non-causal attention types, which could lead to incorrect model outputs. Please see the detailed comment for more information.

[chatgpt-codex-connector]

💡 Codex Review

https://github.com/vllm-project/vllm/blob/734c32c8757999c5ff3eca602befe19cfc102ce3/vllm/v1/attention/backends/cpu_attn.py#L182-L186
Encoder-decoder CPU attention uses causal mask

The CPU backend now advertises support for AttentionType.ENCODER_DECODER, but scheduler metadata is still built with causal taken directly from CommonAttentionMetadata (initialized as True for all KV cache groups in vllm/v1/worker/gpu/attn_utils.py:167-179). For encoder–decoder cross-attention this flag should be False; leaving it True forces cpu_attention_with_kv_cache to treat cross-attention as causal (sliding_window_right gets clamped to zero and queries start at seq_len - q_len), so decoder tokens will only attend to a truncated slice of the encoder memory when running encoder–decoder models on CPU (e.g., Whisper), producing incorrect outputs.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[aditew01]

cc: @fadara01 @cfRod @bigPYJ1151

[aditew01]

this should close this issue: #29861

[bigPYJ1151]

Thanks for your work! Some nits to resolve.

[fadara01]

you can run vLLM's precommit hook locally as follows:

python -m pip install -U pre-commit
python -m pre_commit run --all-files

[fadara01]

can we add this test to https://github.com/vllm-project/vllm/blob/main/.buildkite/scripts/hardware_ci/run-cpu-test-arm.sh

[aditew01]

Done

[fadara01]

Great work!
just a few nits

[fadara01]

Arm CI running here: https://buildkite.com/vllm/ci/builds/42114#019aee5b-2446-48f6-b203-bbd9d2c52a29

[fadara01]

triggered a new set of CI builds, third time is the charm 🤞

[aditew01]

LGTM: https://buildkite.com/vllm/ci/builds/42136/steps/canvas?sid=019aef3c-53e5-4d81-8224-00c40b1b7b24

[aditew01]

@bigPYJ1151 i've rebased to latest - can you please enable the necessary jobs for the merge?

[aditew01]

@bigPYJ1151 the failure for multi-modal-processor-test-cpu seems unrelated?

[2025-12-10T11:12:51Z] ERROR: Could not find a version that satisfies the requirement decord (from mantis-vl) (from versions: none)
--
[2025-12-10T11:12:51Z] ERROR: No matching distribution found for decord
[2025-12-10T11:12:52Z] 🚨 Error: The command exited with status 1
 ```

[bigPYJ1151]

@aditew01 Yes, I suspect this is due to the ARM CPU nightly images are pushed to the x86 repo as I saw:

 WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v4) and no specific platform was requested

There is a fix vllm-project/ci-infra#243

[aditew01]

Is that a blocker for merge?

[aditew01]

Brill! thanks!

[bigPYJ1151]

I have asked for force merge.