[Bugfix][Model] Fix CohereASR FP16 attention bias dtype

[ActiveSky]

Summary

Fixes #54377.

When Cohere Transcribe runs with --dtype float16 on Turing GPUs, the
relative-position biases can remain in torch.float32 while the query is in
torch.float16. Adding the bias promotes the query to FP32, and the following
matmul with the FP16 positional projection fails during engine initialization.

Cast both relative-position biases to the query dtype before the addition, and
add a CPU regression test for the mixed-dtype forward path.

Duplicate-work check

• gh issue view 54377 --repo vllm-project/vllm --comments: no comments or
  claimants, and no linked pull request.
• gh pr list --repo vllm-project/vllm --state open --search "54377 in:body":
  no results.
• Keyword searches found [Bugfix][Model] Fix CohereASR streaming audio-token estimate (unit + subsampling) #53829 (streaming audio-token accounting) and Fix Cohere ASR failing to load when librosa is not installed #39259
  (lazy librosa import); neither changes the relative-position attention
  dtype path addressed here.

Testing

• .venv/bin/python -m pytest tests/model_executor/test_cohere_asr.py -v (1
  passed)
• pre-commit run --files vllm/model_executor/models/cohere_asr.py tests/model_executor/test_cohere_asr.py
  (passed)
• pre-commit run mypy-3.12 --files vllm/model_executor/models/cohere_asr.py tests/model_executor/test_cohere_asr.py --hook-stage manual
  (passed)

AI assistance

AI assistance from OpenAI Codex was used to investigate the issue, implement the
change, and run the validation above. The human submitter must review every
changed line and be able to explain and defend the change end-to-end.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[ekagra-ranjan]

Hi @ActiveSky - thanks for the PR!

The issue was that the pos_bias variables were being casted to the dtype of the original weight rather than the runtime type decided by --dtype. A direct fix would be to remove this block of code during weight loading. I removed it and tried on H100 and it worked and the existing tests WER tests passed for cohere ASR.

Can you do that instead in this PR and confirm it works on your Turing GPU?

[ActiveSky]

Hi @ekagra-ranjan, thank you for the detailed diagnosis and for testing the alternative on H100. We checked the loading path and confirmed that the custom pos_bias conversion changes the shared relative-position bias buffers to the checkpoint dtype, overriding the runtime dtype. We updated this PR to remove that conversion and restored the original attention additions. We also added a regression test covering FP32 checkpoint values loaded into FP16 runtime buffers and a corresponding attention forward pass.

The focused test, .venv/bin/python -m pytest tests/model_executor/test_cohere_asr.py -v, passed, and the project pre-commit checks passed. The update is in commit fee8e03. Could you please take another look when convenient? Thank you again for your guidance.

[ekagra-ranjan]

Thanks - LGTM!

[ekagra-ranjan]

@ActiveSky - after #55190 is merged, could you move the testfile you created in this PR to tests/models/multimodal/test_cohere_asr.py before we merge this PR?

[ActiveSky]

Hi @ekagra-ranjan, thank you for the follow-up suggestion. We’ve noted it and will keep an eye on #55190. Once #55190 is merged, we’ll update this PR to move the test file to tests/models/multimodal/test_cohere_asr.py as recommended, before proceeding with the merge. Thank you again for your guidance.