[Refactor] GLM-ASR Modeling

[JaredforReal]

🚀 Key Improvements

1. Native vLLM Audio Encoder Implementation (glmasr.py)

• Completely rewrote GlmAsrEncoder as a vLLM-native implementation with full optimization support:
  • QKVParallelLinear: Fused Q/K/V projections for efficient attention computation
  • ColumnParallelLinear/RowParallelLinear: Tensor parallelism support for distributed inference
  • Quantization support: Compatible with vLLM's quantization framework
  • Flash Attention (SDPA): Leverages PyTorch's scaled_dot_product_attention for optimized attention
• Implemented GlmAsrRotaryEmbedding with pre-computed cos/sin cache for faster RoPE computation
• Built GlmAsrAttention, GlmAsrMLP, and GlmAsrEncoderLayer with optimized layer norms and residual connections
• Proper grouped query attention (GQA) handling with k/v repetition

2. Cleaner Architecture: Direct BaseMultiModalProcessor Inheritance (glmasr.py)

• Refactored GlmAsrMultiModalProcessor to inherit directly from BaseMultiModalProcessor
• Refactored GlmAsrProcessingInfo to inherit directly from BaseProcessingInfo
• Refactored GlmAsrMultiModalDataParse to inherit directly from BaseMultiModalDataParse
• Removed dependency on AudioFlamingo3 for cleaner, more maintainable code
• Streamlined processing pipeline with better performance and reduced complexity
• Maintained full compatibility with existing API and functionality

Purpose

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This is a great refactoring that replaces the GlmAsrEncoder with a native vLLM implementation, improving performance and maintainability by removing dependencies on AudioFlamingo3. The new implementation is well-structured and leverages vLLM's optimizations. I found one critical issue related to handling input sequences longer than the configured maximum, which could lead to a runtime error. I've provided a suggestion to fix it. Overall, this is a solid contribution.

[Copilot]

Pull request overview

This pull request refactors the GLM-ASR model implementation to use a native vLLM encoder with comprehensive performance optimizations including tensor parallelism, quantization support, and Flash Attention. The refactoring also simplifies the architecture by removing dependencies on AudioFlamingo3 base classes and instead directly inheriting from vLLM's base multimodal processor classes.

Key Changes

• Native vLLM encoder: Complete rewrite of GlmAsrEncoder with optimized components (GlmAsrRotaryEmbedding, GlmAsrAttention, GlmAsrMLP, GlmAsrEncoderLayer) using QKVParallelLinear for fused projections, tensor parallelism support, and Flash Attention
• Direct base class inheritance: Refactored GlmAsrProcessingInfo, GlmAsrMultiModalProcessor, and GlmAsrMultiModalDataParser to inherit from BaseProcessingInfo, BaseMultiModalProcessor, and MultiModalDataParser respectively, removing AudioFlamingo3 dependencies
• Enhanced utility functions: Added rotary embedding helpers (_rotate_half, _apply_rotary_pos_emb, _repeat_kv) and improved audio length calculation logic in glmasr_utils.py

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 10 comments.

File	Description
vllm/model_executor/models/glmasr_utils.py	Added RoPE and GQA utility functions; improved audio output length calculation with better documentation and refactored logic
vllm/model_executor/models/glmasr.py	Complete native encoder implementation with optimized attention/MLP layers; refactored processor classes to remove AudioFlamingo3 dependency; inlined and improved audio processing logic in _call_hf_processor

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[JaredforReal]

@DarkLight1337 PTAL, thanks!

[DarkLight1337]

/gemini review

[gemini-code-assist]

Code Review

This pull request is a significant refactoring of the GLM-ASR modeling implementation. It successfully replaces the HuggingFace GlmAsrEncoder with a vLLM-native version, which should improve performance and maintainability. The code is well-structured and leverages vLLM's optimized components effectively. I've found one minor correctness issue in a helper function where the implementation doesn't match its documentation, which I've flagged for correction to prevent potential future bugs.

[DarkLight1337]

LGTM now. @Isotr0py can you check as well?

[Isotr0py]

LGTM now

[JaredforReal]

@DarkLight1337 @Isotr0py Thanks

[JaredforReal]

@DarkLight1337 @Isotr0py Sorry guys, I need one more change to pass the device to transformers.whisper.feature_extractor(), for better performance
PTAL

[JaredforReal]

@DarkLight1337 @Isotr0py Back to the version you guys approved. Let's land it for now, Thanks!