Repository navigation
[Bugfix] Enable dynamic_dims for different embeds shape - #31223
Conversation
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
DarkLight1337
added
ready
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a bugfix to enable dynamic dimension shapes for embeddings in several multimodal models, including AudioFlamingo3, MiniCPMV, and Qwen2Audio. The changes correctly utilize the dynamic_dims parameter within TensorShape annotations. This allows for variable-length feature sequences (e.g., number of audio features or image slices) across different items in a batch, which is crucial for robustly handling diverse multimodal inputs. The modifications are consistent, targeted, and correctly implemented according to the TensorSchema API. The pull request effectively addresses the issue of rigid shape validation for multimodal embeddings.
…ct#31223) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
…ct#31223) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
…ct#31223) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
…ct#31223) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Gemma3nAudioInputs declares input_features_padded and input_features_mask with fixed sequence-length dimensions. When two concurrent requests carry audio clips of different durations, TensorSchema.validate() sees mismatched shapes and kills EngineCore with a ValueError. Mark the sequence-length dimension as dynamic so the validator allows variable-length audio inputs in the same batch, matching the pattern used by MiniCPM-o, Qwen2Audio, and AudioFlamingo3 (PR vllm-project#31223). Fixes vllm-project#38297 Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
…ct#31223) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Purpose
Without setting
dynamic_dims, settinglist[torch.Tensor]doesn't actually enable tensors with different shape.FIX #31219
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.