[trainer] fix: fallback vision tower to flash_attention_2 for Qwen2.5-VL when u…#4670
Merged
wuxibin89 merged 1 commit intoverl-project:mainfrom Dec 29, 2025
Conversation
…sing flash_attention_3 Qwen2.5-VL vision tower does not support flash_attention_3, so when attn_implementation is set to flash_attention_3, we need to set the vision tower's _attn_implementation to flash_attention_2 instead.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
The pull request introduces a code change in verl/workers/fsdp_workers.py within the _build_model_optimizer function. This change adds a specific patch for the qwen2_5_vl model. If the model type is qwen2_5_vl and the attention implementation is set to flash_attention_3, the patch overrides the vision tower's attention implementation to flash_attention_2, as the vision tower does not support flash_attention_3.
wuxibin89
changed the title
wuxibin89
approved these changes
Dec 29, 2025
boren-ms
pushed a commit
to boren-ms/verl
that referenced
this pull request
Dec 30, 2025
…-VL when u… (verl-project#4670) # Fix: Fallback Vision Tower to Flash Attention 2 for Qwen2.5-VL when using Flash Attention 3 ## Description This PR adds a patch for Qwen2.5-VL models to fallback the vision tower's attention implementation to flash_attention_2 when the main model uses flash_attention_3. ## Motivation Qwen2.5-VL's vision tower does not support flash_attention_3 properly. When `attn_implementation` is set to `flash_attention_3`, using FA3 for the vision tower causes significant performance degradation compared to flash_attention_2. ## Experimental Validation We have tested this patch across the entire Qwen2.5-VL series (3B, 7B, 32B, and 72B models) using the Transformers library on an 8×H100 GPU machine with auto device placement. Below is the performance comparison for Qwen2.5-VL-7B with input of one 1260×700 image + 150 tokens of text: ``` ====================================================================== COMPARISON SUMMARY ====================================================================== Implementation Avg Latency (ms) Throughput (tok/s) ------------------------------------------------------------- flash_attention_2 102.85 12503.46 flash_attention_3 309.49 4155.19 FA3 vs FA2 Speedup: 0.33x Memory Difference: +0.00 GB ``` **Test Environment:** - Hardware: 8×H100 GPUs - Library: Transformers with auto device placement - Models tested: Qwen2.5-VL-3B, 7B, 32B, 72B **Key Findings:** - Flash Attention 3 is **3x slower** than Flash Attention 2 for the vision tower - No memory benefit from using FA3 for vision components - Consistent behavior observed across all model sizes (3B, 7B, 32B, 72B) ## Changes - Added a check for `qwen2_5_vl` model type - When `attn_implementation == "flash_attention_3"`, automatically set `actor_model_config.vision_config._attn_implementation = "flash_attention_2"` for the vision tower - This allows the language model to use FA3 while the vision tower uses FA2, achieving optimal performance ## Impact This change ensures that Qwen2.5-VL models can benefit from flash_attention_3 for text processing while maintaining optimal performance for vision encoding. ## Technical Details The patch is applied in `verl/workers/fsdp_workers.py` in the `_build_model_optimizer` method: ```python # patch for qwen2.5-vl: when using flash_attention_3, set vision tower to use flash_attention_2 # because the vision tower does not support flash_attention_3 if ( getattr(actor_model_config, "model_type", None) == "qwen2_5_vl" and attn_implementation == "flash_attention_3" and hasattr(actor_model_config, "vision_config") ): actor_model_config.vision_config._attn_implementation = "flash_attention_2" ``` ## Testing Tested on: - Qwen2.5-VL-3B - Qwen2.5-VL-7B - Qwen2.5-VL-32B - Qwen2.5-VL-72B All models show consistent performance improvements with this patch when using flash_attention_3 for the language model.
jsfanfanfan
pushed a commit
to meituan-search/verl
that referenced
this pull request
Jan 9, 2026
…-VL when u… (verl-project#4670) # Fix: Fallback Vision Tower to Flash Attention 2 for Qwen2.5-VL when using Flash Attention 3 ## Description This PR adds a patch for Qwen2.5-VL models to fallback the vision tower's attention implementation to flash_attention_2 when the main model uses flash_attention_3. ## Motivation Qwen2.5-VL's vision tower does not support flash_attention_3 properly. When `attn_implementation` is set to `flash_attention_3`, using FA3 for the vision tower causes significant performance degradation compared to flash_attention_2. ## Experimental Validation We have tested this patch across the entire Qwen2.5-VL series (3B, 7B, 32B, and 72B models) using the Transformers library on an 8×H100 GPU machine with auto device placement. Below is the performance comparison for Qwen2.5-VL-7B with input of one 1260×700 image + 150 tokens of text: ``` ====================================================================== COMPARISON SUMMARY ====================================================================== Implementation Avg Latency (ms) Throughput (tok/s) ------------------------------------------------------------- flash_attention_2 102.85 12503.46 flash_attention_3 309.49 4155.19 FA3 vs FA2 Speedup: 0.33x Memory Difference: +0.00 GB ``` **Test Environment:** - Hardware: 8×H100 GPUs - Library: Transformers with auto device placement - Models tested: Qwen2.5-VL-3B, 7B, 32B, 72B **Key Findings:** - Flash Attention 3 is **3x slower** than Flash Attention 2 for the vision tower - No memory benefit from using FA3 for vision components - Consistent behavior observed across all model sizes (3B, 7B, 32B, 72B) ## Changes - Added a check for `qwen2_5_vl` model type - When `attn_implementation == "flash_attention_3"`, automatically set `actor_model_config.vision_config._attn_implementation = "flash_attention_2"` for the vision tower - This allows the language model to use FA3 while the vision tower uses FA2, achieving optimal performance ## Impact This change ensures that Qwen2.5-VL models can benefit from flash_attention_3 for text processing while maintaining optimal performance for vision encoding. ## Technical Details The patch is applied in `verl/workers/fsdp_workers.py` in the `_build_model_optimizer` method: ```python # patch for qwen2.5-vl: when using flash_attention_3, set vision tower to use flash_attention_2 # because the vision tower does not support flash_attention_3 if ( getattr(actor_model_config, "model_type", None) == "qwen2_5_vl" and attn_implementation == "flash_attention_3" and hasattr(actor_model_config, "vision_config") ): actor_model_config.vision_config._attn_implementation = "flash_attention_2" ``` ## Testing Tested on: - Qwen2.5-VL-3B - Qwen2.5-VL-7B - Qwen2.5-VL-32B - Qwen2.5-VL-72B All models show consistent performance improvements with this patch when using flash_attention_3 for the language model.
vyomakesh0728
added a commit
to vyomakesh0728/verl
that referenced
this pull request
Jan 22, 2026
…-VL when u… (verl-project#4670) # Fix: Fallback Vision Tower to Flash Attention 2 for Qwen2.5-VL when using Flash Attention 3 ## Description This PR adds a patch for Qwen2.5-VL models to fallback the vision tower's attention implementation to flash_attention_2 when the main model uses flash_attention_3. ## Motivation Qwen2.5-VL's vision tower does not support flash_attention_3 properly. When `attn_implementation` is set to `flash_attention_3`, using FA3 for the vision tower causes significant performance degradation compared to flash_attention_2. ## Experimental Validation We have tested this patch across the entire Qwen2.5-VL series (3B, 7B, 32B, and 72B models) using the Transformers library on an 8×H100 GPU machine with auto device placement. Below is the performance comparison for Qwen2.5-VL-7B with input of one 1260×700 image + 150 tokens of text: ``` ====================================================================== COMPARISON SUMMARY ====================================================================== Implementation Avg Latency (ms) Throughput (tok/s) ------------------------------------------------------------- flash_attention_2 102.85 12503.46 flash_attention_3 309.49 4155.19 FA3 vs FA2 Speedup: 0.33x Memory Difference: +0.00 GB ``` **Test Environment:** - Hardware: 8×H100 GPUs - Library: Transformers with auto device placement - Models tested: Qwen2.5-VL-3B, 7B, 32B, 72B **Key Findings:** - Flash Attention 3 is **3x slower** than Flash Attention 2 for the vision tower - No memory benefit from using FA3 for vision components - Consistent behavior observed across all model sizes (3B, 7B, 32B, 72B) ## Changes - Added a check for `qwen2_5_vl` model type - When `attn_implementation == "flash_attention_3"`, automatically set `actor_model_config.vision_config._attn_implementation = "flash_attention_2"` for the vision tower - This allows the language model to use FA3 while the vision tower uses FA2, achieving optimal performance ## Impact This change ensures that Qwen2.5-VL models can benefit from flash_attention_3 for text processing while maintaining optimal performance for vision encoding. ## Technical Details The patch is applied in `verl/workers/fsdp_workers.py` in the `_build_model_optimizer` method: ```python # patch for qwen2.5-vl: when using flash_attention_3, set vision tower to use flash_attention_2 # because the vision tower does not support flash_attention_3 if ( getattr(actor_model_config, "model_type", None) == "qwen2_5_vl" and attn_implementation == "flash_attention_3" and hasattr(actor_model_config, "vision_config") ): actor_model_config.vision_config._attn_implementation = "flash_attention_2" ``` ## Testing Tested on: - Qwen2.5-VL-3B - Qwen2.5-VL-7B - Qwen2.5-VL-32B - Qwen2.5-VL-72B All models show consistent performance improvements with this patch when using flash_attention_3 for the language model.
sophiayyya
pushed a commit
to sophiayyya/verl
that referenced
this pull request
Jan 25, 2026
…-VL when u… (verl-project#4670) # Fix: Fallback Vision Tower to Flash Attention 2 for Qwen2.5-VL when using Flash Attention 3 ## Description This PR adds a patch for Qwen2.5-VL models to fallback the vision tower's attention implementation to flash_attention_2 when the main model uses flash_attention_3. ## Motivation Qwen2.5-VL's vision tower does not support flash_attention_3 properly. When `attn_implementation` is set to `flash_attention_3`, using FA3 for the vision tower causes significant performance degradation compared to flash_attention_2. ## Experimental Validation We have tested this patch across the entire Qwen2.5-VL series (3B, 7B, 32B, and 72B models) using the Transformers library on an 8×H100 GPU machine with auto device placement. Below is the performance comparison for Qwen2.5-VL-7B with input of one 1260×700 image + 150 tokens of text: ``` ====================================================================== COMPARISON SUMMARY ====================================================================== Implementation Avg Latency (ms) Throughput (tok/s) ------------------------------------------------------------- flash_attention_2 102.85 12503.46 flash_attention_3 309.49 4155.19 FA3 vs FA2 Speedup: 0.33x Memory Difference: +0.00 GB ``` **Test Environment:** - Hardware: 8×H100 GPUs - Library: Transformers with auto device placement - Models tested: Qwen2.5-VL-3B, 7B, 32B, 72B **Key Findings:** - Flash Attention 3 is **3x slower** than Flash Attention 2 for the vision tower - No memory benefit from using FA3 for vision components - Consistent behavior observed across all model sizes (3B, 7B, 32B, 72B) ## Changes - Added a check for `qwen2_5_vl` model type - When `attn_implementation == "flash_attention_3"`, automatically set `actor_model_config.vision_config._attn_implementation = "flash_attention_2"` for the vision tower - This allows the language model to use FA3 while the vision tower uses FA2, achieving optimal performance ## Impact This change ensures that Qwen2.5-VL models can benefit from flash_attention_3 for text processing while maintaining optimal performance for vision encoding. ## Technical Details The patch is applied in `verl/workers/fsdp_workers.py` in the `_build_model_optimizer` method: ```python # patch for qwen2.5-vl: when using flash_attention_3, set vision tower to use flash_attention_2 # because the vision tower does not support flash_attention_3 if ( getattr(actor_model_config, "model_type", None) == "qwen2_5_vl" and attn_implementation == "flash_attention_3" and hasattr(actor_model_config, "vision_config") ): actor_model_config.vision_config._attn_implementation = "flash_attention_2" ``` ## Testing Tested on: - Qwen2.5-VL-3B - Qwen2.5-VL-7B - Qwen2.5-VL-32B - Qwen2.5-VL-72B All models show consistent performance improvements with this patch when using flash_attention_3 for the language model.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix: Fallback Vision Tower to Flash Attention 2 for Qwen2.5-VL when using Flash Attention 3
Description
This PR adds a patch for Qwen2.5-VL models to fallback the vision tower's attention implementation to flash_attention_2 when the main model uses flash_attention_3.
Motivation
Qwen2.5-VL's vision tower does not support flash_attention_3 properly. When
attn_implementationis set toflash_attention_3, using FA3 for the vision tower causes significant performance degradation compared to flash_attention_2.Experimental Validation
We have tested this patch across the entire Qwen2.5-VL series (3B, 7B, 32B, and 72B models) using the Transformers library on an 8×H100 GPU machine with auto device placement.
Below is the performance comparison for Qwen2.5-VL-7B with input of one 1260×700 image + 150 tokens of text:
Test Environment:
Key Findings:
Changes
qwen2_5_vlmodel typeattn_implementation == "flash_attention_3", automatically setactor_model_config.vision_config._attn_implementation = "flash_attention_2"for the vision towerImpact
This change ensures that Qwen2.5-VL models can benefit from flash_attention_3 for text processing while maintaining optimal performance for vision encoding.
Technical Details
The patch is applied in
verl/workers/fsdp_workers.pyin the_build_model_optimizermethod:Testing
Tested on:
All models show consistent performance improvements with this patch when using flash_attention_3 for the language model.