Skip to content

[BugFix] ChunkedLocalAttention is currently not CG compatible #26034

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 1, 2025
Merged

[BugFix] ChunkedLocalAttention is currently not CG compatible #26034

merged 1 commit into from
Oct 1, 2025

Conversation

@LucasWilkinson
Copy link
Collaborator

@LucasWilkinson LucasWilkinson commented Oct 1, 2025

potential fix for: #25960

@LucasWilkinson LucasWilkinson mentioned this pull request Oct 1, 2025
Closed
5 tasks
@mgoin
Copy link
Member

mgoin commented Oct 1, 2025 •
edited
Loading

Verified it does work right before capture

vllm serve RedHatAI/Llama-4-Scout-17B-16E-Instruct-quantized.w4a16 --load-format dummy --max-model-len 32K
...
(EngineCore_DP0 pid=2401224) WARNING 10-01 12:23:25 [gpu_model_runner.py:3772] CUDAGraphMode.FULL_AND_PIECEWISE is not supported with ChunkedLocalAttentionBuilder backend (support: AttentionCGSupport.NEVER); setting cudagraph_mode=PIECEWISE because attention is compiled piecewise
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████████████████████████████████████████████████████████████████████████| 67/67 [00:07<00:00,  8.85it/s]
(EngineCore_DP0 pid=2401224) INFO 10-01 12:23:33 [gpu_model_runner.py:3583] Graph capturing finished in 8 secs, took 0.52 GiB

@mgoin mgoin marked this pull request as ready for review October 1, 2025 16:25
mgoin
mgoin approved these changes Oct 1, 2025
View reviewed changes
Copy link
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the better approach

@mgoin mgoin added bug Something isn't working ready ONLY add when PR is ready to merge/full CI is needed llama Related to Llama models labels Oct 1, 2025
@cjackal
Copy link
Contributor

cjackal commented Oct 1, 2025

I wonder if this PR has a chance to be cherry-picked for 0.11.0? If so, I'd like to note that there's another llama 4 performance bugfix at #25889 which would be cherry-picked together.

@tlrmchlsmth tlrmchlsmth added this to the v0.11.0 Cherry Picks milestone Oct 1, 2025
@mgoin
Copy link
Member

mgoin commented Oct 1, 2025

Hybrid test failures are also on main, merging

@vllm-bot vllm-bot merged commit 4134312 into vllm-project:main Oct 1, 2025
48 of 51 checks passed
@cjackal cjackal mentioned this pull request Oct 2, 2025
Closed
1 task
simon-mo pushed a commit that referenced this pull request Oct 2, 2025
@LucasWilkinson @simon-mo
[BugFix] ChunkedLocalAttention is currently not CG compatible (#26034)
c536881 
Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: simon-mo <[email protected]>
pdasigi pushed a commit to pdasigi/vllm that referenced this pull request Oct 2, 2025
@LucasWilkinson @pdasigi
[BugFix] ChunkedLocalAttention is currently not CG compatible (vllm-p…
4095627 
…roject#26034)

Signed-off-by: Lucas Wilkinson <[email protected]>
yewentao256 pushed a commit that referenced this pull request Oct 3, 2025
@LucasWilkinson @yewentao256
[BugFix] ChunkedLocalAttention is currently not CG compatible (#26034)
ac1598d 
Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: yewentao256 <[email protected]>
tomeras91 pushed a commit to tomeras91/vllm that referenced this pull request Oct 6, 2025
@LucasWilkinson @tomeras91
[BugFix] ChunkedLocalAttention is currently not CG compatible (vllm-p…
4fff719 
…roject#26034)

Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: Tomer Asida <[email protected]>
southfreebird pushed a commit to southfreebird/vllm that referenced this pull request Oct 7, 2025
@LucasWilkinson @southfreebird
[BugFix] ChunkedLocalAttention is currently not CG compatible (vllm-p…
8b1d3ff 
…roject#26034)

Signed-off-by: Lucas Wilkinson <[email protected]>
xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Oct 10, 2025
@LucasWilkinson @xuebwang-amd
[BugFix] ChunkedLocalAttention is currently not CG compatible (vllm-p…
0c1a26b 
…roject#26034)

Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: xuebwang-amd <[email protected]>
choprahetarth pushed a commit to Tandemn-Labs/vllm that referenced this pull request Oct 11, 2025
@LucasWilkinson @choprahetarth
[BugFix] ChunkedLocalAttention is currently not CG compatible (vllm-p…
1f4cf51 
…roject#26034)

Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: simon-mo <[email protected]>
shyeh25 pushed a commit to shyeh25/vllm that referenced this pull request Oct 14, 2025
@LucasWilkinson @shyeh25
ChunkedLocalAttention is currently not CG compatible (vllm-project#26034
c5687a8 
)

Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: simon-mo <[email protected]>
lywa1998 pushed a commit to lywa1998/vllm that referenced this pull request Oct 20, 2025
@LucasWilkinson @lywa1998
[BugFix] ChunkedLocalAttention is currently not CG compatible (vllm-p…
9a1603d 
…roject#26034)

Signed-off-by: Lucas Wilkinson <[email protected]>
alhridoy pushed a commit to alhridoy/vllm that referenced this pull request Oct 24, 2025
@LucasWilkinson @alhridoy
[BugFix] ChunkedLocalAttention is currently not CG compatible (vllm-p…
4554632 
…roject#26034)

Signed-off-by: Lucas Wilkinson <[email protected]>
xuebwang-amd pushed a commit to xuebwang-amd/vllm that referenced this pull request Oct 24, 2025
@LucasWilkinson @xuebwang-amd
[BugFix] ChunkedLocalAttention is currently not CG compatible (vllm-p…
79b2cd0 
…roject#26034)

Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: xuebwang-amd <[email protected]>
rtourgeman pushed a commit to rtourgeman/vllm that referenced this pull request Nov 10, 2025
@LucasWilkinson @rtourgeman
[BugFix] ChunkedLocalAttention is currently not CG compatible (vllm-p…
f7c48e4 
…roject#26034)

Signed-off-by: Lucas Wilkinson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tlrmchlsmth tlrmchlsmth tlrmchlsmth approved these changes

@mgoin mgoin mgoin approved these changes

Assignees

No one assigned

Labels

bug Something isn't working llama Related to Llama models ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

v0.11.0 Cherry Picks

Development

Successfully merging this pull request may close these issues.

5 participants

@LucasWilkinson @mgoin @cjackal @tlrmchlsmth @vllm-bot