Skip to content

Attention Sinks Perf Boost#78

Merged
LucasWilkinson merged 25 commits intomainfrom
lwilkinson/aux-fast-api-clean
Aug 9, 2025
Merged

Attention Sinks Perf Boost#78
LucasWilkinson merged 25 commits intomainfrom
lwilkinson/aux-fast-api-clean

Conversation

@LucasWilkinson
Copy link
Collaborator

@LucasWilkinson LucasWilkinson commented Aug 7, 2025 •
edited
Loading

Shout-out to @jayhshah (the performance wizard 🪄) for the implementation

PR

reasoning-effort: low
model: openai/gpt-oss-20b

Results (n=5)
mean_chars: 53.7662878788
stderr_chars: 1.6426099883
mean_score: 0.5571969697
stderr_score: 0.0072987006


Main

reasoning-effort: low
model: openai/gpt-oss-20b

Results (n=5)
mean_chars: 53.2223484848
stderr_chars: 1.8772311021
mean_score: 0.5637626263
stderr_score: 0.0020261119

@LucasWilkinson LucasWilkinson mentioned this pull request Aug 7, 2025
Closed
jayhshah
jayhshah reviewed Aug 7, 2025
View reviewed changes
@jayhshah jayhshah force-pushed the lwilkinson/aux-fast-api-clean branch from 994e966 to deb7484 Compare August 8, 2025 00:46
@LucasWilkinson LucasWilkinson force-pushed the lwilkinson/aux-fast-api-clean branch from deb7484 to e1f506c Compare August 8, 2025 01:07
jayhshah and others added 22 commits August 8, 2025 01:08
@jayhshah @LucasWilkinson
add sink test to attention_ref
18a0cae 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
embed sink test into main flash attn test script
684c203 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
change local block positioning for fast path
49be39d 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
add tma gqa modifications
bf2ad39 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
fix exploding build
07b6e77 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
try to fix exploding build again
3f8cfbe 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
prune other hdims to keep build stable
8d4de3c 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
tweak tile size for causal not local case
2145fea 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
split compilation
f293967 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
fix error with varlen q
95cc109 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
split compilation for root setup
e1217ab 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
renable use one mma wg
1a3cd09 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
update for hdim diff
5d58d4b 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
include comments on how to enable hdim diff in cmakelists
6a2c39a 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
fix test variable
0529f4f 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
update pack gqa heuristic
a4e9b0f 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
fix comment
53b576f 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
split disable hdim diff macro into 64 and 192, enable 192 by default …
a763abf 
…in cmakelists

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
add assert to check using Hopper kernels with s_aux
786873d 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
more logical placement of assert checks for hdim diff in flash api
e5935eb 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
cherrypick
323bb43 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
typo fix
4bca7a3 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson LucasWilkinson force-pushed the lwilkinson/aux-fast-api-clean branch from e1f506c to 4bca7a3 Compare August 8, 2025 01:08
@LucasWilkinson
review comments
330171b 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson LucasWilkinson mentioned this pull request Aug 8, 2025
Merged
jayhshah
jayhshah reviewed Aug 8, 2025
View reviewed changes
@LucasWilkinson
review comment
4e44ba2 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson LucasWilkinson changed the title [WIP] Attention Sinks Perf Boost Attention Sinks Perf Boost Aug 9, 2025
@LucasWilkinson LucasWilkinson marked this pull request as ready for review August 9, 2025 03:58
@LucasWilkinson
Merge remote-tracking branch 'upstream/main' into lwilkinson/aux-fast…
6a3d9c8 
…-api-clean

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson LucasWilkinson merged commit 2d3b750 into main Aug 9, 2025
1 check passed
@mickaelseznec
Copy link

mickaelseznec commented Sep 3, 2025

This breaks FlashAttention3 (required for FP8 attention) in vLLM v0.

I know V0 is deprecated but it's still worth a note announcing that support is removed

@mickaelseznec mickaelseznec mentioned this pull request Sep 3, 2025
Closed
@LucasWilkinson LucasWilkinson mentioned this pull request Oct 31, 2025
Open
1 task
@zminglei zminglei mentioned this pull request Feb 27, 2026
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jayhshah jayhshah jayhshah left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@LucasWilkinson @mickaelseznec @jayhshah