[WIP] Support for cached multi-query attention towards speculative decoding #1679

Closed

skrider wants to merge 14 commits intovllm-project:mainfrom

skrider:cached-mqa

Commits on Nov 30, 2023

add multi query attn stub

skrider
authored and
Ubuntu
committed
add test for cached multiquery attention

skrider
authored and
Ubuntu
committed
fix slot_mapping construction

skrider
authored and
Ubuntu
committed

Commits on Dec 1, 2023

multi-query cached attention implementation passing tests

skrider
authored and
Ubuntu
committed
remove debugger breakpoints

skrider
authored and
Ubuntu
committed
comments and minor improvement

skrider
authored and
Ubuntu
committed
refactor multi query attention to match attention changes
skrider
committed
vendor nvidia/cutlass at v3.1.0
skrider
committed
copy DeepSpeed-Kernels blocked flash attention
skrider
committed

Commits on Dec 2, 2023

adds flash attention operator stub to _C module
skrider
committed

Commits on Dec 23, 2023

Commits on Dec 24, 2023

kernel launch working; errors with segfualt
skrider
committed

Commits on Jan 9, 2024

Merge branch 'main' into cached-mqa
skrider
committed