Commit 24e86ca
authored
vulkan: KHR_coopmat flash attention (#13506)
This shader uses coopmat1 to do the Q*K^T multiply. The P*V multiply is more
difficult for various reasons so I haven't done it. Performance for this
shader is around 2.5x better than for the scalar shader when doing prompt
processing. Some of the benefit may be from other optimizations like staging
through shared memory, or splitting by rows.1 parent bb1681f commit 24e86ca
File tree
3 files changed
+702
-54
lines changed- ggml/src/ggml-vulkan
- vulkan-shaders
3 files changed
+702
-54
lines changed
0 commit comments