vulkan: roll bk loop in matmul for asahi linux - #24663
Conversation
github-actions
Bot
added
Vulkan
|
While I'm OK with this change in theory, we have a problem with the number of mul_mm variants. I'd rather fix this by editing the SPIR-V when we create the pipeline, similar to what we do for ggml_vk_strip_decode_vector, for example. Do you want to make this change? If you're not comfortable doing it, I can do it. |
Yes I can try tonight. |
|
@jeffbolznv Hi, I implemented the change as requested. Currently I tried to limit my change to only editing the spirv during pipeline creation and the performance is identical to the previous version. |
jeffbolznv
left a comment
jeffbolznv
left a comment
There was a problem hiding this comment.
Overall I think this looks correct, though it may be fragile if we rewrite mul_mm.
| } | ||
|
|
||
| auto encloses = [](const vk_spv_loop & a, const vk_spv_loop & b) { | ||
| return a.header < b.header && b.header < a.end; |
There was a problem hiding this comment.
This assumes the blocks are laid out in an order that matches the control flow nesting, which is not strictly required by SPIR-V but I think is true for glslang, so this should be fine.
| // hardware like Apple M1/M2. | ||
| // Assumes 1. code comes from mul_mm.comp 2. the K-tile loop has no loop | ||
| // control hint and 3. the BK loop is the last loop nested directly inside | ||
| // the K-tile loop. |
There was a problem hiding this comment.
There is whitespace at the end of some lines, which CI will reject,
* vulkan: roll bk loop in matmul for asahi linux * vulkan: fix inline comment * vulkan: revert BK-loop unroll change * vulkan: edit spirv directly for asahi roll bk loop * vulkan: remove trailing whitespace at the end of comments
* vulkan: roll bk loop in matmul for asahi linux * vulkan: fix inline comment * vulkan: revert BK-loop unroll change * vulkan: edit spirv directly for asahi roll bk loop * vulkan: remove trailing whitespace at the end of comments
* vulkan: roll bk loop in matmul for asahi linux * vulkan: fix inline comment * vulkan: revert BK-loop unroll change * vulkan: edit spirv directly for asahi roll bk loop * vulkan: remove trailing whitespace at the end of comments
* vulkan: roll bk loop in matmul for asahi linux * vulkan: fix inline comment * vulkan: revert BK-loop unroll change * vulkan: edit spirv directly for asahi roll bk loop * vulkan: remove trailing whitespace at the end of comments
* vulkan: roll bk loop in matmul for asahi linux * vulkan: fix inline comment * vulkan: revert BK-loop unroll change * vulkan: edit spirv directly for asahi roll bk loop * vulkan: remove trailing whitespace at the end of comments
3 participants
Overview
Removing the unroll tag on the fma loop in mul_mm.comp gives a big performance boost on Asahi Linux, possibly because the unrolled loop body is otherwise too large to fit in Apple's instruction cache (12 KB). This PR adds an UNROLL macro in mul_mm.comp and generates rolled and unrolled variants at build time. It then only selects the rolled variant at pipeline-creation time for Asahi Linux.
related results from test-backend-ops:
matmul_perf_table.txt
see #10982 for full discussion.
Requirements