vulkan: use spec constant for matrix matrix multiplication A-type - #25773
vulkan: use spec constant for matrix matrix multiplication A-type#257730cc4m wants to merge 1 commit into
Conversation
github-actions
Bot
added
Vulkan
|
I had experimented with something like this and ended up with one spirv for all the legacy and K quants and then separate spirvs for all the iq quants. That resolves the shared memory issue and probably isn't a ton of bloat because those spirvs are dominated by the table size anyway. Fwiw, even if our compiler dead code eliminated the shared memory arrays, they would still count against the spec limit. |
a619971 to
496b099
Compare
|
I think the shared memory arrays with length 1 could run into this compiler bug again #24924. |
|
I don't see these CI failures locally, not sure what is causing them. Is that the compiler bug? |
I also can't reproduce the cm1 failures locally. It's conceivable it's the Turing-only bug. I think it's more likely to happen if you use more fp16 math, is there any place you've changed fp32 math to fp16? |
|
No, the PR should not contain any functional changes, it's all just restructuring. The running code in the end should be basically identical. |
cbf3468 to
001600c
Compare
vulkan: use map for mul_mm shapes cleanup fix indentation fix cm2 and shmem init fix cm2 spec constants fix cm2 bindings consolidate shmem tables and reduce size by type spec constant fix compiler warning fix missing Q2_0 type fix unused warning when integer dot glslc support is missing use minimal shmem size 8 instead of 1 to workaround cm2 compiler bug fix missing Q2_0 type in cm2 matmul fix types
001600c to
197898e
Compare
|
@ggerganov The Nvidia-CM pipeline is timing out in test-backend-ops, I haven't seen that before. Is that expected? Is it possible to increase the timeout? |
|
Hm, not sure - I can't see a reason for the increased time. It almost doubled than usual for this job (~15 mins). Also, looking at the most recent runs in https://github.com/ggml-org/llama.cpp/actions/workflows/build-self-hosted.yml it seems to be fluctuating between slow/fast without a clear point where it started happening. Let's keep monitoring. These workflows are currently running on DGX Spark that I self-host. We are already looking for ways to migrate this to some more stable and scalable infrastructure, but don't have ETA atm. |
|
Did it recently switch from T4 to the Spark? That would explain why it now passed some tests that were failing earlier due (likely to the Turing driver bug. |
|
The DGX Spark has been running these and other jobs for about 3 months now (added on May 26). The T4 runners are also running - whoever runner picks of the job first will execute it. |
Maybe this change effects enough shaders that it get no benefit from the shader disk cache from previous runs? If so, a rerun might be enough to make it complete. |
3 participants
Overview
This is an attempt to reduce the amount of matrix multiplication shaders we need by using a spec constant for the type of the A input, similar to how it works for KV cache types in Flash Attention. Additionally, it replaces the current static s, m and l shapes with a map and allows improved tuning through more shapes and a custom selection function, which should help with work like #24407.
It's still draft because currently it regresses Nvidia, because for dynamic type selection all quants with shared memory requirements must also be available. I thought it would eliminate those that are not used, this works on AMD, but not on Nvidia. @jeffbolznv Do you know why? There are other ways to resolve this, hopefully I'll find something that works.
Requirements