IK's experiments with DS4 - #2147
Conversation
…oftplus gating function
# Conflicts: # convert_hf_to_gguf.py # ggml/src/ggml.c # src/CMakeLists.txt # src/llama-build-context.cpp # src/llama-build-context.h # src/llama-hparams.cpp # src/llama-load-tensors.cpp
… post-processing with new helper functions
This reverts commit 04f9b42.
…a-dflash.cpp (#2134) llama-dflash.cpp calls ggml_backend_is_metal() and ggml_backend_metal_set_n_cb() inside an #ifdef GGML_USE_METAL block but never includes ggml-metal.h, so any Metal-enabled build fails to compile. Add the same guarded include llama.cpp already uses.
* Add ggml_sum_rows_ext * openPangu: use ggml_sum_rows_ext also in mhc_post * openPangu: use ggml_sum_rows_ext also in mhc_tail * Minor
|
I tested it against the PR draft and even though it’s on Windows, which is much slower, I saw 39% better performance using this PR. Regarding SWA it does indeed have the issue you pointed out there. Since it’s an open issue, if you’d like, we can proceed with this PR, resolve the SWA issue, reapply the fixes from the commits that are in the draft but didn’t make it here, of course, with the necessary adjustments to prevent regression, and them move forward with any remaining pending issues. Regarding the implementation of Sinkhorn/HCA, did you check to see if the PPL changed? I didn’t have time to test it in this PR but I hope it doesn’t cause the same problem I encountered when I tried it in the draft. |
Obviously. Especially after the claim in the other PR that using I think this branch is in a good shape and as everything that it needs to have for now. I cherry-picked some but not all of your changes in the last round of edits of #2110 |
|
edit: oops, I now see the example args above |
|
I compared decode throughput between
(afaik FD's fork is the best available "mainline" version, but maybe that has changed in the last few days) First I sweep for ik_llama's optimal thread count for the MXFP4 FFN, and I find t=64 to be optimal on my hw. I'm interested in cpu-moe decode perf only, so I measure at low (0-100) ctx.
I notice that llama.cpp doesn't warm up the ffn tensors the way ik_ does, so I run a number of "tell me a story" prompts until TG rate settles. CPU is 2x EPYC 9B14 w/24x DDR5 4800. ik_llama.cpp ds4_play1 = 28.37t/s and if one doesn't mind running a mysterious closed-source binary : llama.cpp shows about 20% improvement in decode throughput at 0-100ctx. |
Since Mainline merged DSV4 a few weeks ago they've had time to make optimizations. See ggml-org/llama.cpp#25370, ggml-org/llama.cpp#24231, ggml-org/llama.cpp#25545, and ggml-org/llama.cpp#25585. We may already have many of these as Sinkhorn, Indexer, etc., but that doesn’t mean we’ll have a PR that can implement everything from scratch with the best possible performance (although Ik is already optimizing several factors). |
|
@SamuelOliveirads sorry, to be clear it's fairydreaming llama.cpp which I think has the advance PRs applied, I said this at the top but omitted it in the line you quoted also the goal was just to bench the MXFP4 FFN so DSA/attention/etc. impact should be minimal at 0-100tok context |
|
No need to worry, I just wanted to point out that from the perspective of implementing the architecture and having the functional model in place this PR has already achieved that, in my view, it’s ready for a merge. |
|
|
I didn't use routed expert offloading for any of those measurements nor MTP. (with MTP, I observe lvllmds4 reach 60-70t/s decode, all other params unchanged. It's quite impressive excepting the binary situation.) My intent is to measure as best as possible the routed MoE/FFN throughput (i.e. CPU's part of the attn/moe duty cycle) only by comparing e.g. 0-100 ctx range decode and all _exps tensors on CPU. The economics of sparse MoE for single/low user are terrible wrt. putting _exp layers in $VRAM. (Kimi K2.x being the most extreme example at 8/384 2% density, K3 worse at 16/896 1.8% density) I'll wait until ds4 support is merged to do a more thorough exposition of the figures I gave above, perhaps in discussions. |
|
Closing in favor of #2165 |
4 participants
I started from commit
3d6acd350d781e7bb527f5b59aa506234bf7f113in PR #2110 and added a few optimizations.After the latest commits:
llama.cpp. The issue was that the SWIGLU limits were not taken into accountThe missing piece is that SWA is not taken into account. Or rather, it is taken into account by using the appropriate masks, but the masks are built in such a way that the usual SWA tricks that
ik_llama.cppimplements cannot be used. This makes performance decrease with context much more than it should.In any case, here is what I get with the latest version using the
Q4_Kmodel from antirez (the model is 164.6 GB).Hybrid
2x3090+Ryzen-3995WX. Didn't bother trying to have some experts on the GPUs.
CPU only