model: qwen4exp: reduce number of graph splits - #27880
Conversation
Cherry-pick of ggml-org/llama.cpp PR ggml-org#27880 (aa1b06d, ngxson). Regroups the PLE embd lookup into the same split as build_inp_embd, reducing graph splits (4->2 full offload, 7->5 mixed CPU/GPU). Fewer split boundaries = fewer H2D/D2H transfers + sync points.
This comment was marked as off-topic.
This comment was marked as off-topic.
|
Measured this on AMD Strix Halo (Ryzen AI MAX+ 395 / gfx1151, RADV, Mesa 26.1.7) with Qwen3.8-Flash-Next UD-Q4_K_XL. It is a two-way trade on this hardware rather than a straight win — helping short generations and costing long ones by about the same margin. Setup:
Reverting this commit alone reproduces the pre-commit numbers to within 0.1% on both cases, so it is the only variable among the 15 commits between the two baselines I compared. #26686 (shader hoisting for row IDs / expert count) has no measurable effect on this model. +4.8% on short generation, −4.7% on long generation. I have not established the mechanism and would rather not guess — graph node count and split count are unchanged in my logs (8642 nodes, 28 splits at Not asking for a revert — the short-generation case is the more common one in my own traffic and I am shipping with this commit. Flagging it in case the long-generation regression is unintended, or in case it is worth a per-backend guard. Happy to run further measurements on this hardware if useful. |
(cherry picked from commit 6fe7498)
Ports every upstream qwen4exp (Qwen3.8-Flash-Next) commit from the past 10 days that this fork's manual PR port had not received: - reduce graph splits by hoisting the PLE embedding gather out of the per-layer loop (ggml-org#27880) - sum indexer heads via strided adds instead of transpose+sum_rows (ggml-org#28023) - support recurrent state rollback for MTP speculative decoding (ggml-org#28123) - rewrite QSA sparse-attention block/bias selection: fixes NaN-producing bias rows for short sequences, fixes cross-sequence block pooling in a unified KV cache, adds mrope duplicate-position ranking, and fixes a CUDA rms_norm gridDim.y overflow (ggml-org#27941) - indexer cache seq_cp staleness fix, ext.x/ext.y state-restore fix, PLE-must-be-linear-attention validation, correct -sm tensor disablement (ggml-org#27941) - Hadamard k_rot context-shift crash fix, shared with other archs (ggml-org#27967) Also replaces raw GGML_ASSERT aborts in hparams loading with proper error messages, and adds test coverage: a PLE fixture in test-llama-archs (which required porting the per_layer_token_embd row-count-from-metadata fix to make it loadable) and a state round-trip test in test-save-load-state. Verified against the real Qwen3.8-Flash-Next model: correct generation at short and long (~66k token) context, and test-llama-archs passes qwen4exp on both CUDA and CPU.
5 participants
Overview
As discussed with @ggerganov , we can regroup the PLE embd lookup to the same split as build_inp_embd, similar to #21612
Validation
On my mac M5:
Mac + CUDA via RPC:
llama-bench -p 128 -n 32 -r 3Requirements