model : allow reshape of tensors during load - #26531
Merged
Merged
Conversation
Contributor
|
@ggerganov could this have caused a speed regression on the new deepseek? I think I narrowed it down to this commit. My t/sec dropped by more than 50%. b10267 : speed regression Edit: --no-repack fixes the speed. |
danielhanchen
pushed a commit
to unslothai/unsloth
that referenced
this pull request
Aug 6, 2026
* Add opt-in DSpark speculative decoding * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix lint blockers, retry-guard test and DSpark broken-build gate for PR #7968 Three fixes for the failing CI on this branch: - Drop the unused `Literal` import from unsloth_cli/commands/chat.py and inference.py. The options are typed with `SpeculativeType`, so the added import was a leftover and tripped the import-hoist blocker. - Make test_the_startup_retry_drops_the_mtp_the_extras_and_the_env_carry whitespace insensitive. The guard now names both drafters, so formatting wrapped the call and the literal substring assertion no longer matched. It asserts on both `_extra_args_requests_mtp` and `_extra_args_requests_dspark` now, so the DSpark half is covered too. - Gate DSpark on the whole broken build window instead of one tag. The reshape regression is ggml-org/llama.cpp#26531 and the fix is #26577, so every prebuilt based on b10259 through b10268 aborts on a DSpark load, not only b10265-mix-89aa77b. Matching the base build number keeps source builds unaffected, since those carry no install marker. * Probe DSpark support before downloading the sidecar The ~11 GB DSpark sidecar was fetched at llama_cpp.py:8664 while the first supports_dspark consumer sat ~480 lines later, so a binary that cannot run draft-dspark paid for the whole download and then fell back without ever opening the file. probe_server_capabilities is already called just above for supports_kv_unified, so the answer is in scope and cached and the check costs nothing. This is the default path right now, not an edge case: the shipped unslothai/llama.cpp prebuilt b10265-mix-89aa77b sits inside the known-broken b10259..b10268 window, so supports_dspark is False on a standard install. Also swaps the order of the first two DSpark fallbacks. Now that the fetch is gated on the same answer, a gated binary leaves no sidecar, and checking the drafter first reported "no matching dspark-*.gguf sidecar was found" and told the user to place a file that was never the problem, while re-loading on every Apply through the drafter_not_found dedup branch. Adds three regression tests, all of which fail without this change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Enforce fixed fit for pass-through DSpark, size split sidecars, correct the hint Three fixes from the latest review round. Extras that own --spec-type return from _build_speculative_flags before _speculative_type is set, so the --fit strip keyed only on that field never fired for a pass-through DSpark launch and a user --fit on survived. DSpark's layout cannot be reshaped, so that aborts the load. The strip now also reads the accumulated spec types, which covers both the flag and the env. The training coexistence estimate sized the drafter with a bare stat(), while the main weight beside it already used the split-aware helper. Discovery hands back shard 1, so a split sidecar was counted at one shard and the guard could admit a load that evicts the training run it exists to protect. The Speculative Decoding hint promised "no accuracy hit" unconditionally, which DSpark does not meet: on a quantized target its greedy output can differ from a non speculative run (ggml-org/llama.cpp#25618). Measured here on DeepSeek-V4-Flash-0731 UD-Q4_K_XL, where the same greedy conversation produced 10570 tokens without a drafter and 14687 with one. The claim now stays with Auto, and DSpark carries its own caveat. Both backend fixes have regression tests that fail without them. * Pin fit off for pass-through DSpark, keep cached sidecars visible, reclaim them on delete * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Allow DSpark under --fit on: fitting only skips the sidecar reserve * Gate the training-guard DSpark estimate on binary support, fix the picker contract marker * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Default Auto to DSpark when a sidecar is available, and stop the two reload loops it exposed * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Charge no drafter when forced DSpark is gated off by the binary * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Gate remote Auto DSpark sizing, retry only a failed sidecar fetch, refresh the Auto hint * Apply ruff kwarg-spacing formatting --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <unslothai@gmail.com>
Nathanw1014
added a commit
to Nathanw1014/llama.cpp
that referenced
this pull request
Aug 9, 2026
TENSOR_ALLOW_RESHAPE (ggml-org#26531) recomputes nb from the caller's shape as nb[dim] = dim == 0 ? ggml_type_size(type) : ne[dim-1]*nb[dim-1] so nb[1] comes out as ne[0]*nb[0]. A row of a block-quantised tensor is ne[0]/blck_size blocks, not ne[0] elements, so for q8_0 with ne[0]=4096 the stride is 4096*34 = 139264 instead of 128*34 = 4352. F16/F32 are unaffected because their block size is 1, which is why this went unnoticed. The malformed meta is what buft_supported() probes the backend with, so the weight looks non-contiguous: ggml_vk_dim01_contiguous() is false, the Vulkan supports_op refuses the mul_mat (quantised src0 has no F32/F16/BF16 bypass), and the loader falls back to a CPU buffer for that weight. Reachable since ggml-org#26577 gave dflash and deepseek4 a 3d wo_a. On DeepSeek-V4-Flash UD-IQ3_XXS + DSpark, gfx1151/RADV, that put attn_wo_a on the CPU backend for all 43 attention layers, moved 1462 MiB of weights off the GPU (Vulkan0 model buffer 6624.99 -> 5162.99 MiB) and took decode splits from 88 to 174, which is a large host-side cost per token with the GPU idle. With this patch the load-time probe sees nb=[34,4352,...], the demotion is gone, splits are back to 88 (bs=1) and 2 for the draft context, and generation is unchanged. The same expression in the dummy-tensor path above is corrected too; it is latent there, not reached by this path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
satindergrewal
pushed a commit
to satindergrewal/llama.cpp
that referenced
this pull request
Aug 12, 2026
brittlewis12
pushed a commit
to brittlewis12/llama.cpp
that referenced
this pull request
Aug 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
fix #25837
fix #26521
fix #26537
Allow reshaping of the weight tensors at loading time. This is useful in cases where the stored shape in the model file is not suitable for the ggml operation in the graph that uses it. Instead of reshaping in the graph with
ggml_reshape_nd(), we can now do it during loading and avoid this extra node.This is also correct because it respects the
llm_tensor_info. Otherwise, a reshape could cause a repacked weight to no longer be computable, causing the issues above.Additional information
Also remove unused
llama_model_loader::create_tensor_as_view()Requirements