Rebase onto upstream ggml-org/llama.cpp (2026-07-31) - #65
Merged
Conversation
…olation (ggml-org#25781) The Qwen3-VL learned position embedding is interpolated to the runtime patch grid with the default bilinear+antialias (align_corners=False) sampling, while the transformers reference uses align_corners=True (torch.linspace(0, side-1, T)). The mismatch scales grounding coordinates about the image center, growing with image size and per-axis for non-square images (see ggml-org#16880).
Signed-off-by: wendadawen <wendadawen@qq.com>
* cuda: add k-quant support to GET_ROWS Device-side embedding lookups require GET_ROWS to handle the k-quants used by common GGUF recipes (Q4_K_M stores token_embd as q6_K). Without it the backend rejects the op and the scheduler falls back to the host, copying the full embedding matrix back on every token in single-device graphs. Factor the super-block dequantizers out of the dequantize_block kernels in convert.cu into shared device functions in dequantize.cuh and reuse them from a new k_get_rows_kq kernel : one thread block dequantizes one (dst row, super-block) pair with the existing thread layouts, 32 threads for q4_K and 64 for the other k-quants. Covers q2_K to q6_K in get_rows_cuda and supports_op. i-quants are left as a TODO. * cuda: add i-quant support to GET_ROWS Extends the shared super-block dequantizers to the nine i-quants and reuses them from k_get_rows_kq with the 32-thread layout of the matching convert.cu kernels. supports_op gates the k-quant and i-quant path on ne0 being a multiple of QK_K, which iq4_nl does not guarantee on its own (QK4_NL sub-blocks). mxfp4 is left as a TODO. * cuda: add mxfp4 support to GET_ROWS Moves the mxfp4 dequantizer into the shared super-block helpers and reuses it from k_get_rows_kq with the 32-thread layout of the matching convert.cu kernel. mxfp4 joins the ne0 % QK_K gate in supports_op since its 32-value sub-blocks do not guarantee QK_K-aligned rows on their own. This closes GET_ROWS type coverage on CUDA: every quantized GGML type now takes the direct device path. * cuda: gate the GET_ROWS row size only for 32-value sub-block types Address review from @pwilkin: the i-quant commit replaced the return shared by the whole supported type cascade, so f16/f32/bf16/i32 and the legacy quants also inherited the ne0 % QK_K == 0 gate and any row size that is not a multiple of 256 fell back to the scheduler. Split the cascade: unconditional support is restored everywhere, the gate stays only on iq4_nl and mxfp4 whose 32-value sub-blocks do not guarantee the QK_K super-blocks the kernel iterates on.
* webgpu : add CONV_2D_DW (depthwise conv2d) kernel Implement GGML_OP_CONV_2D_DW for the WebGPU backend, ported from the Vulkan backend's conv2d_dw.comp. Assisted-by: Claude Opus-4.8 * Remove unnecessary comments in webgpu support * update supported ops tables, triggered by adding webgpu CONV_2D_DW
* ggml: enable PowerPC backend variants on AIX Allow the PowerPC CPU backend variants to be built on AIX by extending the platform check in the CMake configuration. This reuses the existing PowerPC backend implementations without changing their behavior. Also fix a missing semicolon in the PowerPC Q0 matmul implementation. * Fix missing semicolon in sgemm.cpp
* chat: fix DS4 template to explicitly follow reference behavior * Support DeepSeekv4 flag (`drop_reasoning`). * fix: hook DS3.2 parser for DS4 as well * fix: add tool result reordering * fix: post-merge
…-org#25989) With -hfd pointing to a repo that ships mtp-/dflash-/eagle3- sidecars and no --spec-type given, the draft resolved to a full model while the sidecar was the intended draft. When the speculative types are still at their default, discover the sidecars of the draft repo, pick the first available following the existing mtp > dflash > eagle3 priority, and set the corresponding type, so this now works without any extra flag: llama-server -hf repo:Q3_K_M -hfd repo:Q8_0 An explicit --spec-type disables the inference, and a draft repo without sidecars keeps resolving to a full model as before.
…org#25948) * feat(ui): add symbolic math support to JS sandbox via nerdamer Preload nerdamer (with decimal.js) in the sandboxed worker, exposing the `nerdamer` global for symbolic computation: simplify, expand, factor, diff, integrate, solve, laplace, ilt, limit, partfrac, gcd/lcm, roots, coefficients, and more. Mirrors the math.js integration pattern from the feature/sandbox-symbolic-math branch, but uses nerdamer for a lighter, more focused symbolic math engine. * Update sandbox-harness.ts * docs(ui): update sandbox tool description with detailed nerdamer usage guide * Clarify nerdamer usage in sandbox tool description Updated the description of the sandbox tool to clarify usage of nerdamer. * ui: build nerdamer sandbox prelude from vendored source Replace the vendored all.min.js with the readable nerdamer-prime source and its two bundled deps (big-integer, decimal.js), licenses included. A vite plugin bundles and minifies them at build time with the upstream esbuild flags, exposed as virtual:nerdamer and imported lazily on first sandbox use. The vendors package.json pins commonjs so the project level type: module does not break esbuild format detection. The harness gains a CSP removing network egress from the worker, and browser tests cover the prelude, exact arithmetic, the fetch block and the timeout. Upstream snapshot: together-science/nerdamer-prime@1936145 * feat(ui): make symbolic math (nerdamer) a user-toggleable setting - Add SYMBOLIC_MATH_ENABLED setting key and registry entry (checkbox, default false) - Convert SANDBOX_TOOL_DEFINITION to buildSandboxToolDefinition(includeSymbolicMath) so the tool description includes/excludes nerdamer API docs dynamically - Cache sandbox harness per variant ('nerdamer' / 'plain') for instant toggle - Deprecate SANDBOX_TOOL_DEFINITION constant alias for backward compatibility - Update tools store to pass symbolic math config into tool definition * docs(ui): tell LLM to list nerdamer functions first, do not guess * test(ui): enable symbolic math in sandbox tests via settingsStore config * style(ui): fix formatting for tools.svelte.ts --------- Co-authored-by: Pascal <admin@serveurperso.com>
…rg#25723) * mtmd: use RAII for setting and resetting non-causal attention * mtmd: drop dependency on <optional> * mtmd: shorten class and variable names
* hex-geglu: optimized all-in-one geglu microkernel * hex-geglu: enable non-contiguous src and strided DMA * hex-act: enable non-contiguous srs and strided DMA for rest of ACT ops * hex-act: generalize GLU per-thread functions via DEFINE_GLU_PER_THREAD macro * hexagon: move UNARY_SILU and UNARY_GELU to unary-ops * hex-act: replace the generic ops_context scratchpad usage with a local htp_vtcm_layout computation per act op. --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* Squash history before conflict-resolution during rebase on master WIP commit Add 32-byte loads, restore per-block amax Use nvfp4x4 intrinsic when available Fuse per-channel amax and quantization kernels Do pointer arithmetic only once on x Remove unnecessary ternary in the load We assert on host side that ne00 is 64-aligned Add back scale-search, but optimize it with intrinsics Code cleanup Make scale in MMQ-epilogue NVFP4-specific/restrictive for now Remove unneeded include, add comment Fix trailing whitespace Guard __builtin_align__(32) struct to NVIDIA Seems like HIP doesn't have this available, see https://github.com/ggml-org/llama.cpp/actions/runs/29438651734/job/87431623001 * compiler massaging to avoid unnecessary LDCs * kvalues_mxfp4 -> kvalues_nvfp4 in quantize_mmq_nvfp4 * Always pass in src1_scale.ptr * Extract ggml_cuda_is_aligned helper
* ui: add Default reasoning option that defers to the server The webui always injected enable_thinking, overriding the chat template default and the --reasoning flag, breaking models that reason unconditionally (e.g. Gemma 4 E4B) on a fresh client. Default sends nothing so the server decides, Off and effort levels force the value as before. All choices are remembered. Also remove the boolean thinking API from the conversations store and drop ChatFormReasoningEffortSubmenu.svelte (dead code). * ui: close the whole menu tree on reasoning level selection The reasoning levels were raw buttons inside the SubContent, so selecting one only closed the submenu via manual state while the root dropdown stayed open. DropdownMenu.Item closes the full tree on select like the sibling entries and brings native keyboard navigation. * ui: prevent the add menu tooltip from flashing when the dropdown closes
* args: overhaul mmap/mlock/dio into single arg Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: update docs with llama-gen-docs Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * chore: satisfy code quality Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * args: make the `+` sign an actual modifier now Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * chore: general code clean up + comments Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: fix deprecated flags support Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: quick sanity check Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * bench: sync llama-bench argument parsing Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * fix: bugfix variable behaviour + llama-bench lm column size Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: inverse commands should do the opposite instead of doing nothing Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * bench: fix incorrect dash Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * bench: fix missing modifiers for deprecated flags Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * llama: switch back to thread_local Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: switch back to single enum Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: update arg docs Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * chore: fix missing `mlock` from llama_load_mode_from_str + cleanup llama-bench Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * llama: fix mlock not activating Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: add deprecation warning when old and new flags are combined Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: cont add comment for todo in the future Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: sync with upstream Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: re-sync with upstream again Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> --------- Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
…ggml-org#26049) * hex-l2: use dirty ranges for flushing * hex-l2: simplify range based flush logic * hex-l2: optimize dirty range scans * hex-hvx: support for reduce_max_i32 * hex-mm: optimize fused MUL_MAT+ADD to use vtcm for bias when it fits * hex-mmid: optimize mmid row-mapping generation * hex-mmid: optimize mmid row-mapping generation * hex-mmid: optimize mmid row-mapping generation (round2) * hmx-mm: optimize output proc by tiling (col-chunking) * hex-fa: start the next q dmas a bit earlier * hex-fa: prefetch Q even earlier * hvx-fa: optimize softmax to keep things in hvx registers * hex-fa: hoist const register init in softmax loop * hmx-fa: kick off next-qkv DMAs before o-proc * hmx-fa: hoist various checks out of the inner loop * hmx-fa: adjust the cost model to better balance softmax work across hvx threads * hmx-fa: overlap diag rescale build with last HMX task * hmx-fa: optimize idx update in output proc * hmx-fa: unroll the softmax loops for improved perf * hmx-fa: overlap qk-dot with softmax, double-buffer p and s tiles * hex-trace: double the default number of trace entries * hex-trace: add trace events for opbatch and buffer mgmt * hex-trace: overhaul tracing to simplify runtime event handling and support opbatch stats * hex-trace: replace ascii timeline diagram with pipeline bubbles detector * hex-trace: handle missing start/stop events * hex-dma: always log stop/start trace events even for dummy dmas * hex-scripts: fix flake warnings
…f a response schema is provided (ggml-org#26018)
…rs (Settings panel) (ggml-org#26002)
* skill: add new model * add common pitfalls * add code review skill * nits * add codeowners * add security review section * mention about skills in agents.md * add design review * exclude ggml-gh-bot * Apply suggestions from code review Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com> * conversion-time weight modification * nits --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
…vely) + fix op count assert
…o_eff ordering in fattn.cu
…ific ops post-rebase - Add TQ3_4S/TQ3_1S/TQ3_0/TURBO2-4/Q4_0_TQ/Q4_1_TQ type_traits[] entries in ggml.c - Add quantize dispatch cases for all TQ3/Turbo types - Add GGML_OP_TURBO_WHT op name, symbol, and implementation - Add TQ3_4S CASE entries to mmq-config-ampere.cuh (fixes J_best=0 crash) - Fix ggml_cuda_should_use_mmq signature (tensor* not type enum) - Remove upstream placeholder slots [31-33,36-37] reassigned to TQ3 types - Fix static asserts to match actual GGML_OP_COUNT (102) - Restore server-context.cpp from fork (upstream+delta had 10 failed hunks) - Fix duplicate enum entries in llama-arch, llama-model, llama-vocab - Restore common/chat.cpp, common/common.h with fork MTP deltas
# Conflicts: # .gitignore # common/chat.cpp # common/common.h # ggml/include/ggml-rpc.h # ggml/include/ggml.h # ggml/src/ggml-cuda/fattn-common.cuh # ggml/src/ggml-cuda/fattn.cu # ggml/src/ggml-cuda/getrows.cu # ggml/src/ggml-cuda/ggml-cuda.cu # ggml/src/ggml-cuda/mmq.cu # ggml/src/ggml-cuda/mmq.cuh # ggml/src/ggml-cuda/quantize.cu # ggml/src/ggml-cuda/quantize.cuh # ggml/src/ggml-cuda/set-rows.cu # ggml/src/ggml-metal/ggml-metal-device.cpp # ggml/src/ggml-metal/ggml-metal-device.h # ggml/src/ggml-metal/ggml-metal-device.m # ggml/src/ggml-metal/ggml-metal-ops.cpp # ggml/src/ggml.c # src/llama-arch.cpp # src/llama-context.h # src/llama-hparams.h # src/llama-model-loader.cpp # src/llama-quant.cpp # tests/test-backend-ops.cpp # tests/test-quantize-fns.cpp # tools/server/server-context.cpp # tools/server/tests/utils.py
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
Assisted-by: Codex
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Full rebase of
mainonto latest upstreamggml-org/llama.cpp(141 fork commits replayed).What's included
Conflict resolution strategy
Testing