ggml : add ggml_backend_op_alloc_size_may_expand, use it in RPC - #27960
Merged
Conversation
some backends (Metal, SYCL, WebGPU) require additional memory for fleeting data for certain ops, which is reflected in their get_alloc_size implementations. add ggml_backend_op_alloc_size_may_expand() to the backend utils, listing these ops, and assert in ggml_backend_buft_get_alloc_size that a backend expanding the alloc size of a compute op only does so for ops listed in the helper. use the helper in the RPC backend to decide whether to query the remote server for the actual alloc size, instead of a hardcoded list. Assisted-by: pi:llama.cpp/Qwen3.8-27B
ggerganov
marked this pull request as ready for review
August 29, 2026 11:58
Member
Author
|
This fixes Qwen4 garbage generation when running over RPC with Metal. |
github-actions
Bot
added
the
ggml
jbooth
pushed a commit
to jbooth/llama.cpp
that referenced
this pull request
Aug 30, 2026
…-org#27960) some backends (Metal, SYCL, WebGPU) require additional memory for fleeting data for certain ops, which is reflected in their get_alloc_size implementations. add ggml_backend_op_alloc_size_may_expand() to the backend utils, listing these ops, and assert in ggml_backend_buft_get_alloc_size that a backend expanding the alloc size of a compute op only does so for ops listed in the helper. use the helper in the RPC backend to decide whether to query the remote server for the actual alloc size, instead of a hardcoded list. Assisted-by: pi:llama.cpp/Qwen3.8-27B
danielhanchen
added a commit
to unslothai/llama.cpp
that referenced
this pull request
Aug 31, 2026
The pin was 50 commits behind the PR head and sat before "Add MTP support", so the nightly shipped GLM-5-Next without the NextN draft head, without the master merge, and without the pooled-key shift fix. Verified against b10705 by replaying the resolve step: the new commit fetches from ggml-org, is a commit of ggml-org#27754, and merges onto the base plus the pins listed before it with no conflict. Unrelated, and not fixed here: ggml-org#25731 stops merging at b10705. Upstream ggml-org#27960 touched ggml/src/ggml-rpc/ggml-rpc.cpp, which the Inkling branch also edits, and additive_merge.py correctly refuses it. It merges on b10698, the base of the last shipped nightly, so the next run on a newer base will fail there until that branch is merged forward.
ggerganov
added a commit
to ggml-org/ggml
that referenced
this pull request
Sep 4, 2026
This was referenced Sep 4, 2026
Merged
ggerganov
added a commit
that referenced
this pull request
Sep 4, 2026
* ggml : rename and make private ggml_op_alloc_size_may_expand() (ggml/0) cont #27960 * ggml : bump version to 0.23.0 (ggml/1618) * sync : ggml
1 task
fewtarius
pushed a commit
to fewtarius/CachyLLama
that referenced
this pull request
Sep 5, 2026
…-org#27960) some backends (Metal, SYCL, WebGPU) require additional memory for fleeting data for certain ops, which is reflected in their get_alloc_size implementations. add ggml_backend_op_alloc_size_may_expand() to the backend utils, listing these ops, and assert in ggml_backend_buft_get_alloc_size that a backend expanding the alloc size of a compute op only does so for ops listed in the helper. use the helper in the RPC backend to decide whether to query the remote server for the actual alloc size, instead of a hardcoded list. Assisted-by: pi:llama.cpp/Qwen3.8-27B
fewtarius
pushed a commit
to fewtarius/CachyLLama
that referenced
this pull request
Sep 5, 2026
* ggml : rename and make private ggml_op_alloc_size_may_expand() (ggml/0) cont ggml-org#27960 * ggml : bump version to 0.23.0 (ggml/1618) * sync : ggml
alin-o
pushed a commit
to alin-o/llama.cpp-qwen38
that referenced
this pull request
Sep 6, 2026
* ggml : rename and make private ggml_op_alloc_size_may_expand() (ggml/0) cont ggml-org/llama.cpp#27960 * ggml : bump version to 0.23.0 (ggml/1618) * sync : ggml
alin-o
pushed a commit
to alin-o/llama.cpp-qwen38
that referenced
this pull request
Sep 6, 2026
* ggml : rename and make private ggml_op_alloc_size_may_expand() (ggml/0) cont ggml-org/llama.cpp#27960 * ggml : bump version to 0.23.0 (ggml/1618) * sync : ggml
alin-o
pushed a commit
to alin-o/llama.cpp-qwen38
that referenced
this pull request
Sep 6, 2026
* ggml : rename and make private ggml_op_alloc_size_may_expand() (ggml/0) cont ggml-org/llama.cpp#27960 * ggml : bump version to 0.23.0 (ggml/1618) * sync : ggml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Some backends (Metal, SYCL, WebGPU) require additional memory for fleeting data for certain ops, which is reflected in their
get_alloc_sizeimplementations. The RPC backend had a hardcoded list of such ops (FLASH_ATTN_EXT,MUL_MAT_ID): if a backend started expanding the alloc size of another op and the list was not updated, the RPC client would under-allocate.ggml_backend_op_alloc_size_may_expand()to the backend utils, returningtruefor the ops that expand the alloc size in any backend (FLASH_ATTN_EXT,MUL_MAT_ID,CUMSUM,ARGSORT,TOP_K)GGML_ASSERTinggml_backend_buft_get_alloc_size()catching a backend that expands the alloc size of a compute op that is not listed in the helperCUMSUM/ARGSORT/TOP_Kfor e.g. a Metal serverAdditional information
The op list was derived by auditing all
get_alloc_sizeimplementations in the tree. All relevant sites are marked with a[TAG_ALLOC_SIZE_EXPAND]comment so that they can be found easily when adding a new op.The assert exempts non-compute tensors (
ggml_op_is_empty), since type-dependent expansions (e.g. quantized weight padding) apply to weights regardless of op.ref: #15966
Requirements