llama : add llama_model_ftype_name() - #25134
Conversation
Expose the model file type (quantization) name, e.g. "Q8_0" or "Q4_K - Medium", through a new public C API. The returned pointer is valid for the lifetime of the model and nullptr when the model is invalid or the file type is unknown. Signed-off-by: Adrien Gallouët <angt@huggingface.co>
|
no strong opinion about this feature, but IMO the more important thing is to get the for example: if I run without the tag: |
I'll make a PR for that, but I still think this one is useful for the mentioned issue |
ngxson
left a comment
There was a problem hiding this comment.
sounds ok to me, not sure if there are any edge cases, but let's see
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Prepend the "(guessed)" label instead of appending it. This allows removing the non-thread-safe static std::string, making the function allocation-free. Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
llama : add llama_model_ftype_name() (ggml-org#25134)
30 commits from ggml-org/llama.cpp master. Two conflicts resolved by combining our changes with upstream work that did not deprecate them: ggml/src/ggml-cuda/gated_delta_net.cu Upstream (ggml-org#23940) fused the post-GDN cpy into the kernel itself by passing state_d/state_slot_stride through the kernel signature so recurrent-state snapshots land directly in the cache during MTP decode. Our RDNA3.5 tuning (num_warps=8 for S_v=128 non-KDA paths) generalized the kernel template on num_warps and dispatched in launch_gated_delta_net. Both coexist: the upstream signature is preserved, the RDNA3.5 dispatch still routes to the 8-warp variant on gfx1151. tools/server/server-context.cpp Upstream (ggml-org#25241) promoted sse_ping_interval to task_params so the WebUI can request a 1s ping cadence per request while CLI clients keep the 30s default. Our per-user concurrency cap (max_concurrent_per_user, 429 fast-fail) lives in the same function but in disjoint code paths; both retained. Also carried (no conflicts): - MCP server recommendations UI + first-time opt-in - AVX2 NVFP4 dot product on CPU - OpenCL Q1_0 GEMM/GEMV kernels + precompiled-binary support - Hexagon flash-attn rework (kernels split into .h headers) - Vulkan bk-loop matmul for Asahi, flops-based heuristic - WebGPU NVFP4 support - chat: trim messages sent to StepFun parser (ggml-org#25238) - llama_model_ftype_name() (ggml-org#25134) - HTTP/URL bracketed IPv6 literal handling - 25+ other backend, model-arch, and tooling changes Test results: 51/51 main+model tests pass after re-fetching LFS vocab files (test-tokenizers-ggml-vocabs was a stale-pointer issue unrelated to the merge). test-jinja-py / test-quant-type-selection failures are pre-existing environment/snapshot issues. Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> Co-authored-by: Pascal <admin@serveurperso.com>
30 commits from ggml-org/llama.cpp master. Two conflicts resolved by combining our changes with upstream work that did not deprecate them: ggml/src/ggml-cuda/gated_delta_net.cu Upstream (ggml-org#23940) fused the post-GDN cpy into the kernel itself by passing state_d/state_slot_stride through the kernel signature so recurrent-state snapshots land directly in the cache during MTP decode. Our RDNA3.5 tuning (num_warps=8 for S_v=128 non-KDA paths) generalized the kernel template on num_warps and dispatched in launch_gated_delta_net. Both coexist: the upstream signature is preserved, the RDNA3.5 dispatch still routes to the 8-warp variant on gfx1151. tools/server/server-context.cpp Upstream (ggml-org#25241) promoted sse_ping_interval to task_params so the WebUI can request a 1s ping cadence per request while CLI clients keep the 30s default. Our per-user concurrency cap (max_concurrent_per_user, 429 fast-fail) lives in the same function but in disjoint code paths; both retained. Also carried (no conflicts): - MCP server recommendations UI + first-time opt-in - AVX2 NVFP4 dot product on CPU - OpenCL Q1_0 GEMM/GEMV kernels + precompiled-binary support - Hexagon flash-attn rework (kernels split into .h headers) - Vulkan bk-loop matmul for Asahi, flops-based heuristic - WebGPU NVFP4 support - chat: trim messages sent to StepFun parser (ggml-org#25238) - llama_model_ftype_name() (ggml-org#25134) - HTTP/URL bracketed IPv6 literal handling - 25+ other backend, model-arch, and tooling changes Test results: 51/51 main+model tests pass after re-fetching LFS vocab files (test-tokenizers-ggml-vocabs was a stale-pointer issue unrelated to the merge). test-jinja-py / test-quant-type-selection failures are pre-existing environment/snapshot issues.
32 commits from ggml-org/llama.cpp master. Two conflicts resolved by combining our changes with upstream work that did not deprecate them: ggml/src/ggml-cuda/gated_delta_net.cu Upstream (ggml-org#23940) fused the post-GDN cpy into the kernel itself by passing state_d/state_slot_stride through the kernel signature so recurrent-state snapshots land directly in the cache during MTP decode. Our RDNA3.5 tuning (num_warps=8 for S_v=128 non-KDA paths) generalized the kernel template on num_warps and dispatched in launch_gated_delta_net. Both coexist: the upstream signature is preserved, the RDNA3.5 dispatch still routes to the 8-warp variant on gfx1151. tools/server/server-context.cpp Upstream (ggml-org#25241) promoted sse_ping_interval to task_params so the WebUI can request a 1s ping cadence per request while CLI clients keep the 30s default. Our per-user concurrency cap (max_concurrent_per_user, 429 fast-fail) lives in the same function but in disjoint code paths; both retained. Also carried (no conflicts): - MCP server recommendations UI + first-time opt-in - AVX2 NVFP4 dot product on CPU - OpenCL Q1_0 GEMM/GEMV kernels + precompiled-binary support - Hexagon flash-attn rework (kernels split into .h headers) - Vulkan bk-loop matmul for Asahi, flops-based heuristic - WebGPU NVFP4 support - chat: trim messages sent to StepFun parser (ggml-org#25238) - llama_model_ftype_name() (ggml-org#25134) - HTTP/URL bracketed IPv6 literal handling - 25+ other backend, model-arch, and tooling changes Test results: 52/52 main+model tests pass after re-fetching LFS vocab files (test-tokenizers-ggml-vocabs was a stale-pointer issue unrelated to the merge). test-jinja-py / test-quant-type-selection failures are pre-existing environment/snapshot issues. Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> Co-authored-by: Pascal <admin@serveurperso.com>
32 commits from ggml-org/llama.cpp master. Two conflicts resolved by combining our changes with upstream work that did not deprecate them: ggml/src/ggml-cuda/gated_delta_net.cu Upstream ggml-org#23940 fused the post-GDN cpy into the kernel itself by passing state_d/state_slot_stride through the kernel signature so recurrent-state snapshots land directly in the cache during MTP decode. Our RDNA3.5 tuning (num_warps=8 for S_v=128 non-KDA paths) generalized the kernel template on num_warps and dispatched in launch_gated_delta_net. Both coexist: the upstream signature is preserved, the RDNA3.5 dispatch still routes to the 8-warp variant on gfx1151. tools/server/server-context.cpp Upstream ggml-org#25241 promoted sse_ping_interval to task_params so the WebUI can request a 1s ping cadence per request while CLI clients keep the 30s default. Our per-user concurrency cap (max_concurrent_per_user, 429 fast-fail) lives in the same function but in disjoint code paths; both retained. Also carried (no conflicts): - MCP server recommendations UI + first-time opt-in - AVX2 NVFP4 dot product on CPU - OpenCL Q1_0 GEMM/GEMV kernels + precompiled-binary support - Hexagon flash-attn rework (kernels split into .h headers) - Vulkan bk-loop matmul for Asahi, flops-based heuristic - WebGPU NVFP4 support - chat: trim messages sent to StepFun parser (ggml-org#25238) - llama_model_ftype_name() (ggml-org#25134) - HTTP/URL bracketed IPv6 literal handling - 25+ other backend, model-arch, and tooling changes Test results: 52/52 main+model tests pass after re-fetching LFS vocab files (test-tokenizers-ggml-vocabs was a stale-pointer issue unrelated to the merge). test-jinja-py / test-quant-type-selection failures are pre-existing environment/snapshot issues.
* llama : add llama_model_ftype_name() Expose the model file type (quantization) name, e.g. "Q8_0" or "Q4_K - Medium", through a new public C API. The returned pointer is valid for the lifetime of the model and nullptr when the model is invalid or the file type is unknown. Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Export enum Signed-off-by: Adrien Gallouët <angt@huggingface.co> * s/llama_model_ftype_name/llama_ftype_name/ Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Move "(guessed)" to the front in llama_ftype_name Prepend the "(guessed)" label instead of appending it. This allows removing the non-thread-safe static std::string, making the function allocation-free. Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Add LLAMA_FTYPE_PREFIX Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Dont check for model Signed-off-by: Adrien Gallouët <angt@huggingface.co> --------- Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* llama : add llama_model_ftype_name() Expose the model file type (quantization) name, e.g. "Q8_0" or "Q4_K - Medium", through a new public C API. The returned pointer is valid for the lifetime of the model and nullptr when the model is invalid or the file type is unknown. Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Export enum Signed-off-by: Adrien Gallouët <angt@huggingface.co> * s/llama_model_ftype_name/llama_ftype_name/ Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Move "(guessed)" to the front in llama_ftype_name Prepend the "(guessed)" label instead of appending it. This allows removing the non-thread-safe static std::string, making the function allocation-free. Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Add LLAMA_FTYPE_PREFIX Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Dont check for model Signed-off-by: Adrien Gallouët <angt@huggingface.co> --------- Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* llama : add llama_model_ftype_name() Expose the model file type (quantization) name, e.g. "Q8_0" or "Q4_K - Medium", through a new public C API. The returned pointer is valid for the lifetime of the model and nullptr when the model is invalid or the file type is unknown. Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Export enum Signed-off-by: Adrien Gallouët <angt@huggingface.co> * s/llama_model_ftype_name/llama_ftype_name/ Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Move "(guessed)" to the front in llama_ftype_name Prepend the "(guessed)" label instead of appending it. This allows removing the non-thread-safe static std::string, making the function allocation-free. Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Add LLAMA_FTYPE_PREFIX Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Dont check for model Signed-off-by: Adrien Gallouët <angt@huggingface.co> --------- Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Overview
Add llama_model_ftype_name()
Additional information
Expose the model file type (quantization) name, e.g. "Q8_0" or "Q4_K - Medium", through a new public C API. The returned pointer is valid for the lifetime of the model and nullptr when the model is invalid or the file type is unknown.
See #25020
Requirements