ggml : update ggml_prec specification by ggerganov · Pull Request #26675 · ggml-org/llama.cpp

ggerganov · 2026-08-06T11:42:12Z

Overview

The discussion in #24364 has most of the background for this change. In short - this is a proposal to improve the specification of enum ggml_prec and how it is used to declare the accumulator types and the internal source data representation types used by the implementations (i.e. the ggml backends).

Sample usage:

// matrix multiplication that:
//  - can accumulate in F16
//  - can quantize src1 on-the-fly to GGML_TYPE_Q8_0, but cannot do so to GGML_TYPE_NVFP4
ggml_tensor * z = ggml_mul_mat(ctx, x, y);
ggml_prec_set_acc(z, GGML_PREC_F16);
ggml_prec_set_src(z, GGML_PREC_Q8, 1);

@ggml-org/maintainers PTAL at the comments in the code and let me know what you think.

Additional information

The proposal atm is backwards compatible and AFAICT it covers all concerns and remarks that we have had over time about how to communicate the types used during compute. Note that as proposed in #24364, there is going to be an user-level policy (in the llama.cpp case, this is optional meta information stored in the GGUFs) that will be used to provide this precision information for the various ops in the model's compute graph.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: NO

jeffbolznv · 2026-08-06T13:28:09Z

+            enum ggml_prec       prec);
+
+    // set the smallest rank that the implementation can use to internally convert the src1 data to
+    // ranks in decreasing order:


I think it will be difficult to strictly order these and have such coarse groupings, they are probably not so interchangeable. For example, if something can be quantized to MXFP4 is it really also safe to quantize to Q4_0? It's probably model-dependent. Would the gguf be able to encode that distinction?

I'd like to eventually see FP8 (E4M3) as an option, but I imagine it would not be interchangeable with Q8_1.

I'd like to eventually see FP8 (E4M3) as an option

+1, I have some WIP for FP8 (E4M3) support lying around somewhere 😃

It's probably model-dependent. Would the gguf be able to encode that distinction?

Yes, the proposal in #24364 is to include information in the GGUF:

https://github.com/ggml-org/llama.cpp/pull/24364/changes#diff-4f653096980bd7d10518aa909cb648452cd3aa380ff93cb9fb642dca48536526R29-R32

I think the decision in the backend for which 4-bit quantization to use for the src1 should be driven by the src0 type (in addition to the provided ggml_prec for src1).

So for example, if we have a weight tensor in MXFP4/NVFP4 without any additional information about it, the backend will see GGML_PREC_DEFAULT (or GGML_PREC_UNDEFINED if we rename it) and it would likely quantize src1 to MXFP4/NVFP4 (if there is hardware support). There is no reason to quantize to Q4_0 if src0 is MXFP4/NVFP4.

If the model provider decided that FP4 x FP4 is poor quality for that tensor, then the model meta info will provide the optional policy to use GGML_PREC_Q8 for this weight. In this case the llama.cpp context will call ggml_mul_mat_set_prec_src1(x, GGML_PREC_Q8) and the backends will know to not quantize to any 4-bits, even if there is hardware support.

If the weight tensor is any other type (Q4_0, Q8_0, F16 for example), then without additional information, the backend would technically be allowed to quantize src1 to Q4_0. But from experience we know this is not beneficial so no need to do it. The user code (for example llama.cpp) can also be explicit and add a default policy for non-MXFP4/NVFP4 types to set src1 prec to GGML_PREC_Q8 - this way it's unambiguous that the backend should not quantize to Q4_0.

I think these apply to FP8, but let me know if you think of a specific case that would not be clear. I.e., unless the weight is FP8 and there is hardware support for FP8, then the backend ~~would know there is not~~ won't have a reason to prefer FP8 over Q8_1.

ORippler

What's the plan to test correct behavior of a backend? Numerical precision/accuracy + test-backend-ops?

ORippler · 2026-08-06T15:50:18Z

-        GGML_PREC_DEFAULT =  0, // stored as ggml_tensor.op_params, 0 by default
+        GGML_PREC_DEFAULT =  0, // TODO: rename to GGML_PREC_UNDEFINED?
        GGML_PREC_F32     = 10,
+        GGML_PREC_F16     = 20,


BF16 was introduced as FP16 shows numerical issues in Deep Learning applications, and is the de facto standard for LLM (pre-)training. So we may potentially look to discriminate between FP16 and BF16 here

So think about this enum such that it does not necessarily correspond to a specific data type. It would likely be better to name it like this:

enum ggml_prec { GGML_PREC_UNDEFINED = 0, GGML_PREC_32_BITS = 10, GGML_PREC_16_BITS = 20, GGML_PREC_8_BITS = 30, GGML_PREC_4_BITS = 40, };

This way, the same consideration as I explained earlier applies. If the src0 is BF16 then the backend has no reason to consider converting src1 to FP16. If the src0 is Q8_0 for example, respectively there is no reason to convert the src1 to BF16 (the scale factor of Q8_0 is FP16).

I considered the BF16 vs FP16 distinction here, but I still don't see a case where it would really be needed.

If the src0 is Q8_0 for example, respectively there is no reason to convert the src1 to BF16 (the scale factor of Q8_0 is FP16).

There is: weight-only quantization (which sparked this PR/debate). For NVFP4 W4A16, the approach would look roughly like this

// x is NVFP4 (W4) // y is supposed to be A16 (and by virtue of BF16 being training standard most likely BF16 over FP16) ggml_tensor * z = ggml_mul_mat(ctx, x, y); ggml_mul_mat_set_prec_acc (z, GGML_PREC_F32); ggml_mul_mat_set_prec_src1(z, GGML_PREC_BF16);

I considered the BF16 vs FP16 distinction here, but I still don't see a case where it would really be needed.

Numerical stability/exactness (the FP32 we materialize in ggml is expected to reside in BF16's representable value ranges for LLMs as opposed to F16 value range, as LLMs were trained in BF16). But those are most likely nits/someone would need to do a research study on this

Maybe an extra enum member GGML_PREC_WIDE_RANGE which can differentiate and help signal between BF16, FP16, MXFP8, FP8(E4M3, E5M2)

What i mean is for storing BF16 we can have "GGML_PREC_16_BITS | GGML_PREC_WIDE_RANGE" and for FP16 "GGML_PREC_16_BITS" only . We can check GGML_PREC_16_BITS for common case signals and GGML_PREC_WIDE_RANGE signal for specific case when we need wide range

@ORippler Specifically for NVFP4, if the hint for src1 is GGML_PREC_F16, then the backend can easily deduce to use BF16 over FP16 for the representation of src1. I don't think having explicit GGML_PREC_F16 + GGML_PREC_BF16 would help in this case.

@ORippler Specifically for NVFP4, if the hint for src1 is GGML_PREC_F16, then the backend can easily deduce to use BF16 over FP16 for the representation of src1. I don't think having explicit GGML_PREC_F16 + GGML_PREC_BF16 would help in this case.

How would a backend deduce this intention? 🤔 I'm fine with it being undefined/open to the backend, was just wondering if it would make sense to precise it at this granularity (we have had "silent" FP16 overflows in the past iirc)

@ORippler Hm, yes - it can't deduce (I somehow thought that FP16 cannot represent all NVFP4 values exactly).

I've added explicit GGML_PREC_BF16. Later on we can also add GGML_PREC_FP8 if see it's necessary.

ORippler · 2026-08-06T15:53:16Z

+    // for example:
+    //  - GGML_PREC_DEFAULT - the implementation is allowed to choose the most efficient accumulation type (i.e. same as GGML_PREC_F16)
+    //  - GGML_PREC_F32     - requires accumulation of the results in F32
+    //  - GGML_PREC_F16     - can accumulate the results in F16, BF16, F32


Same as above regarding FP16 & BF16. NVGPUs can only accumulate in F16/F32 though

am17an · 2026-08-07T15:22:44Z

Note that in CUDA there is currently no way to force F32 accumulation for FA

ggerganov · 2026-08-07T15:45:52Z

Note that in CUDA there is currently no way to force F32 accumulation for FA

That should be OK. The CUDA backend currently has a mechanism to increase the numerical range of the accumulators: #17746.

For example, if a FA op receives:

ggml_flash_attn_ext_set_prec_acc(z, GGML_PREC_F16);

Then it means the CUDA backend does not have to do the FATTN_KQ_MAX_OFFSET trick and can potentially optimize for that. Though it can also continue to just do what it is currently doing and that would also be fine in this case because the current implementation is aimed at ggml_flash_attn_ext_set_prec_acc(z, GGML_PREC_F32);.

ggerganov · 2026-08-07T17:39:36Z

What's the plan to test correct behavior of a backend?

Not 100% sure on this point, but my first thought is that the configured precisions would only affect the max err of the op in test-backend-ops. Still, if we want to be exhaustive, we technically need to run all ops in test-backend-ops with all possible precision configurations. This is obviously not feasible. So not really sure about this yet.

am17an · 2026-08-07T17:54:09Z

I think we can maybe just start with mul_mat and try to get backends to conform there. It is not possible to do for every op imo.

ORippler · 2026-08-11T12:30:54Z

Not 100% sure on this point, but my first thought is that the configured precisions would only affect the max err of the op in test-backend-ops

yeah I thought along those lines as well, and we are already doing this for W4A4 in NVFP4/MXFP4 tests inside test-backend-ops (here, we relax the max err bound for HW-acceleration as opposed to doing W4A8)

ynankani · 2026-08-12T08:56:09Z

Note that in CUDA there is currently no way to force F32 accumulation for FA

On similar note, I think CPU FA will abort with GGML_PREC_FP16

ynankani · 2026-08-12T10:45:49Z

+    //   - ggml_mul_mat_set_prec_src1(a, GGML_PREC_Q4):
+    //     - allows the implementation to quantize F32, F16 data down to GGML_TYPE_Q8_0 or GGML_TYPE_NVFP4
+    //
+    GGML_API void ggml_mul_mat_set_prec_src1(


This can be made generic something like
GGML_API void ggml_set_prec_src(struct ggml_tensor * a, int isrc, enum ggml_prec prec);

This can be used for FA(K,V, Q), mat_mul(activations) ,moe mat_mul_id(activations) . I mean src[0], src[1], src[2]

This can be made generic something like
GGML_API void ggml_set_prec_src(struct ggml_tensor * a, int isrc, enum ggml_prec prec);

Yes, it's possible.

On similar note, I think CPU FA will abort with GGML_PREC_FP16

Currently it will abort, but we'll have to adjust it when the spec is completed.

This commit adds conversion support for NVIDIA Nemotron 3.5 Lightning. It does not include NVFP4 models, including dflash and dspark. The motivation for not including these conversion is that those checkpoint was designed as W4A16, but the converted GGUF only says that Its weights are NVFP4. On Blackwell, llama.cpp sees NVFP4 weights and will automatically selects its native FP4 path, dynamically converting activations to FP4. That makes the executed operation W4A4 which can affect the accuracy of the draft model and therefor in turn effect the acceptance rate of the target model. The references PR below has more details on a solution to address this issue. Once it has been merged we should be able to convert and publish these models, but for now we are just commenting them out. Refs: ggml-org/llama.cpp#26675

ynankani · 2026-08-27T04:37:12Z

+                ggml_set_op_params_i32(a, 3, prec_i32);
+            }
+            break;
+        default:


Can we also handle MUL_MAT_ID here?

ynankani · 2026-08-27T05:00:31Z

+                ggml_set_op_params_i32(a, 2 + idx, prec_i32);
+            }
+            break;
+        default:


same, can we handle MUL_MAT_ID here as well

ynankani · 2026-08-27T10:46:27Z


                ggml_set_op_params_i32(a, 0, prec_i32);
            }
-            break;


looks like break got dropped mistakenly here?

Yes, my mistake - thank you.

Signed-off-by: ynankani <ynankani@nvidia.com>

ORippler · 2026-08-28T12:13:06Z

    };

    // precision
+    // this enum is used to declare the allowed floating-point types that can be used during the compute of an op


Suggested change

// this enum is used to declare the allowed floating-point types that can be used during the compute of an op

// this enum is used to declare the allowed numerical precision/data-types that can be used during the compute of an op

Q4/Q8 are not floating point, so the comment is a bit misleading. Also, ggmL-prec is now overloaded to signal either specific data-types (FP32/BF16/FP16) and precision (8-bit, 4-bit)

ORippler · 2026-08-28T12:18:16Z

+    //     - allows the implementation to quantize F32, BF16, F16 data of src[1] down to GGML_TYPE_Q8_0
+    //     - cannot quantize it down to GGML_TYPE_Q4_0 or GGML_TYPE_NVFP4
+    //   - ggml_prec_set_src(a, GGML_PREC_Q4, 1):
+    //     - allows the implementation to quantize F32, BF16, F16 data of src[1] down to GGML_TYPE_Q8_0 or GGML_TYPE_NVFP4


Suggested change

// - allows the implementation to quantize F32, BF16, F16 data of src[1] down to GGML_TYPE_Q8_0 or GGML_TYPE_NVFP4

// - allows the implementation to quantize F32, BF16, F16 data of src[1] down to 4-bit datatypes such as GGML_TYPE_Q4_K, GGML_TYPE_NVFP4 etc.

//

// Note this is currently limited to src[1] of GGML_OP_MUL_MAT and GGML_OP_MUL_MAT_ID

We currently effectively limit this to src[1] of MUL_MAT/MUL_MAT_ID. I feel it may be worth to reflect this in the descriptive comment.

ORippler · 2026-08-28T12:33:44Z

+void ggml_prec_set_acc(
+        struct ggml_tensor * a,
+        enum ggml_prec       prec) {
+    switch (a->op) {


In the docs we say only

// - GGML_PREC_F32 - requires accumulation of the results in F32 // - GGML_PREC_BF16 - can accumulate the results in BF16, F32 // - GGML_PREC_F16 - can accumulate the results in F16, F32

are supported. If the other values are truly not allowed, we should check prec here and error if the other values are supplied

Yes, I was think that we should actually make this function return a bool and return false on failure. Also return false when applied to unsupported op, etc.

ORippler · 2026-08-28T12:37:40Z

+            {
+                const int32_t prec_i32 = (int32_t) prec;
+
+                ggml_set_op_params_i32(a, 3, prec_i32);


Suggested change

ggml_set_op_params_i32(a, 3, prec_i32);

ggml_set_op_params_i32(a, 3, prec_i32); // scale is on first pos, max_bias on second

Added a search tag [TAG_GGML_PREC] that should make looking up this information more simple.

[no ci]

github-actions Bot added the ggml changes relating to the ggml tensor library for machine learning label Aug 6, 2026

ggerganov mentioned this pull request Aug 6, 2026

Force NVFP4 W4A8 path for NVFP4_W4A16 layers on Blackwell, where NVFP4 normally uses the native W4A4 path. #24364

Draft

jeffbolznv reviewed Aug 6, 2026

View reviewed changes

ORippler reviewed Aug 6, 2026

View reviewed changes

ynankani reviewed Aug 12, 2026

View reviewed changes

danbev mentioned this pull request Aug 14, 2026

nemotron-3.5-lightning : add model conversion support ggml-org/convert#16

Merged

ggerganov force-pushed the gg/ggml-prec-update branch from 89983d5 to 36b6666 Compare August 26, 2026 11:58

github-actions Bot added model Model specific testing Everything test related mtmd Related to multimodal functionality (video/image/audio) labels Aug 26, 2026

ggerganov marked this pull request as ready for review August 26, 2026 12:01

ggerganov requested review from a team and CISC as code owners August 26, 2026 12:01

ynankani reviewed Aug 27, 2026

View reviewed changes

ggerganov mentioned this pull request Aug 27, 2026

metal: fix NaN in mul_mm_id when activations exceed f16 range #26223

Open

ynankani reviewed Aug 27, 2026

View reviewed changes

ynankani added a commit to ynankani/llama.cpp that referenced this pull request Aug 27, 2026

Rebase and update based on ggml-org#26675

d47e561

Signed-off-by: ynankani <ynankani@nvidia.com>

ORippler reviewed Aug 28, 2026

View reviewed changes

ggerganov added 3 commits August 28, 2026 15:45

ggml : update ggml_prec specification

83d5ed8

[no ci]

cont : add GGML_PREC_BF16

9b970f6

cont : rework API

7f05174

ggerganov added 5 commits August 28, 2026 15:45

cont : use new API

76c0f1e

cont : swap arg order

bddf805

cont : support for MUL_MAT_ID

9372029

cont : fix accidental remove of "break;"

7886f52

cont : return bools, add doc TAG_GGML_PREC, clean-up

085ae2d

ggerganov force-pushed the gg/ggml-prec-update branch from 8f1ee04 to 085ae2d Compare August 28, 2026 12:52

ggerganov added 2 commits August 28, 2026 15:53

cont : add search tag

81cce78

cont : ws

5851394

	// this enum is used to declare the allowed floating-point types that can be used during the compute of an op
	// this enum is used to declare the allowed numerical precision/data-types that can be used during the compute of an op

-    //     - allows the implementation to quantize F32, BF16, F16 data of src[1] down to GGML_TYPE_Q8_0 or GGML_TYPE_NVFP4
+    //     - allows the implementation to quantize F32, BF16, F16 data of src[1] down to 4-bit datatypes such as GGML_TYPE_Q4_K, GGML_TYPE_NVFP4 etc.
+    //
+    // Note this is currently limited to src[1] of GGML_OP_MUL_MAT and GGML_OP_MUL_MAT_ID

	ggml_set_op_params_i32(a, 3, prec_i32);
	ggml_set_op_params_i32(a, 3, prec_i32); // scale is on first pos, max_bias on second

Uh oh!

Conversation

ggerganov commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggerganov Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ORippler left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggerganov Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ORippler Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

am17an commented Aug 7, 2026

Uh oh!

ggerganov commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggerganov commented Aug 7, 2026

Uh oh!

am17an commented Aug 7, 2026

Uh oh!

ORippler commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ynankani commented Aug 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ynankani Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ggerganov commented Aug 6, 2026 •

edited

Loading

ggerganov Aug 6, 2026 •

edited

Loading

ggerganov Aug 6, 2026 •

edited

Loading

ORippler Aug 11, 2026 •

edited

Loading

ggerganov commented Aug 7, 2026 •

edited

Loading

ORippler commented Aug 11, 2026 •

edited

Loading

ynankani Aug 27, 2026 •

edited

Loading