Skip to content

CUDA: native 4-bit float quant (Blackwell PP +40%) #23572

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
sanmai wants to merge 4 commits into
base: master
Choose a base branch
from
+6 −0
Draft

CUDA: native 4-bit float quant (Blackwell PP +40%) #23572

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7767cf2
CUDA: native 4-bit float quant
sanmai May 23, 2026
5337f3a
Apply suggestion from @sanmai
sanmai May 24, 2026
af17965
Merge branch 'master' into feat/N4_0-rebase
sanmai Aug 18, 2026
920ab3a
Merge branch 'feat/N4_0-rebase' of github.com:sanmai/llama.cpp into f…
sanmai Aug 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions src/llama-quant.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -400,6 +400,7 @@ static ggml_type tensor_type_fallback(quantize_state_impl & qs, const ggml_tenso
case GGML_TYPE_Q4_K: return_type = GGML_TYPE_Q5_0; break;
case GGML_TYPE_Q5_K: return_type = GGML_TYPE_Q5_1; break;
case GGML_TYPE_Q6_K: return_type = GGML_TYPE_Q8_0; break;
case GGML_TYPE_NVFP4: return_type = GGML_TYPE_Q4_0; break;
default:
throw std::runtime_error(format("no tensor type fallback is defined for type %s",
ggml_type_name(target_type)));
Expand Down Expand Up @@ -496,6 +497,9 @@ static ggml_type llama_tensor_get_type_impl(quantize_state_impl & qs, ggml_type
else if (ftype == LLAMA_FTYPE_MOSTLY_TQ1_0 || ftype == LLAMA_FTYPE_MOSTLY_TQ2_0 || ftype == LLAMA_FTYPE_MOSTLY_Q2_0) {
new_type = GGML_TYPE_Q4_K;
}
else if (ftype == LLAMA_FTYPE_MOSTLY_NVFP4) {
new_type = GGML_TYPE_Q4_0;
}
}
} else if (ftype == LLAMA_FTYPE_MOSTLY_IQ2_XXS || ftype == LLAMA_FTYPE_MOSTLY_IQ2_XS || ftype == LLAMA_FTYPE_MOSTLY_IQ1_S ||
ftype == LLAMA_FTYPE_MOSTLY_IQ2_S || ftype == LLAMA_FTYPE_MOSTLY_IQ2_M || ftype == LLAMA_FTYPE_MOSTLY_IQ1_M) {
Expand Down Expand Up @@ -816,6 +820,7 @@ ggml_type llama_ftype_get_default_type(llama_ftype ftype) {
case LLAMA_FTYPE_MOSTLY_Q2_0: return GGML_TYPE_Q2_0;

case LLAMA_FTYPE_MOSTLY_MXFP4_MOE: return GGML_TYPE_MXFP4;
case LLAMA_FTYPE_MOSTLY_NVFP4: return GGML_TYPE_NVFP4;

// K-quants
case LLAMA_FTYPE_MOSTLY_Q2_K_S:
Expand Down
1 change: 1 addition & 0 deletions tools/quantize/quantize.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ static const std::vector<quant_option> QUANT_OPTIONS = {
{ "Q4_0", LLAMA_FTYPE_MOSTLY_Q4_0, " 4.34G, +0.4685 ppl @ Llama-3-8B", },
{ "Q4_1", LLAMA_FTYPE_MOSTLY_Q4_1, " 4.78G, +0.4511 ppl @ Llama-3-8B", },
{ "MXFP4_MOE",LLAMA_FTYPE_MOSTLY_MXFP4_MOE," MXFP4 MoE", },
{ "N4_0", LLAMA_FTYPE_MOSTLY_NVFP4, " 4.5 bpw native 4-bit float", },

@sanmai sanmai May 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My original intent was to prevent:

llama-quantize foo-bf16.gguf foo-N4_0.gguf MXFP4

That will create something that I assumed will be substandard, but as we test pre-quantized models, it turns out they are better, but not that better.

So I guess we could do this, and eliminate a future BC break.

Suggested change
{ "N4_0", LLAMA_FTYPE_MOSTLY_NVFP4, " 4.5 bpw native 4-bit float", },
{ "MXFP4", LLAMA_FTYPE_MOSTLY_NVFP4, " 4.5 bpw native 4-bit float", },

{ "Q5_0", LLAMA_FTYPE_MOSTLY_Q5_0, " 5.21G, +0.1316 ppl @ Llama-3-8B", },
{ "Q5_1", LLAMA_FTYPE_MOSTLY_Q5_1, " 5.65G, +0.1062 ppl @ Llama-3-8B", },
{ "IQ2_XXS", LLAMA_FTYPE_MOSTLY_IQ2_XXS, " 2.06 bpw quantization", },
Expand Down