Skip to content

CUDA: permit quantizing into NVFP4 (Blackwell PP +40%) #23572

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sanmai wants to merge 6 commits into from
Closed

CUDA: permit quantizing into NVFP4 (Blackwell PP +40%) #23572

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7767cf2
CUDA: native 4-bit float quant
sanmai May 23, 2026
5337f3a
Apply suggestion from @sanmai
sanmai May 24, 2026
af17965
Merge branch 'master' into feat/N4_0-rebase
sanmai Aug 18, 2026
920ab3a
Merge branch 'feat/N4_0-rebase' of github.com:sanmai/llama.cpp into f…
sanmai Aug 18, 2026
ccd23d7
Merge branch 'master' into feat/N4_0-rebase
sanmai Sep 8, 2026
0c2ae64
rename
sanmai Sep 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions src/llama-quant.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -387,6 +387,7 @@ static ggml_type tensor_type_fallback(quantize_state_impl & qs, const ggml_tenso
case GGML_TYPE_Q4_K: return_type = GGML_TYPE_Q5_0; break;
case GGML_TYPE_Q5_K: return_type = GGML_TYPE_Q5_1; break;
case GGML_TYPE_Q6_K: return_type = GGML_TYPE_Q8_0; break;
case GGML_TYPE_NVFP4: return_type = GGML_TYPE_Q4_0; break;
default:
throw std::runtime_error(format("no tensor type fallback is defined for type %s",
ggml_type_name(target_type)));
Expand Down Expand Up @@ -483,6 +484,9 @@ static ggml_type llama_tensor_get_type_impl(quantize_state_impl & qs, ggml_type
else if (ftype == LLAMA_FTYPE_MOSTLY_TQ1_0 || ftype == LLAMA_FTYPE_MOSTLY_TQ2_0) {
new_type = GGML_TYPE_Q4_K;
}
else if (ftype == LLAMA_FTYPE_MOSTLY_NVFP4) {
new_type = GGML_TYPE_Q4_0;
}
}
} else if (ftype == LLAMA_FTYPE_MOSTLY_IQ2_XXS || ftype == LLAMA_FTYPE_MOSTLY_IQ2_XS || ftype == LLAMA_FTYPE_MOSTLY_IQ1_S ||
ftype == LLAMA_FTYPE_MOSTLY_IQ2_S || ftype == LLAMA_FTYPE_MOSTLY_IQ2_M || ftype == LLAMA_FTYPE_MOSTLY_IQ1_M) {
Expand Down Expand Up @@ -802,6 +806,7 @@ ggml_type llama_ftype_get_default_type(llama_ftype ftype) {
case LLAMA_FTYPE_MOSTLY_Q1_0: return GGML_TYPE_Q1_0;

case LLAMA_FTYPE_MOSTLY_MXFP4_MOE: return GGML_TYPE_MXFP4;
case LLAMA_FTYPE_MOSTLY_NVFP4: return GGML_TYPE_NVFP4;

// K-quants
case LLAMA_FTYPE_MOSTLY_Q2_K_S:
Expand Down
1 change: 1 addition & 0 deletions tools/quantize/quantize.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ static const std::vector<quant_option> QUANT_OPTIONS = {
{ "Q4_0", LLAMA_FTYPE_MOSTLY_Q4_0, " 4.34G, +0.4685 ppl @ Llama-3-8B", },
{ "Q4_1", LLAMA_FTYPE_MOSTLY_Q4_1, " 4.78G, +0.4511 ppl @ Llama-3-8B", },
{ "MXFP4_MOE",LLAMA_FTYPE_MOSTLY_MXFP4_MOE," MXFP4 MoE", },
{ "N4_0", LLAMA_FTYPE_MOSTLY_NVFP4, " 4.6 bpw native 4-bit float", },
Comment thread
sanmai marked this conversation as resolved.
Resolved
Comment thread
sanmai marked this conversation as resolved.
Resolved
{ "Q5_0", LLAMA_FTYPE_MOSTLY_Q5_0, " 5.21G, +0.1316 ppl @ Llama-3-8B", },
{ "Q5_1", LLAMA_FTYPE_MOSTLY_Q5_1, " 5.65G, +0.1062 ppl @ Llama-3-8B", },
{ "IQ2_XXS", LLAMA_FTYPE_MOSTLY_IQ2_XXS, " 2.06 bpw quantization", },
Expand Down