fix(gguf): Disable bfloat16 for GGUF on blackwell device

[kitaekatt]

Summary

Fixes incorrect output from GGUF models on Blackwell (SM 120+) GPUs by defaulting to float16 dtype.

Changes

• Default GGUF quantization to float16 (was auto-selecting bfloat16)
• Add warning when bfloat16 is explicitly requested on Blackwell

Root Cause

bfloat16 causes precision issues with GGUF quantized weights on Blackwell architecture.

Testing

Tested with multiple GGUF models on RTX 5090.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.

[gemini-code-assist]

Code Review

This pull request aims to fix precision issues with GGUF models on Blackwell GPUs by defaulting to float16. The change correctly disables bfloat16 for GGUF on Blackwell devices. However, I've found a critical issue where the device capability check uses 120 instead of 100 for Blackwell, which would prevent the fix from being applied. I've provided a suggestion to correct this.

[kitaekatt]

This PR is a re-opening of #30090. The original branch was accidentally deleted, preventing that PR from being reopened.

@Isotr0py You had approved the previous PR - would appreciate your review on this one as well.

[yewentao256]

Thanks for the work! One minor update before landing

[mergify]

Hi @kitaekatt, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[yewentao256]

It would be a little bit unclear for SM (10.0+), let's just remove them and we all know blackwell.

[mergify]

Hi @kitaekatt, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[kitaekatt]

I know this PR is already approved, but I just did some isolated testing of this PR so sharing results.

Tested on Blackwell Hardware

GPU: RTX 5090 (SM 12.0, compute capability 120)

Test: Direct validation of GGUFConfig().get_supported_act_dtypes()

Before (upstream/main):

>>> GGUFConfig().get_supported_act_dtypes()
[torch.float16, torch.bfloat16, torch.float32]  # bfloat16 included ❌

After (this PR):

>>> GGUFConfig().get_supported_act_dtypes()
WARNING: GGUF has precision issues with bfloat16 on Blackwell.
[torch.float16, torch.float32]  # bfloat16 excluded ✓

Validated that bfloat16 is correctly excluded on Blackwell devices.

[kitaekatt]

Fixed DCO (added sign-off) and pre-commit (single-line format).