fix(gguf): Auto-select compatible dtype for GGUF models on Blackwell

[kitaekatt]

Summary

Fixes dtype conflict for Gemma3 GGUF models on Blackwell GPUs (SM 120+) where --dtype auto fails because:

1. Gemma3 blocks float16 (numerical instability) - only allows [bfloat16, float32]
2. GGUF on Blackwell blocks bfloat16 (dequant kernels use fp16) - only allows [float16, float32]
3. Intersection: Only float32 works

Error before fix:

torch.bfloat16 is not supported for quantization method gguf.
Supported dtypes: [torch.float16, torch.float32]

Changes

1. gguf.py: Block bfloat16 on Blackwell (SM 120+) via current_platform.has_device_capability(120)

2. vllm.py: Add _resolve_dtype_conflict() to find compatible dtype when model restrictions and quantization restrictions conflict. Falls back to float32 when no other option exists.

Test Plan

• Tested with google/gemma-3-1b-it GGUF on RTX 5090 (Blackwell)
• Server starts successfully with --dtype auto
• Inference produces correct output (211 tok/s)
• Non-Gemma3 GGUF models still work (Gemma2, Qwen, etc.)

Related

• Builds on fix: Force float16 dtype for GGUF models to fix incorrect output #30090 (general GGUF bfloat16 fix)
• Specific to models that block BOTH float16 AND bfloat16 (Gemma3 + GGUF + Blackwell)

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.

[gemini-code-assist]

Code Review

This pull request effectively resolves a dtype conflict for GGUF models on Blackwell GPUs, particularly for models like Gemma3 with specific dtype restrictions. The changes are well-implemented. In vllm/model_executor/layers/quantization/gguf.py, bfloat16 is correctly excluded on SM 120+ devices. The new logic in vllm/config/vllm.py for automatic dtype conflict resolution is robust; it finds a compatible dtype by intersecting model and quantization-supported types, selects the most performant option, and warns the user. This is a solid fix that also handles future similar conflicts. I have not identified any high or critical severity issues.

[mergify]

⚠️ The sha of the head commit of this PR conflicts with #30410. Mergify cannot evaluate rules on this PR. ⚠️