fix(gguf): GGUF model support fixes for Blackwell GPUs#30497

Closed

kitaekatt wants to merge 17 commits intovllm-project:mainfrom

kitaekatt:cherry-picked

Commits on Dec 10, 2025

fix: Address review feedback - platform-agnostic docs and context manager
kitaekatt
committed
fix: Lazy tokenizer init in StructuredOutputManager to prevent semaphore leak
kitaekatt
committed
fix(gguf): Ensure Gemma2 configs have hidden_act for backward compatibility
kitaekatt
committed
fix(gguf): Skip lm_head mapping for models with tied word embeddings
kitaekatt
committed
fix(gemma2): Skip missing parameters during GGUF weight loading
kitaekatt
committed
fix(gemma2): Add quant_config to embedding layer for GGUF support
kitaekatt
committed
fix(gguf): Extract attn_logit_softcapping from GGUF metadata
kitaekatt
committed
fix(nemotron_h): Add rotary positional embeddings to attention layers
kitaekatt
committed
fix: Default GGUF to float16 while preserving bfloat16 option
kitaekatt
committed
fix(gguf): Disable bfloat16 on Blackwell (SM 120+) via device capability check

kitaekatt
and
Isotr0py
committed
fix(gguf): Remove dtype auto-override, rely on quantization layer
kitaekatt
committed
fix(gguf): Auto-select compatible dtype for GGUF models on Blackwell

kitaekatt
and
claude
committed
fix: Address review feedback - correct Blackwell compute capability
kitaekatt
committed
fix(gguf): Use EOS token ID from GGUF metadata instead of HF tokenizer
kitaekatt
committed

Commits on Dec 11, 2025