[Qwen 3.5][gemma4] Qwen35 and Gemma 4 fast inference (clean-base mirror of #588)

[danielhanchen]

Staging mirror of unslothai#603

Original PR: unslothai#603
Author: danielhanchen

This is a staging copy for review and editing. Once finalized, changes will be pushed back to the original PR.

---

Original description

Summary

Clean-base mirror of unslothai#588. Contains only the original author's 9 commits from @Datta0 (7fa143f through b7052b5, 2026-03-30 to 2026-04-17), with none of the later auto-review fix commits that accumulated on that PR's branch.

Why

Provides a fresh starting point to re-run the PR-review pipeline against the author's original diff without interference from prior review-loop commits layered on top of unslothai#588. Useful for validating review-pipeline changes end-to-end.

Branch contents

Nine commits from @Datta0:

• 7fa143f [WIP] initial fast_inference support for qwen3.5
• 31c1bc3 [WIP] fixes for rope deltas
• 112f1b5 cleanup
• a31dee8 fix lm_head detection and remove moe
• 5d07504 [WIP] gemma 4 dense fast inference
• a85a4f4 Fix dtype setting
• d2f7983 Merge remote-tracking branch 'origin/main' into qwen35-shared-final
• 43ed24f Fix gemma4 load on vllm 0.19.0
• b7052b5 fix bnb loader for gemam4

Note

Not a replacement for unslothai#588. See unslothai#588 for the actively-maintained PR and associated review history.

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for Gemma 4 and Qwen 3.5 (GDN) architectures, adding specialized vLLM patches for LoRA and k_eq_v weight handling and a model finalization utility. It also expands layer configuration templates and improves dtype management. Reviewer feedback recommends simplifying helper functions like _get_model_device and _normalize_state_dict_tensor, and streamlining attribute-setting logic to improve code readability.

[gemini-code-assist]

The _get_model_device function can be simplified using next() with a generator expression to avoid explicit loops and improve readability.

def _get_model_device(model):
    return next((t.device for t in model.parameters()), next((t.device for t in model.buffers()), torch.device("cpu")))
pass

[gemini-code-assist]

The nested try-except blocks for setting attributes are redundant. You can simplify this by attempting the standard setattr first and falling back to __dict__ only if it fails, or simply using setattr if the object is a standard Python object.

Suggested change

	try:
	setattr(config, field, runtime_dtype)
	success = True
	continue
	except Exception:
	pass
	try:
	config.__dict__[field] = runtime_dtype
	success = True
	except Exception:
	pass
	for field in target_fields:
	try:
	setattr(config, field, runtime_dtype)
	success = True
	break
	except Exception:
	try:
	config.__dict__[field] = runtime_dtype
	success = True
	break
	except Exception:
	pass

[gemini-code-assist]

The _normalize_state_dict_tensor function can be simplified by chaining the operations directly.

    def _normalize_state_dict_tensor(value):
        if isinstance(value, torch.nn.Parameter):
            value = value.detach()
        return value.to_dense().contiguous() if value.is_sparse else value.contiguous()

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for Gemma 4 and Qwen 3.5 (GDN) architectures, featuring vLLM patches for LoRA and k_eq_v support, a new finalize_huggingface_model utility, and extract_gdn_layers for Gated Delta Net. It also improves dtype configuration and tensor unwrapping across the codebase. Feedback highlights opportunities to improve the robustness of language model detection, ensure safer attribute checks for rotary embeddings, and align with modern PyTorch practices by using .detach() instead of .data.

[gemini-code-assist]

hasattr(module, "rotary_pos_emb") will return True even if the attribute is set to None. If it is None, the subsequent call to module.rotary_pos_emb.__class__(...) on line 713 will raise a TypeError. It is safer to check if the attribute is not None.

Suggested change

	if hasattr(module, "rotary_pos_emb"):
	if getattr(module, "rotary_pos_emb", None) is not None:

[gemini-code-assist]

The detection of language_model here is less robust than the logic used in set_additional_modules (lines 541-549). It may fail to identify the language model component for standard causal LM models or models where language_model is a direct attribute of new_model. Using a more robust detection logic ensures that layer_idx is correctly patched across different model architectures.

Suggested change

	language_model = getattr(getattr(new_model, "model", None), "language_model", None)
	language_model = getattr(new_model, "language_model", None)
	if language_model is None:
	language_model = getattr(new_model, "model", None)
	if language_model is not None:
	language_model = getattr(language_model, "language_model", language_model)

[gemini-code-assist]

Using .data is discouraged in modern PyTorch as it can lead to silent bugs by bypassing autograd tracking and safety checks. It is recommended to use .detach() instead when a view of the tensor is needed without gradient tracking.

Suggested change

	store(f"{prefix}.conv1d.weight", gdn.conv1d.weight.data)
	store(f"{prefix}.dt_bias", gdn.dt_bias.data)
	store(f"{prefix}.A_log", gdn.A_log.data)
	store(f"{prefix}.conv1d.weight", gdn.conv1d.weight.detach())
	store(f"{prefix}.dt_bias", gdn.dt_bias.detach())
	store(f"{prefix}.A_log", gdn.A_log.detach())

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for Gemma 4 and Qwen 3.5 (Gated Delta Net) architectures, featuring vLLM to Hugging Face conversion utilities, LoRA patches, and quantization support. Key additions include the finalize_huggingface_model function for post-conversion cleanup, robust dtype configuration handling, and expanded layer mapping. Feedback identifies a critical issue where skipping v_proj extraction for Gemma 4 k_eq_v layers causes missing keys in the Hugging Face state dict; a suggestion is provided to duplicate the k_proj shard instead. Additionally, the slicing logic in extract_gdn_layers should be validated to prevent potential IndexError when handling different fused projection counts.

[gemini-code-assist]

For Gemma 4 layers where attention_k_eq_v is enabled, vLLM's qkv_proj typically contains only Q and K shards. However, the Hugging Face model architecture still expects a v_proj weight. Skipping the extraction of v_proj here will result in a missing key in the converted state dictionary, which can cause errors when loading the model in Transformers (e.g., missing keys or uninitialized weights).

Instead of skipping, you should duplicate the k_proj shard (index 1) into v_proj to ensure the Hugging Face model is fully populated and functional.

Suggested change

	if kk not in gemma4_k_eq_v_layers:
	get_state_dict(f"{prefix}.v_proj", 2, state_dict, qkv_proj)
	if kk not in gemma4_k_eq_v_layers:
	get_state_dict(f"{prefix}.v_proj", 2, state_dict, qkv_proj)
	else:
	# For k_eq_v layers, v_proj is a copy of k_proj
	get_state_dict(f"{prefix}.v_proj", 1, state_dict, qkv_proj)

[gemini-code-assist]

The slicing logic for qkv_weight and z_weight assumes that offsets has at least 5 elements (derived from 4 output sizes: Q, K, V, and Z). If a model variant has a different number of fused projections, this will raise an IndexError. It would be safer to validate the length of output_sizes or offsets before performing these slices.

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for Gemma4 and Qwen 3.5 (Gated Delta Net) architectures, implementing specialized vLLM patches for LoRA and quantization state management. Key additions include the finalize_huggingface_model function for centralized model post-processing, expanded layer configuration templates for new architectures, and enhanced robustness in dtype configuration and state dict conversion. Review feedback identifies potential TypeError vulnerabilities when re-initializing rotary embeddings if attributes are None, and suggests more robust string parsing for layer names to prevent potential ValueError during model conversion.

[gemini-code-assist]

The hasattr check will return True even if module.rotary_emb is None. Accessing module.rotary_emb.__class__ on line 699 when it is None will result in a TypeError because NoneType is not callable. It is safer to check if the attribute is not None.

Suggested change

	if hasattr(module, "rotary_emb"):
	if getattr(module, "rotary_emb", None) is not None:

[gemini-code-assist]

Similar to rotary_emb, module.rotary_pos_emb should be checked for None before accessing its __class__ on line 713 to avoid a TypeError.

Suggested change

	if hasattr(module, "rotary_pos_emb"):
	if getattr(module, "rotary_pos_emb", None) is not None:

[gemini-code-assist]

Similar to rotary_emb, module.rotary_emb_local should be checked for None before accessing its __class__ on line 718 to avoid a TypeError.

Suggested change

	if hasattr(module, "rotary_emb_local"):
	if getattr(module, "rotary_emb_local", None) is not None:

[gemini-code-assist]

If layer_name_br does not contain a dot (e.g., if a top-level attribute is added to layer_names), rsplit(".", 1) will raise a ValueError. While current templates include dots, adding a guard or using a safer way to split would be more robust for future changes.

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request implements support for Gemma 4 and Qwen 3.5 (Gated Delta Net) architectures, including vLLM integration, LoRA patching, and conversion utilities. Key changes include the addition of finalize_huggingface_model for model post-processing, extract_gdn_layers for linear attention support, and more robust dtype configuration handling. Review feedback highlights opportunities to improve code quality by using isinstance for type checking, adopting more idiomatic Python patterns like next() for device detection, and simplifying conditional checks. Additionally, it is recommended to remove redundant pass statements and avoid the use of eval() for security and maintainability.

[gemini-code-assist]

Using isinstance is more robust than comparing class names as strings. It correctly handles subclassing and is generally considered better practice.

Suggested change

	def patched_supports_lora(model):
	if model.__class__.__name__ == "Gemma4ForConditionalGeneration":
	return True
	return original_supports_lora(model)
	def patched_supports_lora(model):
	if isinstance(model, Gemma4ForConditionalGeneration):
	return True
	return original_supports_lora(model)

[gemini-code-assist]

Similar to the patched_supports_lora function, using isinstance here is more robust than comparing class names as strings.

Suggested change

	def patched_create_lora_manager(model, *args, **kwargs):
	if model.__class__.__name__ == "Gemma4ForConditionalGeneration":
	lora_manager_cls = kwargs.pop("lora_manager_cls", vllm_lora_model_manager.LoRAModelManager)
	return lora_manager_cls(model, *args, **kwargs)
	return original_create_lora_manager(model, *args, **kwargs)
	def patched_create_lora_manager(model, *args, **kwargs):
	if isinstance(model, Gemma4ForConditionalGeneration):
	lora_manager_cls = kwargs.pop("lora_manager_cls", vllm_lora_model_manager.LoRAModelManager)
	return lora_manager_cls(model, *args, **kwargs)
	return original_create_lora_manager(model, *args, **kwargs)

[gemini-code-assist]

The use of eval() can be a security risk if the string being evaluated can be influenced by external input. While layer_name_br is constructed from internal data here, it's a good practice to avoid eval() for maintainability and security. Consider refactoring to use a safer method for accessing nested attributes, even though it might be more complex with attribute names containing [].

[gemini-code-assist]

For conciseness, you can use next with a generator expression to find the model's device. This avoids explicit loops and is a more idiomatic way to achieve this in Python.

Suggested change

	def _get_model_device(model):
	for tensor in model.parameters():
	return tensor.device
	for tensor in model.buffers():
	return tensor.device
	return torch.device("cpu")
	pass
	def _get_model_device(model):
	try:
	return next(model.parameters()).device
	except StopIteration:
	try:
	return next(model.buffers()).device
	except StopIteration:
	return torch.device("cpu")

[gemini-code-assist]

The pass statement here is redundant and can be removed for cleaner code.

[gemini-code-assist]

This condition can be simplified. Checking for not quantization_config is more Pythonic for checking if a dictionary is empty or None.

Suggested change

	if (quantization_config or {}) == {} and bnb_config is None:
	if not quantization_config and bnb_config is None:

[gemini-code-assist]

The pass statement here is redundant and can be removed for cleaner code.