[Qwen 3.5][gemma4] Qwen35 and Gemma 4 fast inference (clean-base mirror of #588)

[danielhanchen]

Staging mirror of unslothai#603

Original PR: unslothai#603
Author: danielhanchen

This is a staging copy for review and editing. Once finalized, changes will be pushed back to the original PR.

---

Original description

Summary

Clean-base mirror of unslothai#588. Contains only the original author's 9 commits from @Datta0 (7fa143f through b7052b5, 2026-03-30 to 2026-04-17), with none of the later auto-review fix commits that accumulated on that PR's branch.

Why

Provides a fresh starting point to re-run the PR-review pipeline against the author's original diff without interference from prior review-loop commits layered on top of unslothai#588. Useful for validating review-pipeline changes end-to-end.

Branch contents

Nine commits from @Datta0:

• 7fa143f [WIP] initial fast_inference support for qwen3.5
• 31c1bc3 [WIP] fixes for rope deltas
• 112f1b5 cleanup
• a31dee8 fix lm_head detection and remove moe
• 5d07504 [WIP] gemma 4 dense fast inference
• a85a4f4 Fix dtype setting
• d2f7983 Merge remote-tracking branch 'origin/main' into qwen35-shared-final
• 43ed24f Fix gemma4 load on vllm 0.19.0
• b7052b5 fix bnb loader for gemam4

Note

Not a replacement for unslothai#588. See unslothai#588 for the actively-maintained PR and associated review history.

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for Gemma 4 and Qwen 3.5 (GDN) models, including vLLM integration patches for LoRA and quantization, expanded layer configuration patterns, and a new model finalization utility. Feedback suggests improving maintainability by using constants for configuration attributes, optimizing device detection logic with iterators, and refactoring nested exception handling in utility functions.

[gemini-code-assist]

The dictionary passed to _set_config_attrs contains magic numbers that are repeated in multiple places. Consider defining these as constants or a configuration object to improve maintainability.

[gemini-code-assist]

The _get_model_device function iterates over parameters and buffers to find a device. This can be simplified by checking next(model.parameters(), None) or next(model.buffers(), None) to avoid unnecessary iteration.

Suggested change

	def _get_model_device(model):
	for tensor in model.parameters():
	return tensor.device
	for tensor in model.buffers():
	return tensor.device
	return torch.device("cpu")
	pass
	def _get_model_device(model):
	param = next(model.parameters(), None)
	if param is not None:
	return param.device
	buffer = next(model.buffers(), None)
	if buffer is not None:
	return buffer.device
	return torch.device("cpu")

[gemini-code-assist]

The nested try-except blocks for setting attributes are redundant. Using getattr and setattr directly or a single try-except block would be cleaner.

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for Gemma 4 and Qwen 3.5 (GDN) models, featuring vLLM patches for LoRA and BitsAndBytes, improved dtype configuration, and enhanced state dictionary conversion. Key updates include the finalize_huggingface_model utility and GDN layer extraction logic. Feedback highlights the need to correctly initialize v_proj for Gemma 4 shared K/V layers, improve dtype string parsing, and replace .data with .detach() for safer tensor access. It is also suggested to disable stride checks in state dictionary comparisons to avoid false positives.

[gemini-code-assist]

For Gemma 4 models with k_eq_v layers, the v_proj shard is missing from the vLLM qkv_proj because K and V are shared. However, the Hugging Face model expects a v_proj module. Skipping the extraction here will leave the HF model's v_proj with uninitialized weights. Instead, shard 1 (K) should be duplicated into v_proj for these specific layers to ensure compatibility.

Suggested change

	if kk not in gemma4_k_eq_v_layers:
	get_state_dict(f"{prefix}.v_proj", 2, state_dict, qkv_proj)
	# For Gemma4 k_eq_v layers, shard 1 (K) is reused as V.
	v_shard = 1 if kk in gemma4_k_eq_v_layers else 2
	get_state_dict(f"{prefix}.v_proj", v_shard, state_dict, qkv_proj)

[gemini-code-assist]

The getattr(torch, dtype, dtype) call will fail to resolve string dtypes that include the torch. prefix (e.g., "torch.float16"), resulting in the string itself being used instead of the torch.dtype object. This can cause issues with configuration serialization or downstream processing that expects a proper dtype object or a short string identifier.

Suggested change

	runtime_dtype = getattr(torch, dtype, dtype) if isinstance(dtype, str) else dtype
	runtime_dtype = getattr(torch, dtype.split(".")[-1], dtype) if isinstance(dtype, str) else dtype

[gemini-code-assist]

Using .data is discouraged in modern PyTorch as it can bypass autograd checks and lead to silent errors. It is safer to use .detach() or simply access the tensor directly if no gradient tracking is required in this context.

Suggested change

	store(f"{prefix}.conv1d.weight", gdn.conv1d.weight.data)
	store(f"{prefix}.dt_bias", gdn.dt_bias.data)
	store(f"{prefix}.A_log", gdn.A_log.data)
	store(f"{prefix}.conv1d.weight", gdn.conv1d.weight.detach())
	store(f"{prefix}.dt_bias", gdn.dt_bias.detach())
	store(f"{prefix}.A_log", gdn.A_log.detach())

[gemini-code-assist]

Using .data is discouraged. Use .detach() instead to safely access the underlying tensor without gradient tracking.

Suggested change

	state_dict[f"{vllm_text_model_prefix}.layers.{kk}.layer_scalar"] = layer.layer_scalar.data
	quant_state_dict[f"{vllm_text_model_prefix}.layers.{kk}.layer_scalar"] = layer.layer_scalar.data
	state_dict[f"{vllm_text_model_prefix}.layers.{kk}.layer_scalar"] = layer.layer_scalar.detach()
	quant_state_dict[f"{vllm_text_model_prefix}.layers.{kk}.layer_scalar"] = layer.layer_scalar.detach()

[gemini-code-assist]

In assert_same_state_dict, check_stride is set to True in the fallback path but False in the main comparison loop (line 1242). Strides often differ between model loading implementations without affecting correctness. It is better to consistently set check_stride=False to avoid false positives in state dict comparisons.

Suggested change

	check_stride = True,
	check_stride = False,

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for Gemma 4 and Qwen 3.5 (GDN) architectures, including vLLM integration patches for LoRA and quantization state handling. Key changes involve refactoring model finalization logic, expanding layer configuration patterns for vision and linear attention modules, and improving dtype management in configurations. Feedback was provided to enhance the robustness of dtype resolution in set_dtype_in_config by handling string inputs that include a 'torch.' prefix.

[gemini-code-assist]

The runtime_dtype resolution logic does not account for the "torch." prefix in string-based dtypes (e.g., "torch.float16"). If such a string is passed, getattr(torch, dtype, dtype) will return the original string instead of the torch.dtype object. This can lead to the config being populated with strings like "torch.float16", which may cause issues in downstream Transformers functions that expect either a torch.dtype object or a standard short string like "float16". Stripping the prefix before calling getattr would make this more robust.

Suggested change

	runtime_dtype = getattr(torch, dtype, dtype) if isinstance(dtype, str) else dtype
	if isinstance(dtype, str): dtype = dtype.replace("torch.", "")
	runtime_dtype = getattr(torch, dtype, dtype) if isinstance(dtype, str) else dtype

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for Gemma 4 and Qwen 3.5 (GDN) architectures, featuring vLLM integration patches for LoRA and quantization state handling. It introduces finalize_huggingface_model for consistent model initialization, expands layer configuration patterns, and enhances weight extraction and state dict comparison utilities. Review feedback recommends using isinstance() for more robust type checking in the Gemma 4 vLLM patches.

[gemini-code-assist]

Using isinstance() for type checking is generally more robust than comparing class names as strings. It correctly handles subclasses and is considered better practice.

Suggested change

	if model.__class__.__name__ == "Gemma4ForConditionalGeneration":
	return True
	if isinstance(model, Gemma4ForConditionalGeneration):
	return True

[gemini-code-assist]

Similar to the patched_supports_lora function, using isinstance() for type checking is more robust than comparing class names.

Suggested change

	if model.__class__.__name__ == "Gemma4ForConditionalGeneration":
	if isinstance(model, Gemma4ForConditionalGeneration):

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces comprehensive support for Gemma4 and Qwen 3.5 (GDN) models, featuring specialized vLLM patches for LoRA and k_eq_v attention, and new extraction logic for Gated Delta Net layers. It refactors model finalization into a centralized utility, improves dtype configuration robustness, and enhances state dict comparison logic. Feedback was provided to simplify the check for empty quantization configurations using a more Pythonic approach.

[gemini-code-assist]

The condition (quantization_config or {}) == {} is a bit verbose for checking if quantization_config is None or an empty dictionary. A more Pythonic and readable way is to use if not quantization_config.

Suggested change

	if (quantization_config or {}) == {} and bnb_config is None:
	if not quantization_config and bnb_config is None:

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds support for Gemma 4 and Qwen 3.5 (Gated Delta Net) models, implementing specialized patches for vLLM integration, LoRA support, and a new model finalization workflow. Key changes include the finalize_huggingface_model utility, improved dtype configuration in hf_utils.py, and expanded layer extraction logic for GDN and vision architectures. The review feedback suggests adopting isinstance() for idiomatic type checking, using .detach() instead of .data for safer tensor operations, and correcting a regex pattern to ensure robust layer name mapping.

[gemini-code-assist]

Using isinstance() is generally preferred over comparing __class__.__name__ strings, as it is more robust and idiomatic. Since Gemma4ForConditionalGeneration is imported in the local scope, it can be used directly.

Suggested change

	if model.__class__.__name__ == "Gemma4ForConditionalGeneration":
	if isinstance(model, Gemma4ForConditionalGeneration):

[gemini-code-assist]

Using isinstance() is generally preferred over comparing __class__.__name__ strings. Since Gemma4ForConditionalGeneration is imported in the local scope, it can be used directly.

Suggested change

	if model.__class__.__name__ == "Gemma4ForConditionalGeneration":
	if isinstance(model, Gemma4ForConditionalGeneration):

[gemini-code-assist]

Using .detach() is generally preferred over .data in modern PyTorch as it is safer and more explicit about detaching the tensor from the computational graph.

Suggested change

	store(f"{prefix}.conv1d.weight", gdn.conv1d.weight.data)
	store(f"{prefix}.dt_bias", gdn.dt_bias.data)
	store(f"{prefix}.A_log", gdn.A_log.data)
	store(f"{prefix}.conv1d.weight", gdn.conv1d.weight.detach())
	store(f"{prefix}.dt_bias", gdn.dt_bias.detach())
	store(f"{prefix}.A_log", gdn.A_log.detach())

[gemini-code-assist]

The regex \.([\d]{1,})\. requires a trailing dot to match, which might fail if the numeric index is at the end of the string. A more robust approach is to match the dot and digits and replace them with bracket notation, consistent with the implementation used later in this file (line 1521).

Suggested change

	layer_name_br = re.sub(r"\.([\d]{1,})\.", r"[\1].", layer_name)
	layer_name_br = re.sub(r"\.([\d]{1,})", lambda x: f"[{x.group(1)}]", layer_name)

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for Gemma 4 and Qwen 3.5 (Gated Delta Net) architectures, focusing on vLLM integration and Hugging Face model conversion. Key additions include specialized patches for Gemma 4 LoRA and k_eq_v support, a finalize_huggingface_model utility for consistent model setup, and expanded layer configuration mappings. Review feedback identifies a missing v_proj duplication for Gemma 4 k_eq_v layers and suggests using more robust regex patterns for converting layer paths to bracket notation. Furthermore, it is recommended to replace string-based class name checks with isinstance() for better type safety in the newly added patches.

[gemini-code-assist]

The extraction logic for Gemma 4 k_eq_v layers currently skips v_proj entirely. However, the Hugging Face model architecture expects v_proj to be present in the state dict for all layers. For k_eq_v layers where K and V are shared, you should duplicate the K shard (index 1) into the v_proj prefix to ensure the resulting state dict is complete and passes validation.

Suggested change

	if kk not in gemma4_k_eq_v_layers:
	get_state_dict(f"{prefix}.v_proj", 2, state_dict, qkv_proj)
	if kk in gemma4_k_eq_v_layers:
	get_state_dict(f"{prefix}.v_proj", 1, state_dict, qkv_proj)
	else:
	get_state_dict(f"{prefix}.v_proj", 2, state_dict, qkv_proj)

[gemini-code-assist]

This regex is inconsistent with the more robust one used later in the same function (line 1521). The current regex \.([\d]{1,})\. requires a trailing dot, which would fail to match if a digit appears at the end of a path. Using a more general regex ensures all indexed paths are correctly converted to bracket notation.

Suggested change

	layer_name_br = re.sub(r"\.([\d]{1,})\.", r"[\1].", layer_name)
	layer_name_br = re.sub(r"\.([\d]{1,})", r"[\1]", layer_name)

[gemini-code-assist]

Inconsistent regex usage for path conversion. Using the more general regex pattern (matching line 1521) is safer as it correctly handles digits at any position in the path, including the end.

Suggested change

	# Set weight
	layer_name_br = re.sub(r"\.([\d]{1,})", r"[\1]", layer_name)

[gemini-code-assist]

Using string-based class name comparison is brittle and less efficient than a direct type check. Since Gemma4ForConditionalGeneration is already imported in this scope, it is better to use isinstance().

Suggested change

	if model.__class__.__name__ == "Gemma4ForConditionalGeneration":
	if isinstance(model, Gemma4ForConditionalGeneration):

[gemini-code-assist]

Using string-based class name comparison is brittle. Since Gemma4ForConditionalGeneration is available in this scope, using isinstance() is preferred for type checking.

Suggested change

	if model.__class__.__name__ == "Gemma4ForConditionalGeneration":
	if isinstance(model, Gemma4ForConditionalGeneration):

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for Gemma 4 and Qwen 3.5 (Gated Delta Net) architectures, focusing on vLLM to Hugging Face conversion and compatibility. Key additions include specialized patching for Gemma 4 LoRA and k_eq_v support, extraction logic for GDN layers, and a centralized finalize_huggingface_model utility to handle RoPE re-initialization and attribute cleanup. The feedback suggests replacing .data access with .detach() in several extraction functions to ensure safer tensor handling and avoid bypassing PyTorch's safety checks.

[gemini-code-assist]

Accessing .data is generally discouraged in modern PyTorch as it can bypass safety checks and lead to silent bugs if the tensor is modified in-place. Using .detach() is preferred when you need the underlying tensor without gradient tracking for storage in a state dictionary.

Suggested change

	store(f"{prefix}.conv1d.weight", gdn.conv1d.weight.data)
	store(f"{prefix}.dt_bias", gdn.dt_bias.data)
	store(f"{prefix}.A_log", gdn.A_log.data)
	store(f"{prefix}.conv1d.weight", gdn.conv1d.weight.detach())
	store(f"{prefix}.dt_bias", gdn.dt_bias.detach())
	store(f"{prefix}.A_log", gdn.A_log.detach())

[gemini-code-assist]

Prefer using .detach() over .data to safely obtain the underlying tensor without tracking gradients.

Suggested change

	state_dict[f"{vllm_text_model_prefix}.layers.{kk}.layer_scalar"] = layer.layer_scalar.data
	quant_state_dict[f"{vllm_text_model_prefix}.layers.{kk}.layer_scalar"] = layer.layer_scalar.data
	if hasattr(layer, "layer_scalar"):
	state_dict[f"{vllm_text_model_prefix}.layers.{kk}.layer_scalar"] = layer.layer_scalar.detach()
	quant_state_dict[f"{vllm_text_model_prefix}.layers.{kk}.layer_scalar"] = layer.layer_scalar.detach()

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces support for Gemma4 and Qwen 3.5 (Gated Delta Net) architectures. Key changes include new patching mechanisms for vLLM to support Gemma4 LoRA and bitsandbytes quantization, the addition of a centralized finalize_huggingface_model function to handle model finalization, and expanded layer configuration templates for GDN and vision towers. The set_dtype_in_config utility was also refactored for better compatibility with Transformers 5.x. Feedback focuses on improving the robustness of regex-based layer name conversion, ensuring safe tensor unpacking for conv1d layers, and adding defensive checks when accessing quantization block sizes.

[gemini-code-assist]

The regex \\.([\d]{1,})\\. requires a trailing dot to match a numeric index. If a layer name ends with a numeric index (e.g., model.visual.merger.mlp.0), it won't be converted to bracket notation, which will cause a syntax error in the subsequent exec call. Using a more robust regex that matches numeric indices regardless of trailing characters is recommended.

Suggested change

	layer_name_br = re.sub(r"\.([\d]{1,})\.", r"[\1].", layer_name)
	layer_name_br = re.sub(r"\\.([\d]{1,})", r"[\\1]", layer_name)

[gemini-code-assist]

Unpacking weight.shape into three variables assumes the tensor is 3D. If the extracted weight has a different number of dimensions, this will raise a ValueError. It's safer to verify the number of dimensions before unpacking to ensure robustness against unexpected weight shapes.

                if weight.ndim != 3:\n                    raise ValueError(f"Unsloth: Expected 3D weight for conv1d, got {weight.ndim}D")\n                out_ch, _in_per_group, kernel = weight.shape

[gemini-code-assist]

Accessing proj.weight_block_size[0] assumes the attribute exists and is a non-empty sequence. Using getattr with a default value would be safer to prevent potential AttributeError or TypeError in environments where the projection module might not have this attribute defined.

Suggested change

	block_size = proj.weight_block_size[0]
	block_size = getattr(proj, "weight_block_size", [128])[0]

[danielhanchen]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces comprehensive support for Gemma4 and Qwen3.5 (Gated Delta Net) models, including specialized patching for vLLM LoRA and k_eq_v support. Key changes include the addition of model finalization logic, refined state dict extraction for GDN layers, and improved configuration handling to ensure compatibility with newer Transformers versions. Feedback focuses on improving the robustness of attribute access in model finalization, refining regex patterns used for dynamic code execution to prevent syntax errors, and ensuring logging consistency by using the logger instead of print statements.

[gemini-code-assist]

This line will raise an AttributeError if _inner_model is None. This can occur if new_model does not have a model attribute, which is possible for certain multimodal architectures. It should safely handle the case where _inner_model is None before attempting to access its attributes.

Suggested change

	_layer_owners = [getattr(_inner_model, "language_model", None), _inner_model]
	_layer_owners = [getattr(_inner_model, "language_model", None) if _inner_model is not None else None, _inner_model]

[gemini-code-assist]

The regex \.([\d]{1,})\. requires a trailing dot to match. If a layer name ends with an index (e.g., model.visual.merger.mlp.0), it won't be converted to bracket notation (e.g., mlp[0]), leading to a SyntaxError in the subsequent exec call. Use a more robust regex that handles indices at the end of the string, consistent with the logic used elsewhere in this file (e.g., line 1527).

Suggested change

	layer_name_br = re.sub(r"\.([\d]{1,})\.", r"[\1].", layer_name)
	layer_name_br = re.sub(r"\.([\d]+)", lambda x: f"[{x.group(1)}]", layer_name)

[gemini-code-assist]

Similar to the issue at line 1441, this regex should be updated to handle indices at the end of the string to avoid potential syntax errors in exec.

Suggested change

	layer_name_br = re.sub(r"\.([\d]{1,})\.", r"[\1].", layer_name)
	layer_name_br = re.sub(r"\.([\d]+)", lambda x: f"[{x.group(1)}]", layer_name)

[gemini-code-assist]

Using logger.info is more consistent with the rest of the function (e.g., line 1865) and follows library best practices for logging.

Suggested change

	print("Unsloth: Gemma4 audio_tower weights are not extracted; audio-capable inference is not supported.")
	logger.info("Unsloth: Gemma4 audio_tower weights are not extracted; audio-capable inference is not supported.")

[danielhanchen]

Fixes pushed to unslothai#603.