Skip to content

Note bundled flash-linear-attention kernels for gated-deltanet models #6850

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
danielhanchen merged 4 commits into from
Jul 6, 2026
+39 −0
Merged

Note bundled flash-linear-attention kernels for gated-deltanet models #6850

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
11450bc
Note the bundled flash-linear-attention kernels for gated-deltanet mo…
danielhanchen Jul 4, 2026
f8b28b0
Tighten comments
danielhanchen Jul 4, 2026
a7d4531
Normalize model_types in fla install advisory for None and single string
danielhanchen Jul 4, 2026
bf974da
Cover olmo_hybrid in the gated-deltanet fla advisory
danielhanchen Jul 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions unsloth/models/loader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,44 @@ def _get_user_task_config_attrs(user_config):
"granite,llava_next", # Granite-vision 3
]

# Architectures with gated-deltanet (linear attention) layers. Unsloth bundles the
# flash-linear-attention Triton kernels (unsloth_zoo/_vendored/fla), so no install is
# needed; transformers uses the much slower pure PyTorch path only when they can't be enabled.
FLA_MODEL_TYPE_PREFIXES = ("qwen3_next", "qwen3_5", "kimi_linear", "olmo_hybrid")
_fla_advised = False


def _maybe_advise_fla_install(model_types):
"""One-time note when a gated-deltanet model loads without the fast kernels.

The kernels ship with Unsloth (no install needed); this fires only when they
could not be enabled on this platform (e.g. no CUDA, torch < 2.7 or
triton < 3.3), i.e. exactly when transformers uses the slow pure PyTorch path.
"""
global _fla_advised
if _fla_advised:
return
if model_types is None:
return
if isinstance(model_types, str):
model_types = [model_types] # a lone string would otherwise iterate chars
try:
if not any(
isinstance(t, str) and t.startswith(FLA_MODEL_TYPE_PREFIXES) for t in model_types
):
return
from transformers.utils.import_utils import is_flash_linear_attention_available
if is_flash_linear_attention_available():
return # bundled (or user-installed) fast kernels are active
Comment on lines +236 to +237

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Check causal-conv1d before suppressing the slow-path note

For Qwen3.5/Qwen3.5-MoE gated-delta layers, the transformers fast path requires both flash-linear-attention and causal-conv1d (is_fast_path_available = all((causal_conv1d_fn, causal_conv1d_update, chunk_gated_delta_rule, fused_recurrent_gated_delta_rule))). This return suppresses the new Unsloth note as soon as FLA is available, so a CUDA setup with bundled FLA but missing or broken causal_conv1d still falls back to the pure PyTorch path without the intended guidance. Please include the causal-conv1d availability gate for the Qwen prefixes before returning.

Useful? React with 👍 / 👎.

@danielhanchen danielhanchen Jul 5, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not the case for these archs. In transformers the qwen3_next / qwen3_5 / qwen3_5_moe forward selects each component independently: the conv uses causal_conv1d_fn when present else F.silu(self.conv1d(...)), and the gated-delta uses chunk_gated_delta_rule or torch_chunk_gated_delta_rule. is_fast_path_available only drives the warning_once message, it does not gate the forward. So with fla present but causal_conv1d missing, the expensive gated-delta linear attention still runs on the fla Triton kernels and only the cheap depthwise conv1d falls back to torch, not the whole forward. That is by design here: unsloth_zoo nullifies causal_conv1d on GPUs where its CUDA kernel is broken and relies on the torch conv plus fla kernels. Gating the advisory on fla is therefore correct, so keeping it as-is.

except Exception:
return
Comment on lines +216 to +239

@gemini-code-assist gemini-code-assist Bot Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To make _maybe_advise_fla_install more robust and defensive, we should handle cases where model_types is None, empty, or a single string (which would otherwise cause character-by-character iteration and fail to match).

Additionally, to ensure this advisory is triggered regardless of the loading path, consider calling _maybe_advise_fla_install(model_types) in FastLanguageModel.from_pretrained (around line 620) right after model_types is resolved.

def _maybe_advise_fla_install(model_types):
    """Print a one-time advisory when a gated-deltanet model is loaded without fla."""
    global _fla_advised
    if _fla_advised or not model_types:
        return
    if isinstance(model_types, str):
        model_types = [model_types]
    try:
        if not any(
            isinstance(t, str) and t.startswith(FLA_MODEL_TYPE_PREFIXES) for t in model_types
        ):
            return
        if importlib.util.find_spec("fla") is not None:
            return
    except Exception:
        return

Comment on lines +230 to +239

@gemini-code-assist gemini-code-assist Bot Jul 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If is_flash_linear_attention_available fails to import (for example, on older transformers versions or if there is an ImportError), the entire try block raises an exception, which is caught by except Exception: and silently returns. This prevents the warning from being printed, even though the fast kernels are not active and the model will run on the slower pure PyTorch path.

To make this more robust, we should isolate the import and check of is_flash_linear_attention_available so that any failure/absence of the import is treated as fla_available = False, allowing the warning to be printed correctly. Additionally, we should avoid broad, silent exception handlers (like except Exception:) and instead catch specific exceptions (like ModuleNotFoundError for optional imports) and log them at a debug level to aid in troubleshooting.

Suggested change
try:
if not any(
isinstance(t, str) and t.startswith(FLA_MODEL_TYPE_PREFIXES) for t in model_types
):
return
from transformers.utils.import_utils import is_flash_linear_attention_available
if is_flash_linear_attention_available():
return # bundled (or user-installed) fast kernels are active
except Exception:
return
try:
if not any(
isinstance(t, str) and t.startswith(FLA_MODEL_TYPE_PREFIXES) for t in model_types
):
return
except Exception as e:
import logging
logging.getLogger(__name__).debug("Error checking model types: %s", e)
return
try:
from transformers.utils.import_utils import is_flash_linear_attention_available
fla_available = is_flash_linear_attention_available()
except ModuleNotFoundError as e:
import logging
logging.getLogger(__name__).debug("transformers.utils.import_utils or is_flash_linear_attention_available not found: %s", e)
fla_available = False
if fla_available:
return # bundled (or user-installed) fast kernels are active
References
  1. Avoid using broad, silent exception handlers like except Exception: pass. Instead, log the exception, even if at a debug level, to aid in future debugging.
  2. When catching an ImportError for an optional dependency, prefer catching the more specific ModuleNotFoundError and check the module name to avoid suppressing unrelated import errors.
  3. When handling exceptions, avoid broad except Exception: pass clauses. Instead, catch specific exceptions and log them (at least at a debug level) to aid in troubleshooting. If a failure is expected, log the specific exception type and its details.

@danielhanchen danielhanchen Jul 5, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The broad except with a silent return is intentional here. This is a purely informational advisory, so any probe failure should never affect the load, and narrowing to ModuleNotFoundError risks letting an unexpected error from is_flash_linear_attention_available propagate into from_pretrained. In practice the import-failure branch is also unreachable for these archs: any transformers new enough to expose qwen3_next / qwen3_5 / olmo_hybrid already ships is_flash_linear_attention_available, so a missing advisory here is not a real gap.

_fla_advised = True
print(
"Unsloth: This model uses gated-deltanet linear attention layers. Unsloth\n"
"bundles the flash-linear-attention kernels, but they could not be enabled\n"
"on this setup (they need CUDA with torch >= 2.7 and triton >= 3.3), so\n"
"transformers will use a slower pure PyTorch path."
)

def _fix_rope_inv_freq(model):
"""Fix inv_freq corruption caused by transformers v5 meta-device loading.
Expand Down Expand Up @@ -1304,6 +1342,7 @@ def _dispatch_diffusion():
trust_remote_code = trust_remote_code,
)
model_types_all = ",".join(model_types) + ","
_maybe_advise_fla_install(model_types)

# ---- Text-diffusion models (e.g. DiffusionGemma) take a transformers-only slow path. ----
# These use a custom block-diffusion `generate` and a novel backbone, so we skip Unsloth's
Expand Down
Loading