Skip to content
32 changes: 30 additions & 2 deletions docs/fe-oss-apis/gemm_fusions/grouped_gemm_dglu.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ For jitted JAX programs use the `jax.jit`-compatible XLA custom-call entry point

**Unified Grouped GEMM + dGLU fusion**: one public class and wrapper select a
plain BF16 or legacy block-scaled grouped GEMM fused with a dGLU backward
epilogue (dSwiGLU or dGeGLU) on NVIDIA Blackwell GPUs (SM100+). The operation
epilogue (dSwiGLU, dGeGLU, or block-scaled dSiTU-GLU) on NVIDIA Blackwell GPUs (SM100+). The operation
is implemented with CUTLASS/CuTe DSL.

This is a **unified API** that supports both weight layout modes:
Expand All @@ -22,6 +22,7 @@ This is a **unified API** that supports both weight layout modes:
And both backward activation functions:
- **dSwiGLU**: `act_func="dswiglu"` (default)
- **dGeGLU**: `act_func="dgeglu"`
- **dSiTU-GLU**: `act_func="dsituglu"` (block-scaled SM100/SM103 only)

Groups are contiguous in the M dimension and described by `padded_offsets` (cumulative aligned end offsets).

Expand All @@ -34,6 +35,7 @@ Groups are contiguous in the M dimension and described by `padded_offsets` (cumu

Mixed families and unsupported pairs are rejected before allocation or
compilation. Each backend's argument contract is described below.
`dsituglu` is not available on the BF16 or Rubin backends.

## BF16 contract

Expand Down Expand Up @@ -125,6 +127,30 @@ The block-scaled backend performs:

### Equations

For dSiTU-GLU, define

$$
T_g=\beta_1\tanh(G/\beta_1)\sigma(G),\qquad
T_u=\beta_2\tanh(U/\beta_2).
$$

The fused backward computes

$$
\frac{\partial T_g}{\partial G}=
(1-\tanh^2(G/\beta_1))\sigma(G)+
\beta_1\tanh(G/\beta_1)\sigma(G)(1-\sigma(G)),
$$

$$
\frac{\partial T_u}{\partial U}=1-\tanh^2(U/\beta_2),
$$

and returns `ref * prob * T_u * dT_g/dG` and
`ref * prob * T_g * dT_u/dU`. `dprob` accumulates `ref * T_g * T_u`.
The beta values are compile-time specialization values and therefore belong to
the dGLU compiled-kernel cache key.

- **Inputs**
- `A`: contiguous activation tensor across all groups, shape `(valid_m, K, 1)`
- `B` (dense): weight tensor across all groups, shape `(N, K, L)`
Expand Down Expand Up @@ -495,7 +521,9 @@ Providing both or neither raises `ValueError`.
- `vector_f32`: Enable packed f32 operations. Default: `False`
- `m_aligned`: Must be `256`. Default: `256`
- `discrete_col_sfd`: Generate discrete col-major scale factors. Default: `False`
- `act_func`: Backward activation function. `"dswiglu"` (default) or `"dgeglu"`
- `act_func`: Backward activation function. `"dswiglu"` (default), `"dgeglu"`, or block-scaled `"dsituglu"`
- `situ_beta1`: Positive finite gate tanh scale for dSiTU-GLU. Default: `4.0`
- `situ_beta2`: Positive finite up-branch tanh scale for dSiTU-GLU. Default: `25.0`
- `b_major` (discrete only): B tensor major dimension. `"k"` (default) or `"n"`. Must be `"k"` for FP4.
- `epilogue_op`: Optional post-processing. `None` (default), `"identity"`, `"relu"`, or `"srelu"`

Expand Down
23 changes: 20 additions & 3 deletions docs/fe-oss-apis/gemm_fusions/grouped_gemm_glu.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ For jitted JAX programs use the `jax.jit`-compatible XLA custom-call entry point

**Unified Grouped GEMM + GLU fusion**: one public class and wrapper select a
plain BF16 or legacy block-scaled grouped GEMM fused with a GLU epilogue
(SwiGLU or GeGLU) on NVIDIA Blackwell GPUs (SM100+). The operation is
(SwiGLU, GeGLU, or block-scaled SiTU-GLU) on NVIDIA Blackwell GPUs (SM100+). The operation is
implemented with CUTLASS/CuTe DSL.

This is a **unified API** that supports both weight layout modes:
Expand All @@ -22,6 +22,7 @@ This is a **unified API** that supports both weight layout modes:
And both activation functions:
- **SwiGLU**: `act_func="swiglu"` (default)
- **GeGLU**: `act_func="geglu"`
- **SiTU-GLU**: `act_func="situglu"` (block-scaled SM100/SM103 only)

Groups are contiguous in the M dimension and described by `padded_offsets` (cumulative aligned end offsets).

Expand All @@ -34,6 +35,7 @@ Groups are contiguous in the M dimension and described by `padded_offsets` (cumu

Mixed families and unsupported pairs are rejected before allocation or
compilation. Each backend's argument contract is described below.
SiTU-GLU is not available on the BF16 or Rubin backends.

## BF16 contract

Expand Down Expand Up @@ -91,13 +93,26 @@ always `None`; `c_tensor` is `None` unless `generate_c=True`.

The block-scaled backend performs:
1. **Block-scaled grouped GEMM**: Low-precision GEMM (FP4, FP8) with per-block scale factors across multiple expert groups
2. **GLU activation**: Fused SwiGLU or GeGLU activation applied to the GEMM output
2. **GLU activation**: Fused SwiGLU, GeGLU, or SiTU-GLU activation applied to the GEMM output
3. **Optional quantized output**: Produces row and column scale factors for downstream quantization

### Shapes

### Equations

For SiTU-GLU, with gate branch `G` and up branch `U`, the fused epilogue computes

$$
D = \mathrm{prob}\,
\left[\beta_1\tanh(G/\beta_1)\sigma(G)\right]
\left[\beta_2\tanh(U/\beta_2)\right].
$$

where `beta_1 = situ_beta1` and `beta_2 = situ_beta2`, with defaults
`beta_1 = 4.0` and `beta_2 = 25.0`. `situ_beta1` specializes the compiled kernel
and is part of its cache key; `situ_beta2` is a runtime FP32 scalar and does not
create a new compiled-kernel cache entry.

- **Inputs**
- `A`: contiguous activation tensor across all groups, shape `(valid_m, K, 1)`
- `B` (dense): weight tensor across all groups, shape `(N, K, L)`
Expand Down Expand Up @@ -470,7 +485,9 @@ Providing both or neither raises `ValueError`.
- `vector_f32`: Enable packed f32 operations. Default: `False`
- `m_aligned`: Must be `256` (FIX_PAD_SIZE). Default: `256`
- `discrete_col_sfd`: Generate discrete col-major scale factors. Default: `False`
- `act_func`: Activation function. `"swiglu"` (default) or `"geglu"`
- `act_func`: Activation function. `"swiglu"` (default), `"geglu"`, or block-scaled `"situglu"`
- `situ_beta1`: Positive finite gate tanh scale for SiTU-GLU. Default: `4.0`
- `situ_beta2`: Positive finite up-branch tanh scale for SiTU-GLU. Default: `25.0`
- `b_major` (discrete only): B tensor major dimension. `"k"` (default) or `"n"`. Must be `"k"` for FP4.

### Wrapper-specific Parameters
Expand Down
33 changes: 29 additions & 4 deletions python/cudnn/gemm/cutedsl/grouped/dglu/_blockscaled_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

This module provides a single API class that supports both contiguous (dense)
and discrete weight modes for block-scaled grouped GEMM with dGLU activation
gradient (dSwiGLU / dGeGLU) in MoE (Mixture of Experts) workloads.
gradient (dSwiGLU / dGeGLU / dSiTU-GLU) in MoE (Mixture of Experts) workloads.

Dense mode
All expert weights are packed contiguously in a 3-D tensor (N, K, L).
Expand All @@ -23,6 +23,7 @@
from ..moe_utils import MoEWeightMode
from ..backend_utils import rubin_single_group_offsets_kwarg
from cuda.bindings import driver as cuda
import math
import os
from typing import Literal, Tuple, Optional

Expand Down Expand Up @@ -141,6 +142,8 @@ def __init__(
geglu_alpha: float = 1.702,
glu_clamp_max: float = 7.0,
glu_clamp_min: float = -7.0,
situ_beta1: float = 4.0,
situ_beta2: float = 25.0,
):
"""Initialize the GroupedGemmDgluSm100 API.

Expand Down Expand Up @@ -177,7 +180,7 @@ def __init__(
:param vector_f32: Use vectorized f32 operations
:param m_aligned: Alignment for group M dimension
:param discrete_col_sfd: Generate discrete col-major scale factor tensor
:param act_func: Activation function, one of "dswiglu" or "dgeglu"
:param act_func: Activation function, one of "dswiglu", "dgeglu", or "dsituglu"
:param b_major: Major dimension for B tensor, one of "k" or "n"
:param epilogue_op: Optional epilogue operation. Valid: None, "none", "identity", "relu", "srelu"
:param use_dynamic_sched: Enable dynamic tile scheduling for load balancing
Expand All @@ -190,6 +193,8 @@ def __init__(
``act_func == "dswiglu"``.
:param glu_clamp_min: Compile-time dGeGLU lower clamp. Ignored when
``act_func == "dswiglu"``.
:param situ_beta1: Compile-time gate tanh scale for dSiTU-GLU.
:param situ_beta2: Compile-time up-branch tanh scale for dSiTU-GLU.
"""
from cudnn.tensor_adapter import detect_framework

Expand Down Expand Up @@ -297,6 +302,17 @@ def __init__(
self.geglu_alpha = geglu_alpha
self.glu_clamp_max = glu_clamp_max
self.glu_clamp_min = glu_clamp_min
self.situ_beta1 = float(situ_beta1)
self.situ_beta2 = float(situ_beta2)
if self.act_func == "dsituglu":
self._value_error_if(
not math.isfinite(self.situ_beta1) or self.situ_beta1 <= 0.0,
f"situ_beta1 must be finite and positive, got {self.situ_beta1}",
)
self._value_error_if(
not math.isfinite(self.situ_beta2) or self.situ_beta2 <= 0.0,
f"situ_beta2 must be finite and positive, got {self.situ_beta2}",
)
_reject_unsupported_rubin_glu_tune_params(
self._is_rubin_kernel,
self.geglu_alpha,
Expand Down Expand Up @@ -602,8 +618,12 @@ def check_support(self) -> bool:

# ---- Activation function validation ----
self._value_error_if(
self.act_func not in ["dswiglu", "dgeglu"],
f"act_func must be 'dswiglu' or 'dgeglu', got {self.act_func}",
self.act_func not in ["dswiglu", "dgeglu", "dsituglu"],
f"act_func must be 'dswiglu', 'dgeglu', or 'dsituglu', got {self.act_func}",
)
self._not_implemented_error_if(
self._is_rubin_kernel and self.act_func == "dsituglu",
"Rubin grouped GEMM dGLU does not support dsituglu",
)

# ---- Discrete-mode-specific validation ----
Expand Down Expand Up @@ -742,6 +762,7 @@ def compile(self) -> None:
weight_mode=self.weight_mode,
act_func=self.act_func,
use_dynamic_sched=self.use_dynamic_sched,
**({"situ_beta1": self.situ_beta1} if not self._is_rubin_kernel else {}),
**rubin_single_group_offsets_kwarg(self._is_rubin_kernel, self.use_single_group_runtime_offsets),
# Only the Rubin kernel accepts sf_fp8_dtype_override, and check_support
# rejects "e5m3" unless _is_rubin_kernel -- the same flag that selected
Expand Down Expand Up @@ -997,6 +1018,8 @@ def _compile_dense(self, gemm_dglu, max_active_clusters, fake_stream) -> None:
"geglu_alpha": self.geglu_alpha,
"glu_clamp_max": self.glu_clamp_max,
"glu_clamp_min": self.glu_clamp_min,
"situ_beta1": self.situ_beta1,
"situ_beta2": self.situ_beta2,
}
)
_compiled_kernel = cute.compile(gemm_dglu, **compile_kwargs)
Expand Down Expand Up @@ -1196,6 +1219,8 @@ def _compile_discrete(self, gemm_dglu, max_active_clusters, fake_stream) -> None
"geglu_alpha": self.geglu_alpha,
"glu_clamp_max": self.glu_clamp_max,
"glu_clamp_min": self.glu_clamp_min,
"situ_beta1": self.situ_beta1,
"situ_beta2": self.situ_beta2,
}
)
_compiled_kernel = cute.compile(gemm_dglu, **compile_kwargs)
Expand Down
Loading