Release 1.27.0

[Anerudhan]

cuDNN Frontend v1.27.0 Release Notes

cuDNN Frontend v1.27.0 is the recommended version for cuDNN 9.24.0 and later releases.

New: Python-native cudnn.pygraph 🚀 🚀

cudnn.pygraph is now a Python-native graph class (#336). Graph structure — nodes, tensors, and parameters — lives in Python and is fully introspectable, while execution dispatches through pluggable backends: Python DSL engines and the cuDNN C++ backend.

• Graph IR — an engine-agnostic op DAG whose input/output port names match the C++ pybind kwarg names everywhere. 100% of the C++ op surface is covered (54 pointwise ops, 25 structured ops, the SDPA family, and matmul).
• Backend contract — engines.BaseEngine defines a propose_plans → build_plan → execute lifecycle, with each engine owning a stable engine_id in a reserved region.
• Router — create_execution_plans() produces a ranked plan list mixing Python plans with a single delegating entry for the cuDNN backend. Plan indices are two-level and stable, so select_plan() and the classic at-index APIs keep working.
• Classic-API compatibility is preserved: the same errors at validate(), the same conditional-output behavior, torch dtype acceptance, ragged (THD) offsets, and deserialize/build_plans passthrough.

See docs/python_graph_and_execution_backends.md for the full design.

Note: the internal pybind class pygraph was renamed to backend_graph (reachable only as cudnn._pybind_module.backend_graph). Nothing public imports the pybind name; the public cudnn.pygraph is now the Python class.

New: FROST engines 🚀 🚀

Open-source cuDNN engines written with CUTLASS primitives (#476):

• FROST engine implementations, kernel templates, and engine routing.
• Coverage for GEMM, grouped MoE matmul, fused epilogues, linear attention, and SDPA workloads, with FP4/FP8 and MXFP8 formats, variable-length sequences, recurrent states, masking, and quantized outputs.
• Unified plan discovery, ranking, selection, fallback, workspace handling, and execution reporting.
• Standalone GEMM benchmarks under benchmark/gemm/frost/ and Python tests. See python/cudnn/frost/README.md.

These engines are registered as Python engines and are selected through the new cudnn.pygraph router, so they are reachable from the graph API rather than only as standalone kernels — the linear-attention operations below are the first consumers of that path.

New: Linear attention — GDN 🚀 🚀

A new cudnn.linear_attention package (#476) provides gated linear-attention operations through the Graph API, as well as PyTorch custom operators, exported from cudnn.linear_attention.ops.

• Graph-API native. Each op is a thin adapter that executes cached single-node pygraphs (GDN/GDN_BWD).
• Two backends per operation — a FROST engine (default on SM100/SM103) with a cuTile engine as the fallback elsewhere.
• Registered through torch.library.custom_op, so they compose with autograd, torch.compile, and DDP.
• THD token-packed layout — [total_tokens, heads, dim] tensors plus cu_seqlens boundaries — with grouped-value attention (GVA/GQA) and per-sequence recurrent state ports (initial state in, final state out).
• Benchmarks live in benchmark/linear_attention/ (including a Dockerfile and benchmark_single_linear_attention.py), with tests under test/python/linear_attention/.

Updates to Graph API 🚀 🚀

SDPA

• Deterministic backward with linear workspace on Hopper (SDPA bwd: linear-workspace deterministic path on Hopper (cuDNN 9.25+) #398). On cuDNN 9.25+, use_deterministic_algorithm on sm90 can now route to an ordered-dQ engine whose workspace is linear — rather than quadratic — in sequence length. The faster dP-workspace path is still used when it fits the existing 256 MB limit (CUDNN_FRONTEND_ATTN_DP_WORKSPACE_LIMIT is honored). Long-sequence and THD/ragged deterministic training that previously failed to allocate now runs; results remain bitwise reproducible. No behavior change for cuDNN < 9.25 or other architectures.
• Mixed-form sequence lengths in SDPA forward (Support mixed-form sequence lengths in SDPA forward (cuDNN 9.25+) #430). The Q and KV sides may now independently choose per-batch (seq_len_*) or cumulative (cu_seq_len_*) representation. Paged-attention integrations that hold a cumulative Q prefix sum alongside per-batch KV lengths no longer need to materialize a KV-side prefix sum. Requires cuDNN 9.25.0 or later on the unified surface.
• cu_seq_len_q / cu_seq_len_kv are now exposed on the sdpa_fp8 Python binding (Expose cu_seq_len_q/kv on the sdpa_fp8 python binding #366).
• FP32 I/O is now rejected up front in the unified SDPA node, with the allowlist dtype check applied consistently on both SDPA paths (Reject FP32 I/O in the unified SDPA node #449, Follow-up to #449: allowlist I/O dtype check on both SDPA paths #454).

Data types and operations

• Added DOUBLE (F64) compute data type support for convolution and pointwise scaling attributes (Support F64 compute data type for convolutions #423).
• Causal conv1d Python bindings now precheck kernel sizes and raise range-specific ValueErrors for NCW (2–256), NWH (2–128), B2B projection (2–32), and B2B mixer (2–256) (Guard causal conv1d kernel sizes in Python bindings #465, Fix causal conv1d test collection with older cuDNN #472). This prevents an unsupported width-specialized NWH launch that could fault the CUDA context on SM90.
• The 9.22 reshape-mode attribute is now guarded at runtime rather than only at compile time, on both the graph API and the legacy OperationBuilder_v8 path (Reshape: guard the 9.22 reshape-mode attribute at runtime, not just at compile time #466, Reshape: apply the runtime version guard to the legacy OperationBuilder_v8 too #467).

Serialization and plan management

• Graph JSON v2 — repro extraction now emits UID-based graph JSON 2.0 (Use UID-based graph JSON v2 for repro extraction #280). Integer UIDs are the tensor-table identity used by node references, ragged-offset references, and tensor dumps, so anonymous tensors are no longer dropped and duplicate names no longer collapse. Missing UIDs are assigned before validation while preserving user-supplied UIDs; malformed versions, duplicate identities, and dangling references are rejected with typed errors.
• Ragged-offset tensor descriptors are now serialized in full — dimensions, strides, and data type — with nested offsets followed recursively and shared descriptors deduplicated (Minor errata to #280 #412).
• Graph::serialize accepts a serialize_structure flag (default true), making it symmetric with the handle-based deserialize path and enabling plan-only round trips after deserialize(handle, ...) (Make plan structure serialization optional within serialize() to construct symmetry with deserialize logic #371).
• Documented the behavior of serializing a selected plan (Serialize selected plan behavior notes #364).

Build and integration

• No-RTTI builds are now supported: dynamic_cast has been removed from the public headers (No rtti headers #477).
• Added a PEP 735 dev dependency group, so pip install --group dev works as a prerequisite to deprecating requirements.txt (Add pip install --group dev, prerequisite for deprecating requirements.txt #359).

Open-Source Kernels 🚀 🚀

SDPA

• Native SM100 d192/d128 SDPA forward kernel for DSv3 MLA (Add native SM100 d192/d128 SDPA fprop kernel for DSv3 MLA #488). Instead of selecting a padded d=256 flavor, the D_QK = 192 / D_V = 128 logical shape now has a dedicated Blackwell prefill kernel (BF16 and FP16, dense CGA2 classic pipeline). For top-left causal S=8192 it reaches 87% compute SOL / 820 useful TFLOPS — roughly 1.5× a padded d=256 proxy. The existing d128, d256, and d512 kernels are unchanged.
• The OSS SDPA prefill SM100 engine is now restricted to sm_100, with samples skipped elsewhere (OSS SDPA prefill: restrict the SM100 engine to sm_100 and skip samples elsewhere #473).

GEMM fusions

• GEMM + RoPE + MXFP8 fusion (GEMM+RoPE+MXFP8 fusion #367) — an experimental SM100+ kernel fusing the projection GEMM, per-head RoPE, and dual-direction MXFP8 quantization, intended for DeepSeek-V3 forward training. Returns row-wise and column-wise FP8 outputs with E8M0 scale tensors. A second variant specialized for MXFP8 inputs was added, and the original kernel renamed to the _bf16in flavor, so recipes that project in either BF16 or MXFP8 are covered (Add an mxfp8-input version of the gemm_proj_rope_mxfp8 kernel. #438).
• BF16 grouped GEMM MoE kernels (feat: add BF16 grouped GEMM MoE kernels #415) — BF16 grouped GEMM, GLU, dGLU, and WGrad APIs for both dense and discrete MoE layouts, routed through the shared public APIs alongside the block-scaled implementations.
• Rubin (SM107) support — new block-scaled grouped GEMM GLU/dGLU/quant kernels with architecture-aware dispatch and cache keys (Add CuTe DSL Rubin (sm107) grouped GEMM GLU/dGLU/quant kernels #447), a Rubin WGRAD kernel (Integrate Rubin wgrad cutedsl kernel and fix an issue in rebased rubin branch #456), FP8 K-tiling and output-staging tuning (Tune Rubin FP8 K tiling and output staging #457), and compatibility across Blackwell CUTLASS wheels (Fix Rubin WGRAD compatibility across Blackwell CUTLASS wheels #464). Options the Rubin kernels do not support raise NotImplementedError rather than silently producing invalid results.
• Grouped GEMM quantization no longer requires a probability tensor (Allow grouped GEMM quant without probability tensor #458), and single-group grouped GEMM metadata handling was optimized.
• The GEMM fusion sources were reorganized for clarity (Reorganize Gemm fusion #459).
• MXFP8 performance improvements (MXFP8 Perf Improvement #481).

DSA (DeepSeek Sparse Attention)

• FP8/MXFP8 and compressed Top-K indexer paths (DSA: Add FP8/MXFP8 and compressed Top-K indexer paths #370). Adds an SM90 FP8 indexer path (E4M3 Q/K with per-token/head FP32 descales) and SM100 MXFP8 indexer and dense score-recompute paths (E4M3 Q/K with packed E8M0 block scales), for BSHD and THD inputs including compact padded MXFP8 scale layouts. The new SM100-only indexer_forward_top_k_wrapper produces Top-K indices, selected logits, optional fused softmax, and optional LSE without materializing the dense score tensor; deterministic=True resolves K-th-boundary ties toward the smallest local KV indices. Existing BF16 paths are preserved.
• Added SM90 DSA qh16 indexer forward support (Support SM90 DSA qh16 indexer forward and fix qh32 sparse backward #388).
• indexer_forward_wrapper accepts an optional pre-allocated out tensor, avoiding repeated internal allocation in iterative calls (add out parameter for dsa api #470).

CSA (Compressor)

• Fused Compressor forward and backward CuTe-DSL kernels ported from Megatron-LM (CSA: add fused Compressor forward+backward CuTe-DSL kernels (ported from Megatron-LM) #427). The gated-softmax pooling region between the two projection GEMMs and the RMSNorm — gather-index build, gather, + APE, overlap-window transform, fp32 windowed softmax, gated weighted sum — collapses from roughly 39 forward and 51 backward kernel launches per call into one forward and one backward kernel. Two follow-on optimizations (32-bit vectorized forward access; kernel-side zero-writes in the backward) give 1.20–1.36× on the forward kernel and 1.21–1.50× on the backward region while leaving forward, dKV, and dScore bitwise unchanged.
• The compressor now requests ex2.approx.ftz through fastmath= so it builds at the supported cutlass-dsl floor (CSA: request ex2.approx.ftz through fastmath= so the compressor builds at the cutlass-dsl floor #463).

Block-sparse attention (BSA)

• Removed dead fragment allocations (BSA: remove dead SM100 block-128 fragment allocations #392).
• Fixed a backward hang on cute-dsl 4.6.0 and corrected the bulk-copy elect_one gate for 4.6.2 and 4.7 (Fix BSA backward hang on cute-dsl 4.6.0: version-gate elect_one around bulk stats copies #382, BSA: fix the cute-dsl bulk-copy elect gate for 4.6.2 and 4.7 #453).

Toolchain

• Moved to nvidia-cutlass-dsl 4.6.0 (Update nvidia-cutlass-dsl version to 4.6.0 #368) and cleared CUTLASS DSL deprecation warnings across the CuTe DSL kernels, including the .ptr migration for cute.struct scalar fields (Fix cutlass DSL deprecation: use .ptr for cute.struct scalar fields #365, Fix cutlass DSL deprecation warnings in CuTe DSL kernels #376).

Tooling and Developer Experience ✨✨

• python -m cudnn.collect_env (Add collect_env environment report tool for bug reports #400) — a new environment-forensics tool for bug reports, wired into the issue template and README. It reports the frontend version with mismatch flags (imported cudnn.__version__ vs. pip metadata vs. torch.backends.cudnn.version()), traces the frontend's libcudnn dlopen search order, and distinguishes loaded from installed GPU libraries by parsing /proc/self/maps — flagging the version-confusion cases that dominate unreproducible issues. It is stdlib-only with individually guarded probes, so it still produces a report when import cudnn or torch is broken, and can be run standalone.
• Agent- and contributor-friendly repository layout (Make the repository agent-friendly: AGENTS.md guides, llms.txt, skills discovery #402, Add Claude Code project instructions #413) — a root AGENTS.md plus scoped guides under include/cudnn_frontend/, python/cudnn/, test/, and samples/; an llms.txt docs index; skills discovery for coding agents; and expanded CONTRIBUTING.md sections on development environment, testing, and formatting.
• Improved GitHub issue and pull request templates (Improve GitHub issue and pull request templates #375), with PR label guidance documented in the template and AGENTS.md (docs: add label guidance to PR template and AGENTS.md #489).
• Documentation fixes for FE-OSS links and the DSA architecture code fence (Fix FE-OSS SDPA links and DSA diagram fence #360), and clearer wording throughout the environment-report docs (Use professional wording in env report docs #409).

Samples, Benchmarks and Tests 📊

• LayerNorm and RMSNorm samples now accept (B, S, H) tensor shapes directly instead of reshaping to (B*S, H, 1, 1) (Norm Samples updates for B,S,H style tensor inputs #432).
• Added NWH and back-to-back causal conv1d notebooks with refreshed outputs (Add NWH + B2B causal conv1d notebooks; refresh outputs #246).
• Updated SDPA benchmarking artifacts for cuDNN 9.24.0.43 (Update SDPA Benchmarking Artifacts - 9.24.0.43 #362) and 9.25.0.15 (Update SDPA Benchmarking Artifacts - 9.25.0.15 #417).
• Fixed repo-root path resolution in the bench_moe benchmark (benchmark: fix bench_moe repo root path resolution #348).
• Reorganized the FE OSS tests by feature (Organize FE OSS tests by feature #372) and the GEMM projection tests (Fix broken paths in !367, clang-format #374).
• Added SDPA edge-case coverage (Add SDPA edge case test coverage #328) and fixed architecture-independent SDPA repro failures (Fix architecture-independent SDPA repro failures #386).
• test_mhas_v2: extended backward random head-dim coverage to d=256 (test_mhas_v2: extend bwd random head-dim coverage to d=256 #425); removed ALiBi, score_max/sum_exp, dropout randomization (Remove alibi, score_max/sum_exp, and dropout randomization from test_… #435) and with_rope (test_mhas_v2: remove with_rope from randomized tests #474) from the random forward/backward tests.

Bug Fixes 🐛

C++ frontend

• Fixed Engine_v8::Knob::getMaxValue(), which returned the minimum value (fix: Engine_v8::Knob::getMaxValue() returns the minimum value #443).
• Fixed an uncaught ValueError in flatten_pass_by_value on malformed hex input (Fix uncaught ValueError in flatten_pass_by_value on malformed hex input #343).
• Fixed a handle leak in the BatchNorm sample (Fix BatchNorm sample handle leak issue #399).

Python / OSS kernels

• Fixed a default-stream race: _get_default_stream(None) now resolves to torch's current stream (Fix default-stream race: resolve _get_default_stream(None) to torch's current stream #483).
• Fixed a block-scale quantize output alignment issue (fix-bs-quantize-output-alignment-issue #491).

DSA

• Fixed offset alignment, stream handling, and CUDA Graph capture in the indexer-backward paths; grad_loss is now consistently a single-element FP32 CUDA tensor (Fix DSA offset alignment, stream handling, and CUDA Graph capture #354).
• Fixed two latent synchronization bugs in the SM100 DSA backward kernel (Fix two latent synchronization bugs in the SM100 DSA backward kernel #395) and a latent TMEM WAR race in the SM100 dKV drain at head_dim = 576 (Fix latent TMEM WAR race in the SM100 DSA backward dKV drain (head_dim 576) #396).
• Fixed a latent SMEM handoff race in the SM100 DSA indexer backward kernel (Fix a latent SMEM handoff race in the SM100 DSA indexer backward kernel #426).
• Fixed stream-ordering and input-validation holes in the SM100 DSA backward interface (Fix stream-ordering and input-validation holes in the SM100 DSA backward interface #429).
• Fixed SM100 DSA sink normalization (DSA: fix SM100 sink normalization for 576/512 backward #421) and zero top-k lengths in the SM100 DSA backward (DSA backward SM100: support zero-length top-k rows in the kernel #439).
• Fixed the qh32 sparse backward path (Support SM90 DSA qh16 indexer forward and fix qh32 sparse backward #388).
• indexer_top_k now falls back to scalar stores for odd top_k (DSA indexer_top_k: fall back to scalar stores for odd top_k #407), and out-of-bounds lanes in the variable-length indexer top-k are fixed (Fix IMA caused by OOB lanes in varlen indexer top-k #410).

Licensing 📜

• cuDNN Frontend is now Apache-2.0. NVIDIA-authored sources have been relicensed from MIT to the Apache License 2.0 using the standard NVIDIA OSS SPDX header (Relicense to Apache-2.0 (dual-license; external-contributor files remain MIT) #408). Files that still carry surviving lines from external contributors whose permission has not yet been established remain under MIT, as do FlashAttention- and QuACK-derived files. The package metadata is now license = "Apache-2.0 AND MIT".
• Every source file now carries exactly one SPDX-License-Identifier tag. The complete per-file mapping — including the commit that introduced each surviving external line — is in LICENSING.md, alongside LICENSE.txt (Apache-2.0), LICENSE-MIT.txt, NOTICE, and THIRD_PARTY_LICENSES.txt.
• This change is comment- and header-only; there is no functional or API impact.

Acknowledgements 🙏

Thanks to everyone who contributed to this release:

@adshen, @Anerudhan, @bmanthos, @brandonfzhang, @chaseblock, @derdrdirk, @egilliam-nv, @fallintoplace, @hwanseoc, @hxbai, @JackRao123, @jhjpark, [@jiefan] @jiayus-nvidia, @kangbintNV, @kunlunl, @liujane-dev, @pmdavies-nv, @rmhaskarnvidia, @saltyminty, @sraman-rgb, @terminator123, @vedaanta, @vincejhan, @WanZzzzzz, @yanqinz2, @yanzhuo607, @YangXu1990uiuc, @yeliu-oss, and @zkyue.

Special thanks for the kernel contributions that came from outside this repository:

• The fused CSA Compressor forward and backward kernels were ported from Megatron-LM, following maintainer guidance on Megatron-LM PR #5984.
• The BF16 grouped GEMM, GLU, dGLU, and WGrad kernels originate from the CuTe DSL kernel library.
• The FROST / cuTile engines were developed with contributions from across the cuDNN team.

Summary by CodeRabbit

• New Features
  • Added FlashQLA as a selectable backend for supported GDN forward and backward benchmarks.
  • Added a plotting tool for generating per-batch Forward and Backward TFLOPS comparison charts.
• Documentation
  • Documented FlashQLA benchmark usage, supported configurations, and backend distinctions.
• Benchmarking
  • Improved benchmark handling for backend-specific tensor layouts and reference validation.
  • Added support for filtering chart results by variant and batch size, with GPU and cuDNN metadata.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR adds FlashQLA support to the linear-attention benchmark, updates its container and documentation, and introduces a CSV plotting tool for per-batch Forward and Backward TFLOPS charts.

Changes

FlashQLA linear-attention benchmarking

Layer / File(s)	Summary
Benchmark environment and usage benchmark/linear_attention/Dockerfile, benchmark/linear_attention/README.md	The Docker image installs FlashQLA and plotting dependencies. The README documents FlashQLA commands, supported backends, and GDN restrictions.
FlashQLA benchmark execution benchmark/linear_attention/benchmark_single_linear_attention.py	The CLI accepts flash_qla, validates supported configurations, calls flash_qla.chunk_gated_delta_rule, and preserves dense tensor layouts for FlashQLA and FLA.
Benchmark result plotting benchmark/linear_attention/plot_results.py	A CLI validates benchmark CSV data, filters results, and creates per-batch Forward and Backward TFLOPS charts with backend and hardware metadata.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant BenchmarkCLI
  participant benchmark_single_linear_attention
  participant FlashQLA
  BenchmarkCLI->>benchmark_single_linear_attention: select flash_qla backend
  benchmark_single_linear_attention->>FlashQLA: call chunk_gated_delta_rule
  FlashQLA-->>benchmark_single_linear_attention: return dense attention output

Loading

Possibly related PRs

• NVIDIA/cudnn-frontend#501: Related FlashQLA support, GDN benchmarking, container dependencies, and plotting changes.

Suggested labels: cat-feature

Suggested reviewers: jhjpark

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description provides detailed release notes but omits the required checklist, affected area, why, related issues, API impact, and testing sections.	Add the template sections and complete the checklist, affected area, rationale, related issues, compatibility impact, and exact testing results.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies this as the 1.27.0 release, which matches the pull request objectives.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 1.27.0-rc

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (2)

benchmark/linear_attention/Dockerfile (1)

22-30: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Pin the new benchmark dependencies for reproducible results.

flash-linear-attention, apache-tvm-ffi, the FlashQLA clone, and the plotting packages are unpinned. The measured TFLOPS depend on the kernel versions, so an image rebuilt later can produce different numbers without any repository change. The FLA version also controls whether FLA_DISABLE_BACKEND_DISPATCH exists, which the README relies on to keep the fla and flash_qla backends distinct.

Pin a FlashQLA tag or commit and pin the pip versions.

♻️ Suggested pinning

 # Install the Cutlass DSL runtime (cuDNN FROST engines) and FLA.
-RUN pip install nvidia-cutlass-dsl[cu13]==4.7.0 apache-tvm-ffi flash-linear-attention
+RUN pip install nvidia-cutlass-dsl[cu13]==4.7.0 apache-tvm-ffi==<version> flash-linear-attention==<version>
 
 # Install FlashQLA from source.
-RUN git clone https://github.com/QwenLM/FlashQLA.git
+RUN git clone --depth 1 --branch <tag-or-commit> https://github.com/QwenLM/FlashQLA.git
 RUN pip install -v /workspace/FlashQLA
 
 # Install the chart dependencies for plot_results.py
-RUN pip install pandas matplotlib seaborn
+RUN pip install pandas==<version> matplotlib==<version> seaborn==<version>

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@benchmark/linear_attention/Dockerfile` around lines 22 - 30, Pin all
benchmark dependencies in the Dockerfile for reproducible results: specify
versions for flash-linear-attention, apache-tvm-ffi, pandas, matplotlib, and
seaborn, and clone FlashQLA at a fixed tag or commit before installing it.
Ensure the selected flash-linear-attention version retains the
FLA_DISABLE_BACKEND_DISPATCH behavior required by the README.

benchmark/linear_attention/plot_results.py (1)

56-56: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Use explicit Optional for the parameters that default to None.

Ruff reports RUF013 on cudnn_version: str = None and batch_sizes: list = None. PEP 484 prohibits implicit Optional.

♻️ Proposed fix

 import argparse
 from pathlib import Path
+from typing import List, Optional

-def get_backend_display_name(backend: str, cudnn_version: str = None) -> str:
+def get_backend_display_name(backend: str, cudnn_version: Optional[str] = None) -> str:

-def generate_charts(df: pd.DataFrame, output_dir: Path, gpu_name: str = "", cudnn_version: str = None, variant: str = "gdn", batch_sizes: list = None) -> list:
+def generate_charts(
+    df: pd.DataFrame,
+    output_dir: Path,
+    gpu_name: str = "",
+    cudnn_version: Optional[str] = None,
+    variant: str = "gdn",
+    batch_sizes: Optional[List[int]] = None,
+) -> List[Path]:

Also applies to: 63-63

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@benchmark/linear_attention/plot_results.py` at line 56, Update the type
annotations for the parameters defaulting to None in get_backend_display_name
and the related function at the referenced declaration, importing Optional from
typing if needed and changing their string/list types to explicit Optional types
while preserving the existing defaults and behavior.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@benchmark/linear_attention/benchmark_single_linear_attention.py`:
- Around line 516-536: Pin the FlashQLA dependency to a tested commit or release
in the Docker setup that currently clones main. Update the dependency source or
checkout configuration used by the flash_qla benchmark path, while preserving
the existing flash_qla_linear_attention integration and ensuring builds
consistently use the pinned revision.

In `@benchmark/linear_attention/plot_results.py`:
- Around line 99-105: Update the chart-title construction in the plotting flow
to avoid always labeling results as BF16. Add a main CLI argument for the
data-type label, defaulting to BF16, and pass that value into the title instead
of the hardcoded precision text.

---

Nitpick comments:
In `@benchmark/linear_attention/Dockerfile`:
- Around line 22-30: Pin all benchmark dependencies in the Dockerfile for
reproducible results: specify versions for flash-linear-attention,
apache-tvm-ffi, pandas, matplotlib, and seaborn, and clone FlashQLA at a fixed
tag or commit before installing it. Ensure the selected flash-linear-attention
version retains the FLA_DISABLE_BACKEND_DISPATCH behavior required by the
README.

In `@benchmark/linear_attention/plot_results.py`:
- Line 56: Update the type annotations for the parameters defaulting to None in
get_backend_display_name and the related function at the referenced declaration,
importing Optional from typing if needed and changing their string/list types to
explicit Optional types while preserving the existing defaults and behavior.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1f34c212-c849-4df5-a598-3b3516198804

📥 Commits

Reviewing files that changed from the base of the PR and between 67c7634 and 0018f8d.

⛔ Files ignored due to path filters (2)

• benchmark/linear_attention/results/gdn/b300/gdn.png is excluded by !**/*.png
• benchmark/linear_attention/results/gdn/b300/gdn_20260806.csv is excluded by !**/*.csv

📒 Files selected for processing (4)

• benchmark/linear_attention/Dockerfile
• benchmark/linear_attention/README.md
• benchmark/linear_attention/benchmark_single_linear_attention.py
• benchmark/linear_attention/plot_results.py

[coderabbitai]

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

FlashQLA QwenLM chunk_gated_delta_rule signature initial_state output_final_state

💡 Result:

In the FlashQLA library, chunk_gated_delta_rule is the primary high-level function used to perform fused linear attention computation, specifically for Gated Delta Networks (GDN) [1][2][3]. This operator implements the chunk-wise processing of the gated delta rule, which is designed for efficient training and inference of linear attention models [4]. The signature for the function in the FlashQLA high-level API is as follows [1][5][3]: o, final_state = chunk_gated_delta_rule( q=q, # [B, T, H_q, K] (queries) k=k, # [B, T, H_q, K] (keys) v=v, # [B, T, H_v, V] (values) g=g, # [B, T, H_v] (gating tensor) beta=beta, # [B, T, H_v] (beta tensor) scale=scale, # Scaling factor (float) initial_state=initial_state, # [B, H_v, K, V] (optional) output_final_state=True, # Boolean flag (default True) cu_seqlens=cu_seqlens # Optional for variable-length sequences) Key parameters and concepts: - initial_state: This optional parameter allows the model to pass a hidden state from a previous chunk or sequence segment, which is essential for autoregressive inference or processing long sequences [1][3]. It typically has a shape of [B, H_v, K, V]. - output_final_state: When set to True, the function returns the final hidden state of the computation alongside the output (o) [1][5][3]. This final_state can then be cached or used as the initial_state for subsequent chunks in sequence generation [6]. FlashQLA serves as a high-performance backend for Qwen's linear attention implementations, and is often used alongside or as an alternative to the Flash Linear Attention (FLA) library, which shares similar API signatures for its own chunk-wise implementations [7][8].

Citations:

• 1: https://github.com/QwenLM/FlashQLA
• 2: https://github.com/qwenlm/flashqla
• 3: https://pypi.org/project/flash-qla/0.1.2/
• 4: https://qwen.ai/blog?id=flashqla
• 5: https://github.com/QwenLM/FlashQLA/blob/main/README.md
• 6: FlashQLA GDN backend produces different tokens from FLA Triton on correctness-test prompts QwenLM/FlashQLA#15
• 7: https://github.com/fla-org/flash-linear-attention/blob/78e4488f/fla/ops/gated_delta_rule/chunk.py
• 8: https://github.com/vllm-project/vime/blob/main/vime_plugins/models/qwen_gdn_backend.py

---

🏁 Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- relevant benchmark context ---'
sed -n '420,555p' benchmark/linear_attention/benchmark_single_linear_attention.py
printf '%s\n' '--- FlashQLA references and dependency declarations ---'
rg -n -i 'flash.?qla|chunk_gated_delta_rule|num_kv_heads|store_on' .
printf '%s\n' '--- tracked dependency/config files ---'
git ls-files | rg '(^|/)(README|requirements|pyproject|setup|environment|Dockerfile|.*ya?ml)' | head -200

Repository: NVIDIA/cudnn-frontend

Length of output: 50377

---

🌐 Web query:

site:github.com/QwenLM/FlashQLA chunk_gated_delta_rule source signature grouped heads num_kv_heads

💡 Result:

The chunk_gated_delta_rule is a high-performance linear attention kernel provided by the FlashQLA library, which is designed as a drop-in, faster alternative to existing Triton-based implementations for the Gated Delta Rule (GDN) [1][2][3]. The function signature for chunk_gated_delta_rule in FlashQLA is as follows: o, final_state = chunk_gated_delta_rule( q=q, # [B, T, H_q, K] k=k, # [B, T, H_q, K] v=v, # [B, T, H_v, V] g=g, # [B, T, H_v] beta=beta, # [B, T, H_v] scale=scale, initial_state=initial_state, # optional, [B, H_v, K, V] output_final_state=True, cu_seqlens=cu_seqlens, # optional, for variable-length sequences) Regarding grouped heads and num_kv_heads: The library supports Grouped Query Attention (GQA) configurations [3]. The head dimensions (H_q, H_v) and their respective head sizes (K, V) handle multi-head or grouped-head structures by passing the appropriate tensor shapes [1][2]. Specifically, the library has been benchmarked with various head configurations (h_k,v) typical of large language model families like Qwen, where the number of KV heads is smaller than the number of query heads to optimize memory and computation [2]. Users manage these configurations by ensuring the input tensors q, k, v, g, and beta align with the model's architectural requirements for H_q and H_v [1].

Citations:

• 1: https://github.com/QwenLM/FlashQLA
• 2: https://github.com/QwenLM/FlashQLA/blob/main/README.md
• 3: Add Hopper (SM90) GDN fused-recurrent decode + SGLang verify kernels (TileLang) QwenLM/FlashQLA#20

---

🌐 Web query:

site:github.com/QwenLM/FlashQLA "chunk_gated_delta_rule" "output_final_state"

💡 Result:

In the FlashQLA library, chunk_gated_delta_rule is a high-performance kernel implementing the Gated Delta Rule (GDN) for linear attention [1][2]. When the parameter output_final_state is set to True, the function returns both the attention output and the final recurrent state, which can be used to continue the sequence in subsequent calls [1][2]. The standard usage signature is [1][2]: o, final_state = chunk_gated_delta_rule( q=q, k=k, v=v, g=g, beta=beta, scale=scale, initial_state=initial_state, output_final_state=True, cu_seqlens=cu_seqlens) The final_state represents the accumulated hidden state at the end of the chunk processing, typically used for autoregressive generation or maintaining continuity across sequences [1][2][3]. In the underlying C++/CUDA implementations, the forward kernel (chunk_gated_delta_rule_fwd) explicitly computes and returns this final_state, while the backward kernel (chunk_gated_delta_rule_bwd) uses it (or its gradient) to facilitate backpropagation through the state [1][2]. Performance optimizations for this path, such as coalesced memory access, have been implemented for specific hardware architectures like NVIDIA Hopper (SM90) [4].

Citations:

• 1: https://github.com/QwenLM/FlashQLA/blob/main/README.md
• 2: https://github.com/QwenLM/FlashQLA
• 3: FlashQLA GDN backend produces different tokens from FLA Triton on correctness-test prompts QwenLM/FlashQLA#15
• 4: Add Hopper (SM90) GDN fused-recurrent decode + SGLang verify kernels (TileLang) QwenLM/FlashQLA#20

---

🏁 Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- benchmark output and state handling ---'
sed -n '580,625p' benchmark/linear_attention/benchmark_single_linear_attention.py
sed -n '730,815p' benchmark/linear_attention/benchmark_single_linear_attention.py
printf '%s\n' '--- FlashQLA installation lines ---'
sed -n '1,45p' benchmark/linear_attention/Dockerfile
printf '%s\n' '--- README FlashQLA section ---'
sed -n '55,112p' benchmark/linear_attention/README.md
printf '%s\n' '--- FlashQLA repository tree ---'
curl -fsSL https://api.github.com/repos/QwenLM/FlashQLA/git/trees/main?recursive=1 |
  jq -r '.tree[] | select(.path|test("(^|/)(flash_qla|README|setup|pyproject|chunk|gated)"; "i")) | [.type,.path,.url] | `@tsv`' |
  head -100

Repository: NVIDIA/cudnn-frontend

Length of output: 10721

---

🏁 Script executed:

#!/bin/bash
set -eu
python3 - <<'PY'
import json
import urllib.request

base = "https://api.github.com/repos/QwenLM/FlashQLA"
tree = json.load(urllib.request.urlopen(base + "/git/trees/main?recursive=1"))["tree"]
paths = [x["path"] for x in tree if x["type"] == "blob"]
for path in paths:
    if any(token in path.lower() for token in ("chunk", "gated", "flash_qla", "setup", "pyproject", "readme")):
        print(path)
PY

Repository: NVIDIA/cudnn-frontend

Length of output: 2445

---

🌐 Web query:

QwenLM FlashQLA GitHub source chunk_gated_delta_rule gva num_q_heads num_kv_heads

💡 Result:

The chunk_gated_delta_rule is a core operation used in Gated Delta Networks (GDN), particularly associated with architectures like Qwen3-Next [1][2]. It is implemented in high-performance linear attention libraries such as FlashQLA (developed by QwenLM) [3][4] and the Flash Linear Attention (fla) library [5][6]. The chunk_gated_delta_rule function is designed to handle the fused computation of gated linear attention mechanisms [7]. While specific implementation signatures may vary slightly by library, the kernel generally operates on query (q), key (k), value (v), gate (g), and beta parameters [3][4][5]. Regarding the parameters mentioned: 1. num_q_heads and num_kv_heads: These refer to the number of heads for query and key/value tensors, respectively, which are standard in Multi-Query Attention (MQA) or Grouped-Query Attention (GQA) architectures [8][9][2]. In the context of chunk_gated_delta_rule, these dimensions define the shape of the input tensors (e.g., [B, T, H_q, K] for q/k and [B, T, H_v, V] for v) [3][4]. The libraries ensure that the number of heads allows for the necessary grouping or expansion operations required by the specific attention mechanism [9]. 2. chunk_gated_delta_rule: This is a high-level API function that executes the linear attention kernel [3][4]. It allows for efficient computation over long sequences by processing them in chunks, often utilizing CUDA kernels for improved performance [3][5]. The implementation handles the internal state, gating, and decay operations (often involving cumsum and masking) [5][1]. Repositories such as FlashQLA and fla provide both high-level Python APIs and low-level C++/CUDA-based forward/backward kernels for these operations [3][4][5]. Tools like flashinfer also support these routines for benchmarking and inference tasks [10]. Top results: [3][4][5][2][10]

Citations:

• 1: https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen3_next/modeling_qwen3_next.py
• 2: https://github.com/turboderp-org/exllamav3/blob/cb1a436f/exllamav3/architecture/qwen3_next.py
• 3: https://github.com/QwenLM/FlashQLA
• 4: https://github.com/QwenLM/FlashQLA/blob/main/README.md
• 5: https://github.com/fla-org/flash-linear-attention/blob/78e4488f/fla/ops/gated_delta_rule/chunk.py
• 6: https://github.com/vllm-project/vime/blob/main/vime_plugins/models/qwen_gdn_backend.py
• 7: https://qwen.ai/blog?id=flashqla
• 8: https://docs.flashinfer.ai/generated/flashinfer.xqa.xqa.html
• 9: https://docs.flashinfer.ai/api/attention.html
• 10: https://github.com/flashinfer-ai/flashinfer/blob/2b150b39/benchmarks/samples/sample_testlist.txt

---

🏁 Script executed:

#!/bin/bash
set -eu
tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT
curl -fsSL https://raw.githubusercontent.com/QwenLM/FlashQLA/main/README.md -o "$tmpdir/README.md"
printf '%s\n' '--- README API references ---'
rg -n -C 6 'chunk_gated_delta_rule|output_final_state|initial_state|group|head|GVA|GQA' "$tmpdir/README.md" || true
printf '%s\n' '--- repository refs ---'
git ls-remote --heads --tags https://github.com/QwenLM/FlashQLA.git | head -40
printf '%s\n' '--- archive source paths ---'
curl -fsSL https://codeload.github.com/QwenLM/FlashQLA/tar.gz/refs/heads/main |
  tar -tzf - | rg -i '(^|/)(flash_qla|chunk|gated|setup|pyproject|readme)' | head -120

Repository: NVIDIA/cudnn-frontend

Length of output: 6442

---

🏁 Script executed:

#!/bin/bash
set -eu
tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT
curl -fsSL https://codeload.github.com/QwenLM/FlashQLA/tar.gz/refs/heads/main -o "$tmpdir/flashqla.tgz"
tar -xzf "$tmpdir/flashqla.tgz" -C "$tmpdir"
root="$(find "$tmpdir" -mindepth 1 -maxdepth 1 -type d -name 'FlashQLA-*' -print -quit)"
printf '%s\n' '--- exact function definitions and call examples ---'
rg -n -C 12 'def chunk_gated_delta_rule|chunk_gated_delta_rule\(|output_final_state|initial_state' "$root" --glob '*.py' --glob '*.md' || true

Repository: NVIDIA/cudnn-frontend

Length of output: 50378

---

Pin the FlashQLA dependency.

The Docker image clones main, so benchmark behavior is not reproducible. Pin FlashQLA to a tested commit or release.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@benchmark/linear_attention/benchmark_single_linear_attention.py` around lines
516 - 536, Pin the FlashQLA dependency to a tested commit or release in the
Docker setup that currently clones main. Update the dependency source or
checkout configuration used by the flash_qla benchmark path, while preserving
the existing flash_qla_linear_attention integration and ensuring builds
consistently use the pinned revision.

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

The title hardcodes BF16 and can mislabel float16 runs.

The benchmark accepts --data_type float16. The chart title always reports (BF16). The CSV has no data-type column, so add a CLI flag for the label or remove the precision from the title.

🔧 Proposed fix using a CLI flag

-def generate_charts(df: pd.DataFrame, output_dir: Path, gpu_name: str = "", cudnn_version: str = None, variant: str = "gdn", batch_sizes: list = None) -> list:
+def generate_charts(
+    df: pd.DataFrame, output_dir: Path, gpu_name: str = "", cudnn_version: str = None, variant: str = "gdn", batch_sizes: list = None, data_type: str = "BF16"
+) -> list:

         fig.suptitle(
-            f"{variant.upper()} Linear Attention (BF16) — Batch = {batch_size}, Heads = {heads}, d = {head_dim}{gpu_info}",
+            f"{variant.upper()} Linear Attention ({data_type}) — Batch = {batch_size}, Heads = {heads}, d = {head_dim}{gpu_info}",
             fontsize=TITLE_FONT_SIZE,
         )

Add the matching argument in main:

parser.add_argument("--data-type", default="BF16", help="Data type label for the chart title")

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@benchmark/linear_attention/plot_results.py` around lines 99 - 105, Update the
chart-title construction in the plotting flow to avoid always labeling results
as BF16. Add a main CLI argument for the data-type label, defaulting to BF16,
and pass that value into the title instead of the hardcoded precision text.