Clean up FROST LA files + bprop optimizations

README.md

@@ -66,6 +66,25 @@ Contributor credits for these OSS CuTe DSL kernels are listed in [Acknowledgemen
   <img src="https://github.com/NVIDIA/cudnn-frontend/blob/main/benchmark/attention_training/results/dsv3/gb300/dsv3_top_left.png" alt="DSv3 SDPA Benchmark on GB300 (only cuDNN)" width="600"/>
 </p>
 
+## New OSS Linear Attention Kernels
+
+#### GDN Forward and Bprop (GB300)
+<p align="center">
+  <img src="https://github.com/NVIDIA/cudnn-frontend/blob/main/benchmark/linear_attention/results/gdn/gb300/gdn_fixed_batch_flops.png" alt="GDN Linear Attention Benchmark on GB300" width="600"/>
+</p>
+
+#### KDA Forward and Bprop (GB300)
+
+<p align="center">
+  <img src="https://github.com/NVIDIA/cudnn-frontend/blob/main/benchmark/linear_attention/results/kda/gb300/kda_fixed_batch_flops.png" alt="KDA Linear Attention Benchmark on GB300" width="600"/>
+</p>
+
+#### GDN-2 Forward and Bprop (GB300)
+
+<p align="center">
+  <img src="https://github.com/NVIDIA/cudnn-frontend/blob/main/benchmark/linear_attention/results/gdn2/gb300/gdn2_fixed_batch_flops.png" alt="GDN-2 Linear Attention Benchmark on GB300" width="600"/>
+</p>
+
 ## Key Features
 
 *   **Unified Graph API:** Create reusable, persistent `cudnn_frontend::graph::Graph` objects to describe complex subgraphs.

benchmark/linear_attention/README.md

@@ -8,6 +8,8 @@ This directory contains benchmarking tools for linear attention operations (GDN/
 
 - `Dockerfile` - Docker container setup for running benchmarks
 - `benchmark_single_linear_attention.py` - Single linear attention benchmark script
+- `plot_results.py` - Renders the charts under `results/` from a sweep CSV
+- `results/<variant>/<gpu>/` - Dated sweep CSVs and the charts rendered from them
 
 ## Quick Start
 
@@ -92,4 +94,24 @@ The `kda` and `gdn2` variants fuse q/k L2 normalization in-kernel on every backe
 | `flash_qla` | FlashQLA (TileLang fused GDN kernels, `gdn` variant only) |
 | `flash_kda` | FlashKDA (`kda` forward variant only) |
 
-The cuDNN backend routes through the pygraph engines: FROST (Cutlass DSL) on SM100-class devices, the cuTile engines elsewhere.
+The cuDNN backend routes through the pygraph engines: FROST (Cutlass DSL) on SM100-class devices, the cuTile engines elsewhere.
+
+## Results
+
+Forward and backward TFLOPS at batch 4 over the sequence-length sweep, one
+chart per variant, rendered by `plot_results.py` from the dated CSVs under
+`results/<variant>/<gpu>/`. The `cudnn (state on)` bars dump the per-chunk
+state-checkpoint series in the forward pass and reuse it in the backward
+pass. Runs were captured on GB200 and GB300 (GB300 results shown below).
+
+### GB300 - GDN
+![GDN on GB300](results/gdn/gb300/gdn_fixed_batch_flops.png)
+- `batch=4; num_q_heads=64; num_kv_heads=64; head_dim=128; seqlen 2048-32768; bf16`
+
+### GB300 - KDA
+![KDA on GB300](results/kda/gb300/kda_fixed_batch_flops.png)
+- `batch=4; num_q_heads=64; num_kv_heads=64; head_dim=128; seqlen 2048-32768; bf16`
+
+### GB300 - GDN-2
+![GDN-2 on GB300](results/gdn2/gb300/gdn2_fixed_batch_flops.png)
+- `batch=4; num_q_heads=64; num_kv_heads=64; head_dim=128; seqlen 2048-32768; bf16`

benchmark/linear_attention/results/gdn/gb200/gdn_20260824.csv

@@ -0,0 +1,41 @@
+case_tag,backend,variant,batch_size,seqlen,num_q_heads,num_kv_heads,head_dim,fwd_ms,bwd_ms,fwd_tflops,bwd_tflops,max_diff,num_iters,fwd_bw,bwd_bw
+gdn_h64,cudnn,gdn,4,2048,64,64,128,0.166,0.596,467,389,0.000000,20,3.27,2.95
+gdn_h64,fla,gdn,4,2048,64,64,128,0.755,2.341,102,99,0.000000,20,0.72,0.75
+gdn_h64,flash_qla,gdn,4,2048,64,64,128,0.271,0.732,285,317,0.000000,20,1.99,2.40
+gdn_h64,cudnn,gdn,4,4096,64,64,128,0.288,1.130,537,410,0.000000,20,3.76,3.11
+gdn_h64,fla,gdn,4,4096,64,64,128,1.477,4.637,105,100,0.000000,20,0.73,0.76
+gdn_h64,flash_qla,gdn,4,4096,64,64,128,0.511,1.406,303,330,0.000000,20,2.12,2.50
+gdn_h64,cudnn,gdn,4,8192,64,64,128,0.532,2.186,581,424,0.000000,20,4.06,3.22
+gdn_h64,fla,gdn,4,8192,64,64,128,2.930,9.304,106,100,0.000000,20,0.74,0.76
+gdn_h64,flash_qla,gdn,4,8192,64,64,128,0.995,2.805,311,331,0.000000,20,2.18,2.51
+gdn_h64,cudnn,gdn,4,16384,64,64,128,1.014,4.295,610,432,0.000000,20,4.27,3.27
+gdn_h64,fla,gdn,4,16384,64,64,128,5.844,18.691,106,99,0.000000,20,0.74,0.75
+gdn_h64,flash_qla,gdn,4,16384,64,64,128,1.967,5.603,314,331,0.000000,20,2.20,2.51
+gdn_h64,cudnn,gdn,4,32768,64,64,128,1.966,8.499,629,437,0.000000,20,4.40,3.31
+gdn_h64,fla,gdn,4,32768,64,64,128,11.720,37.833,106,98,0.000000,20,0.74,0.74
+gdn_h64,flash_qla,gdn,4,32768,64,64,128,3.893,11.070,318,335,0.000000,20,2.22,2.54
+gdn_h64,cudnn,gdn,1,8192,64,64,128,0.169,0.557,457,417,0.000000,20,3.20,3.16
+gdn_h64,fla,gdn,1,8192,64,64,128,0.822,2.508,94,92,0.000000,20,0.66,0.70
+gdn_h64,flash_qla,gdn,1,8192,64,64,128,0.349,1.335,222,174,0.000000,20,1.55,1.32
+gdn_h64,cudnn,gdn,2,8192,64,64,128,0.273,1.024,566,453,0.000000,20,3.96,3.43
+gdn_h64,fla,gdn,2,8192,64,64,128,1.481,4.881,104,95,0.000000,20,0.73,0.72
+gdn_h64,flash_qla,gdn,2,8192,64,64,128,0.502,1.433,308,324,0.000000,20,2.16,2.45
+gdn_h64,cudnn,gdn,4,8192,64,64,128,0.533,2.179,580,426,0.000000,20,4.06,3.23
+gdn_h64,fla,gdn,4,8192,64,64,128,2.928,9.301,106,100,0.000000,20,0.74,0.76
+gdn_h64,flash_qla,gdn,4,8192,64,64,128,0.994,2.786,311,333,0.000000,20,2.18,2.52
+gdn_h64,cudnn,gdn,8,8192,64,64,128,1.010,4.493,612,413,0.000000,20,4.29,3.13
+gdn_h64,fla,gdn,8,8192,64,64,128,5.660,17.921,109,104,0.000000,20,0.76,0.78
+gdn_h64,flash_qla,gdn,8,8192,64,64,128,1.971,5.437,314,341,0.000000,20,2.20,2.59
+gdn_h64,cudnn,gdn,16,8192,64,64,128,1.755,8.112,705,457,0.000000,20,4.93,3.47
+gdn_h64,fla,gdn,16,8192,64,64,128,11.150,35.579,111,104,0.000000,20,0.78,0.79
+gdn_h64,flash_qla,gdn,16,8192,64,64,128,3.627,9.704,341,382,0.000000,20,2.39,2.90
+gdn_hon,cudnn_state_on,gdn,4,2048,64,64,128,0.222,0.473,349,490,0.000000,20,2.44,3.71
+gdn_hon,cudnn_state_on,gdn,4,4096,64,64,128,0.401,0.887,386,523,0.000000,20,2.70,3.96
+gdn_hon,cudnn_state_on,gdn,4,8192,64,64,128,0.767,1.697,403,547,0.000000,20,2.82,4.14
+gdn_hon,cudnn_state_on,gdn,4,16384,64,64,128,1.500,3.285,412,565,0.000000,20,2.89,4.28
+gdn_hon,cudnn_state_on,gdn,4,32768,64,64,128,2.970,6.472,416,573,0.000000,20,2.91,4.34
+gdn_hon,cudnn_state_on,gdn,1,8192,64,64,128,0.223,0.425,347,546,0.000000,20,2.43,4.14
+gdn_hon,cudnn_state_on,gdn,2,8192,64,64,128,0.384,0.771,403,602,0.000000,20,2.82,4.56
+gdn_hon,cudnn_state_on,gdn,4,8192,64,64,128,0.768,1.693,403,548,0.000000,20,2.82,4.15
+gdn_hon,cudnn_state_on,gdn,8,8192,64,64,128,1.527,3.475,405,534,0.000000,20,2.83,4.05
+gdn_hon,cudnn_state_on,gdn,16,8192,64,64,128,2.745,6.285,451,590,0.000000,20,3.15,4.47

benchmark/linear_attention/results/gdn/gb300/gdn_20260824.csv

@@ -0,0 +1,41 @@
+case_tag,backend,variant,batch_size,seqlen,num_q_heads,num_kv_heads,head_dim,fwd_ms,bwd_ms,fwd_tflops,bwd_tflops,max_diff,num_iters,fwd_bw,bwd_bw
+gdn_h64,cudnn,gdn,4,2048,64,64,128,0.157,0.582,493,399,0.000000,20,3.45,3.02
+gdn_h64,fla,gdn,4,2048,64,64,128,0.739,2.282,105,102,0.000000,20,0.73,0.77
+gdn_h64,flash_qla,gdn,4,2048,64,64,128,0.272,0.726,284,319,0.000000,20,1.99,2.42
+gdn_h64,cudnn,gdn,4,4096,64,64,128,0.275,1.120,563,414,0.000000,20,3.94,3.14
+gdn_h64,fla,gdn,4,4096,64,64,128,1.444,4.537,107,102,0.000000,20,0.75,0.77
+gdn_h64,flash_qla,gdn,4,4096,64,64,128,0.514,1.398,301,332,0.000000,20,2.11,2.51
+gdn_h64,cudnn,gdn,4,8192,64,64,128,0.516,2.158,600,430,0.000000,20,4.20,3.26
+gdn_h64,fla,gdn,4,8192,64,64,128,2.861,9.065,108,102,0.000000,20,0.76,0.78
+gdn_h64,flash_qla,gdn,4,8192,64,64,128,0.997,2.767,310,335,0.000000,20,2.17,2.54
+gdn_h64,cudnn,gdn,4,16384,64,64,128,0.992,4.224,624,439,0.000000,20,4.37,3.33
+gdn_h64,fla,gdn,4,16384,64,64,128,5.707,18.210,108,102,0.000000,20,0.76,0.77
+gdn_h64,flash_qla,gdn,4,16384,64,64,128,1.972,5.452,314,340,0.000000,20,2.20,2.58
+gdn_h64,cudnn,gdn,4,32768,64,64,128,1.939,8.394,638,442,0.000000,20,4.47,3.35
+gdn_h64,fla,gdn,4,32768,64,64,128,11.440,36.799,108,101,0.000000,20,0.76,0.76
+gdn_h64,flash_qla,gdn,4,32768,64,64,128,3.902,10.938,317,339,0.000000,20,2.22,2.57
+gdn_h64,cudnn,gdn,1,8192,64,64,128,0.160,0.552,482,420,0.000000,20,3.37,3.19
+gdn_h64,fla,gdn,1,8192,64,64,128,0.803,2.440,96,95,0.000000,20,0.67,0.72
+gdn_h64,flash_qla,gdn,1,8192,64,64,128,0.366,1.330,211,174,0.000000,20,1.48,1.32
+gdn_h64,cudnn,gdn,2,8192,64,64,128,0.264,1.014,586,457,0.000000,20,4.10,3.47
+gdn_h64,fla,gdn,2,8192,64,64,128,1.448,4.763,107,97,0.000000,20,0.75,0.74
+gdn_h64,flash_qla,gdn,2,8192,64,64,128,0.503,1.428,308,325,0.000000,20,2.15,2.46
+gdn_h64,cudnn,gdn,4,8192,64,64,128,0.514,2.147,602,432,0.000000,20,4.21,3.27
+gdn_h64,fla,gdn,4,8192,64,64,128,2.861,9.067,108,102,0.000000,20,0.76,0.78
+gdn_h64,flash_qla,gdn,4,8192,64,64,128,0.999,2.760,309,336,0.000000,20,2.17,2.55
+gdn_h64,cudnn,gdn,8,8192,64,64,128,0.988,4.425,626,419,0.000000,20,4.38,3.18
+gdn_h64,fla,gdn,8,8192,64,64,128,5.500,17.423,112,106,0.000000,20,0.79,0.81
+gdn_h64,flash_qla,gdn,8,8192,64,64,128,1.964,5.374,315,345,0.000000,20,2.20,2.62
+gdn_h64,cudnn,gdn,16,8192,64,64,128,1.729,7.956,715,466,0.000000,20,5.01,3.53
+gdn_h64,fla,gdn,16,8192,64,64,128,10.868,34.628,114,107,0.000000,20,0.80,0.81
+gdn_h64,flash_qla,gdn,16,8192,64,64,128,3.630,9.615,341,386,0.000000,20,2.38,2.92
+gdn_hon,cudnn_state_on,gdn,4,2048,64,64,128,0.218,0.468,355,495,0.000000,20,2.48,3.75
+gdn_hon,cudnn_state_on,gdn,4,4096,64,64,128,0.393,0.876,393,530,0.000000,20,2.75,4.01
+gdn_hon,cudnn_state_on,gdn,4,8192,64,64,128,0.753,1.680,411,552,0.000000,20,2.87,4.18
+gdn_hon,cudnn_state_on,gdn,4,16384,64,64,128,1.478,3.277,418,566,0.000000,20,2.93,4.29
+gdn_hon,cudnn_state_on,gdn,4,32768,64,64,128,2.924,6.467,423,574,0.000000,20,2.96,4.35
+gdn_hon,cudnn_state_on,gdn,1,8192,64,64,128,0.219,0.424,354,547,0.000000,20,2.48,4.15
+gdn_hon,cudnn_state_on,gdn,2,8192,64,64,128,0.375,0.768,413,604,0.000000,20,2.89,4.58
+gdn_hon,cudnn_state_on,gdn,4,8192,64,64,128,0.761,1.678,407,553,0.000000,20,2.85,4.19
+gdn_hon,cudnn_state_on,gdn,8,8192,64,64,128,1.489,3.446,415,538,0.000000,20,2.91,4.08
+gdn_hon,cudnn_state_on,gdn,16,8192,64,64,128,2.675,6.195,462,599,0.000000,20,3.24,4.54