diff --git a/README.md b/README.md index 150ab1f12..b611234d6 100644 --- a/README.md +++ b/README.md @@ -57,13 +57,13 @@ Contributor credits for these OSS CuTe DSL kernels are listed in [Acknowledgemen #### Llama 3.1 style Forward and Bprop with causal masking (GB300)

- Llama 3.1 SDPA Benchmark on GB300 (only cuDNN) + Llama 3.1 SDPA Benchmark on GB300 (only cuDNN)

#### Deepseek v3 style Forward and Bprop with causal masking (GB300)

- DSv3 SDPA Benchmark on GB300 (only cuDNN) + DSv3 SDPA Benchmark on GB300 (only cuDNN)

## Key Features diff --git a/benchmark/attention_inference/README.md b/benchmark/attention_inference/README.md index 0c75642e5..4d58be8e6 100644 --- a/benchmark/attention_inference/README.md +++ b/benchmark/attention_inference/README.md @@ -1,7 +1,7 @@ # Attention Inference Benchmark Benchmarks attention for **inference**, split into two phases (mirroring -forward/backward in `../sdpa_benchmark_training`): +forward/backward in `../attention_training`): - **context** — prefill, in two kinds, both reported in **TFLOPS**: - *full*: `s_q == s_kv`, contiguous Q/K/V, compute-bound; @@ -52,7 +52,7 @@ into the fp8 graph — those cases record as unsupported). MLA models run **absorbed** in generation (`kind="mla_absorbed"`: K reads the full record, V a leading slice of the *same* record, so KV bytes are counted once) and **unabsorbed** in prefill — which is dense training-style attention -and lives in `../sdpa_benchmark_training`. +and lives in `../attention_training`. ## Usage diff --git a/benchmark/attention_inference/configs/auto_regressive_dit.py b/benchmark/attention_inference/configs/auto_regressive_dit.py index 30fd87258..5a4ab8a4a 100644 --- a/benchmark/attention_inference/configs/auto_regressive_dit.py +++ b/benchmark/attention_inference/configs/auto_regressive_dit.py @@ -10,7 +10,7 @@ is bidirectional: context_causal=False). Full prefill covers the self- attention over the whole clip, and generation sweeps the standard MTP widths against the cached clip for completeness. The training suite carries its own -forward-only sweep of the same model (`sdpa_benchmark_training`). +forward-only sweep of the same model (`attention_training`). """ from ..config_types import InferenceBenchmarkConfig, ModelPreset diff --git a/benchmark/attention_inference/configs/qwen3vl_vit.py b/benchmark/attention_inference/configs/qwen3vl_vit.py new file mode 100644 index 000000000..90914f85b --- /dev/null +++ b/benchmark/attention_inference/configs/qwen3vl_vit.py @@ -0,0 +1,55 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +""" +Qwen3-VL Vision Encoder (ViT) self-attention — inference. + +The vision tower's self-attention is bidirectional over the patchified image +tokens and runs forward-only: pure context-phase (full prefill) work, so this +config sweeps ONLY the context phase — an encoder has no KV cache to decode +against, so there is no generation phase and no kv-cache dtype axis. + +Architecture (vision tower): 16 MHA heads, head_dim 72 zero-padded to 80 +(fp8 kernels require 16-byte-aligned head dims; production integrations run +the padded contract, and reported TFLOPS count d=80). Sequence lengths are +per-image patch-grid token counts from a production inference trace +(94x94 .. 250x250 grids), spanning that trace's per-forward FLOPs +distribution from the 10th to the 99th percentile. The tower is not +head-shardable in deployments (whole-model per device), so no TP sweep. + +Usage: + python -m benchmark.attention_inference.runner --config qwen3vl_vit +""" + +from ..config_types import InferenceBenchmarkConfig, ModelPreset + +QWEN3VL_VIT = ModelPreset( + name="qwen3vl_vit", + num_q_heads=16, + num_kv_heads=16, + head_dim=80, +) + +CONFIG = InferenceBenchmarkConfig( + name="qwen3vl_vit", + models=[QWEN3VL_VIT], + context_seqlens=[ + 8836, # 94x94 patch grid + 15376, # 124x124 (most frequent single-image forward) + 24336, # 156x156 + 35344, # 188x188 (FLOPs-median forward) + 47376, # non-square grid (e.g. 168x282) + 62500, # 250x250 + ], + context_chunked_shapes=[], # encoder: no chunked prefill against a cache + generation_shapes=[], # encoder: no decode phase + context_causal=False, # bidirectional ViT self-attention + # bf16 only: this suite expresses fp8 solely as the generation-phase + # kv-cache axis (the fp8 attention graph), and an encoder has no + # generation phase — the context path has no fp8 route today. The training + # suite's fp8 forward numbers for this model are dropped by the move; + # restoring them here needs fp8-context support in + # benchmark_single_attention first. + data_types=["bfloat16"], + backends=["cudnn", "cudnn_oss"], +) diff --git a/benchmark/attention_inference/results/auto_regressive_dit/b300/auto_regressive_dit_20260817_132701.csv b/benchmark/attention_inference/results/auto_regressive_dit/b300/auto_regressive_dit_20260817_132701.csv deleted file mode 100644 index a1c13d7ec..000000000 --- a/benchmark/attention_inference/results/auto_regressive_dit/b300/auto_regressive_dit_20260817_132701.csv +++ /dev/null @@ -1,81 +0,0 @@ -config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version -auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,8192,8192,9,9,128,128,gqa,,64,0.189056,1635.693,399.339,4.9917,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,62208,62208,9,9,128,128,gqa,,64,11.083296,1608.926,51.727,0.6466,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,985,62208,9,9,128,128,gqa,,64,0.212224,1330.456,1372.104,17.1513,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,2048,62208,9,9,128,128,gqa,,64,0.370048,1586.465,800.144,10.0018,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,4096,62208,9,9,128,128,gqa,,64,0.652448,1799.587,468.281,5.8535,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,8192,62208,9,9,128,128,gqa,,64,1.271648,1846.638,255.105,3.1888,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,1,62208,9,9,128,128,gqa,,64,0.060768,4.717,4717.27,58.9659,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,2,62208,9,9,128,128,gqa,,64,1.058752,0.541,270.756,3.3845,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,3,62208,9,9,128,128,gqa,,64,1.058752,0.812,270.761,3.3845,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,4,62208,9,9,128,128,gqa,,64,1.057152,1.085,271.175,3.3897,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,9,9,128,128,gqa,,64,0.296832,1041.794,254.344,3.1793,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,62208,62208,9,9,128,128,gqa,,64,12.159232,1466.556,47.15,0.5894,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,985,62208,9,9,128,128,gqa,,64,0.997024,283.197,292.063,3.6508,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,2048,62208,9,9,128,128,gqa,,64,0.99984,587.162,296.139,3.7017,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,4096,62208,9,9,128,128,gqa,,64,1.056032,1111.838,289.318,3.6165,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,8192,62208,9,9,128,128,gqa,,64,1.71808,1366.801,188.817,2.3602,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,1,62208,9,9,128,128,gqa,,64,0.976992,0.293,293.41,3.6676,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,2,62208,9,9,128,128,gqa,,64,0.987456,0.581,290.305,3.6288,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,3,62208,9,9,128,128,gqa,,64,0.989952,0.869,289.578,3.6197,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,4,62208,9,9,128,128,gqa,,64,0.988576,1.16,289.986,3.6248,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,4,4,128,128,gqa,,64,0.10736,1280.169,312.541,3.9068,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp2,context,cudnn,bfloat16,bfloat16,1,62208,62208,4,4,128,128,gqa,,64,4.91264,1613.272,51.867,0.6483,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp2,context,cudnn,bfloat16,bfloat16,1,985,62208,4,4,128,128,gqa,,64,0.123328,1017.538,1049.391,13.1174,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp2,context,cudnn,bfloat16,bfloat16,1,2048,62208,4,4,128,128,gqa,,64,0.200064,1304.179,657.771,8.2221,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp2,context,cudnn,bfloat16,bfloat16,1,4096,62208,4,4,128,128,gqa,,64,0.353088,1477.928,384.58,4.8073,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp2,context,cudnn,bfloat16,bfloat16,1,8192,62208,4,4,128,128,gqa,,64,0.590944,1766.118,243.981,3.0498,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp2,generation,cudnn,bfloat16,bfloat16,1,1,62208,4,4,128,128,gqa,,64,0.051872,2.456,2456.123,30.7015,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp2,generation,cudnn,bfloat16,bfloat16,1,2,62208,4,4,128,128,gqa,,64,1.056416,0.241,120.602,1.5075,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp2,generation,cudnn,bfloat16,bfloat16,1,3,62208,4,4,128,128,gqa,,64,1.05648,0.362,120.597,1.5075,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp2,generation,cudnn,bfloat16,bfloat16,1,4,62208,4,4,128,128,gqa,,64,1.056832,0.482,120.559,1.507,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,4,4,128,128,gqa,,64,0.208864,658.031,160.652,2.0082,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp2,context,cudnn_oss,bfloat16,bfloat16,1,62208,62208,4,4,128,128,gqa,,64,5.13792,1542.535,49.593,0.6199,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp2,context,cudnn_oss,bfloat16,bfloat16,1,985,62208,4,4,128,128,gqa,,64,0.820224,152.996,157.785,1.9723,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,62208,4,4,128,128,gqa,,64,0.821696,317.537,160.152,2.0019,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp2,context,cudnn_oss,bfloat16,bfloat16,1,4096,62208,4,4,128,128,gqa,,64,0.82624,631.582,164.348,2.0543,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,62208,4,4,128,128,gqa,,64,0.850944,1226.493,169.434,2.1179,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,62208,4,4,128,128,gqa,,64,0.802848,0.159,158.69,1.9836,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,62208,4,4,128,128,gqa,,64,0.813984,0.313,156.522,1.9565,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,62208,4,4,128,128,gqa,,64,0.81328,0.47,156.66,1.9582,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,62208,4,4,128,128,gqa,,64,0.81488,0.625,156.355,1.9544,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,2,2,128,128,gqa,,64,0.105184,653.326,159.503,1.9938,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp4,context,cudnn,bfloat16,bfloat16,1,62208,62208,2,2,128,128,gqa,,64,2.351936,1684.872,54.169,0.6771,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp4,context,cudnn,bfloat16,bfloat16,1,985,62208,2,2,128,128,gqa,,64,0.083136,754.733,778.359,9.7295,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp4,context,cudnn,bfloat16,bfloat16,1,2048,62208,2,2,128,128,gqa,,64,0.12224,1067.242,538.27,6.7284,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp4,context,cudnn,bfloat16,bfloat16,1,4096,62208,2,2,128,128,gqa,,64,0.198624,1313.634,341.828,4.2729,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp4,context,cudnn,bfloat16,bfloat16,1,8192,62208,2,2,128,128,gqa,,64,0.350528,1488.721,205.66,2.5708,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp4,generation,cudnn,bfloat16,bfloat16,1,1,62208,2,2,128,128,gqa,,64,0.04288,1.486,1485.588,18.5699,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp4,generation,cudnn,bfloat16,bfloat16,1,2,62208,2,2,128,128,gqa,,64,0.91232,0.14,69.825,0.8728,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp4,generation,cudnn,bfloat16,bfloat16,1,3,62208,2,2,128,128,gqa,,64,0.911488,0.21,69.89,0.8736,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp4,generation,cudnn,bfloat16,bfloat16,1,4,62208,2,2,128,128,gqa,,64,0.912192,0.279,69.837,0.873,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,2,2,128,128,gqa,,64,0.206208,333.253,81.361,1.017,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp4,context,cudnn_oss,bfloat16,bfloat16,1,62208,62208,2,2,128,128,gqa,,64,2.734112,1449.36,46.597,0.5825,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp4,context,cudnn_oss,bfloat16,bfloat16,1,985,62208,2,2,128,128,gqa,,64,0.749536,83.712,86.333,1.0792,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,62208,2,2,128,128,gqa,,64,0.748224,174.359,87.939,1.0992,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp4,context,cudnn_oss,bfloat16,bfloat16,1,4096,62208,2,2,128,128,gqa,,64,0.752352,346.805,90.244,1.1281,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,62208,2,2,128,128,gqa,,64,0.754048,692.049,95.603,1.195,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,62208,2,2,128,128,gqa,,64,0.734112,0.087,86.774,1.0847,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,62208,2,2,128,128,gqa,,64,0.742944,0.171,85.744,1.0718,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,62208,2,2,128,128,gqa,,64,0.744416,0.257,85.576,1.0697,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,62208,2,2,128,128,gqa,,64,0.741984,0.343,85.858,1.0732,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,1,1,128,128,gqa,,64,0.075584,454.59,110.984,1.3873,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp8,context,cudnn,bfloat16,bfloat16,1,62208,62208,1,1,128,128,gqa,,64,1.167264,1697.436,54.573,0.6822,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp8,context,cudnn,bfloat16,bfloat16,1,985,62208,1,1,128,128,gqa,,64,0.065312,480.352,495.389,6.1924,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp8,context,cudnn,bfloat16,bfloat16,1,2048,62208,1,1,128,128,gqa,,64,0.08352,781.008,393.907,4.9238,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp8,context,cudnn,bfloat16,bfloat16,1,4096,62208,1,1,128,128,gqa,,64,0.121504,1073.706,279.395,3.4924,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp8,context,cudnn,bfloat16,bfloat16,1,8192,62208,1,1,128,128,gqa,,64,0.19712,1323.657,182.857,2.2857,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp8,generation,cudnn,bfloat16,bfloat16,1,1,62208,1,1,128,128,gqa,,64,0.04208,0.757,756.916,9.4614,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp8,generation,cudnn,bfloat16,bfloat16,1,2,62208,1,1,128,128,gqa,,64,0.90576,0.07,35.166,0.4396,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp8,generation,cudnn,bfloat16,bfloat16,1,3,62208,1,1,128,128,gqa,,64,0.905888,0.105,35.161,0.4395,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp8,generation,cudnn,bfloat16,bfloat16,1,4,62208,1,1,128,128,gqa,,64,0.905696,0.141,35.169,0.4396,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -auto_regressive_dit,ar_dit-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,1,1,128,128,gqa,,64,0.155808,220.526,53.839,0.673,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp8,context,cudnn_oss,bfloat16,bfloat16,1,62208,62208,1,1,128,128,gqa,,64,1.305312,1517.917,48.801,0.61,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp8,context,cudnn_oss,bfloat16,bfloat16,1,985,62208,1,1,128,128,gqa,,64,0.66688,47.044,48.517,0.6065,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,62208,1,1,128,128,gqa,,64,0.667392,97.738,49.295,0.6162,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp8,context,cudnn_oss,bfloat16,bfloat16,1,4096,62208,1,1,128,128,gqa,,64,0.66736,195.486,50.869,0.6359,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,62208,1,1,128,128,gqa,,64,0.668928,390.056,53.884,0.6736,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,62208,1,1,128,128,gqa,,64,0.667456,0.048,47.72,0.5965,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,62208,1,1,128,128,gqa,,64,0.660192,0.096,48.246,0.6031,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,62208,1,1,128,128,gqa,,64,0.661376,0.144,48.16,0.602,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -auto_regressive_dit,ar_dit-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,62208,1,1,128,128,gqa,,64,0.663552,0.192,48.003,0.6,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], diff --git a/benchmark/attention_inference/results/auto_regressive_dit/b300/auto_regressive_dit_context.png b/benchmark/attention_inference/results/auto_regressive_dit/b300/auto_regressive_dit_context.png deleted file mode 100644 index 131b7154a..000000000 Binary files a/benchmark/attention_inference/results/auto_regressive_dit/b300/auto_regressive_dit_context.png and /dev/null differ diff --git a/benchmark/attention_inference/results/auto_regressive_dit/b300/auto_regressive_dit_generation.png b/benchmark/attention_inference/results/auto_regressive_dit/b300/auto_regressive_dit_generation.png deleted file mode 100644 index 03be998d0..000000000 Binary files a/benchmark/attention_inference/results/auto_regressive_dit/b300/auto_regressive_dit_generation.png and /dev/null differ diff --git a/benchmark/attention_inference/results/auto_regressive_dit/gb200/auto_regressive_dit_20260818_125952.csv b/benchmark/attention_inference/results/auto_regressive_dit/gb200/auto_regressive_dit_20260818_125952.csv new file mode 100644 index 000000000..ddd0a0873 --- /dev/null +++ b/benchmark/attention_inference/results/auto_regressive_dit/gb200/auto_regressive_dit_20260818_125952.csv @@ -0,0 +1,21 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,8192,8192,9,9,128,128,gqa,,64,0.232032,1332.737,325.375,4.0672,20,True,,NVIDIA GB200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,62208,62208,9,9,128,128,gqa,,64,12.190784,1462.761,47.028,0.5879,20,True,,NVIDIA GB200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,985,62208,9,9,128,128,gqa,,64,0.266464,1059.635,1092.806,13.6601,20,True,,NVIDIA GB200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,2048,62208,9,9,128,128,gqa,,64,0.450592,1302.882,657.117,8.214,20,True,,NVIDIA GB200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,4096,62208,9,9,128,128,gqa,,64,0.774688,1515.625,394.39,4.9299,20,True,,NVIDIA GB200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,8192,62208,9,9,128,128,gqa,,64,1.43648,1634.741,225.832,2.8229,20,True,,NVIDIA GB200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,1,62208,9,9,128,128,gqa,,64,0.069632,4.117,4116.772,51.4596,20,True,,NVIDIA GB200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,2,62208,9,9,128,128,gqa,,64,1.059616,0.541,270.535,3.3817,20,True,,NVIDIA GB200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,3,62208,9,9,128,128,gqa,,64,1.060992,0.811,270.189,3.3774,20,True,,NVIDIA GB200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,4,62208,9,9,128,128,gqa,,64,1.059136,1.083,270.667,3.3833,20,True,,NVIDIA GB200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,9,9,128,128,gqa,,64,0.351456,879.876,214.813,2.6852,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,62208,62208,9,9,128,128,gqa,,64,13.334592,1337.289,42.994,0.5374,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,985,62208,9,9,128,128,gqa,,64,1.122272,251.592,259.468,3.2433,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,2048,62208,9,9,128,128,gqa,,64,1.121664,523.391,263.975,3.2997,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,4096,62208,9,9,128,128,gqa,,64,1.15248,1018.791,265.106,3.3138,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,8192,62208,9,9,128,128,gqa,,64,1.950816,1203.739,166.291,2.0786,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,1,62208,9,9,128,128,gqa,,64,1.095648,0.262,261.634,3.2704,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,2,62208,9,9,128,128,gqa,,64,1.12048,0.512,255.84,3.198,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,3,62208,9,9,128,128,gqa,,64,1.121024,0.767,255.72,3.1965,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,4,62208,9,9,128,128,gqa,,64,1.118944,1.025,256.199,3.2025,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], diff --git a/benchmark/attention_inference/results/auto_regressive_dit/gb200/auto_regressive_dit_context.png b/benchmark/attention_inference/results/auto_regressive_dit/gb200/auto_regressive_dit_context.png new file mode 100644 index 000000000..2e1298d1e Binary files /dev/null and b/benchmark/attention_inference/results/auto_regressive_dit/gb200/auto_regressive_dit_context.png differ diff --git a/benchmark/attention_inference/results/auto_regressive_dit/gb200/auto_regressive_dit_generation.png b/benchmark/attention_inference/results/auto_regressive_dit/gb200/auto_regressive_dit_generation.png new file mode 100644 index 000000000..349024f31 Binary files /dev/null and b/benchmark/attention_inference/results/auto_regressive_dit/gb200/auto_regressive_dit_generation.png differ diff --git a/benchmark/attention_inference/results/auto_regressive_dit/gb300/auto_regressive_dit_20260818_130601.csv b/benchmark/attention_inference/results/auto_regressive_dit/gb300/auto_regressive_dit_20260818_130601.csv new file mode 100644 index 000000000..ba1c167b8 --- /dev/null +++ b/benchmark/attention_inference/results/auto_regressive_dit/gb300/auto_regressive_dit_20260818_130601.csv @@ -0,0 +1,21 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,8192,8192,9,9,128,128,gqa,,64,0.179424,1723.502,420.777,5.2597,20,True,,NVIDIA GB300,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,62208,62208,9,9,128,128,gqa,,64,9.415008,1894.019,60.893,0.7612,20,True,,NVIDIA GB300,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,985,62208,9,9,128,128,gqa,,64,0.205088,1376.749,1419.846,17.7481,20,True,,NVIDIA GB300,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,2048,62208,9,9,128,128,gqa,,64,0.362016,1621.664,817.897,10.2237,20,True,,NVIDIA GB300,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,4096,62208,9,9,128,128,gqa,,64,0.602688,1948.167,506.944,6.3368,20,True,,NVIDIA GB300,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,8192,62208,9,9,128,128,gqa,,64,1.155936,2031.491,280.641,3.508,20,True,,NVIDIA GB300,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,1,62208,9,9,128,128,gqa,,64,0.066848,4.288,4288.222,53.6028,20,True,,NVIDIA GB300,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,2,62208,9,9,128,128,gqa,,64,1.055136,0.543,271.684,3.3961,20,True,,NVIDIA GB300,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,3,62208,9,9,128,128,gqa,,64,1.056224,0.814,271.409,3.3926,20,True,,NVIDIA GB300,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,4,62208,9,9,128,128,gqa,,64,1.055936,1.086,271.487,3.3936,20,True,,NVIDIA GB300,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,9,9,128,128,gqa,,64,0.307744,1004.854,245.326,3.0666,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,62208,62208,9,9,128,128,gqa,,64,10.21984,1744.861,56.098,0.7012,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,985,62208,9,9,128,128,gqa,,64,0.965952,292.307,301.457,3.7682,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,2048,62208,9,9,128,128,gqa,,64,0.969824,605.335,305.305,3.8163,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,4096,62208,9,9,128,128,gqa,,64,1.006496,1166.559,303.557,3.7945,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,8192,62208,9,9,128,128,gqa,,64,1.616224,1452.938,200.717,2.509,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,1,62208,9,9,128,128,gqa,,64,0.943008,0.304,303.984,3.7998,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,2,62208,9,9,128,128,gqa,,64,0.98736,0.581,290.333,3.6292,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,3,62208,9,9,128,128,gqa,,64,0.989728,0.869,289.644,3.6205,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,4,62208,9,9,128,128,gqa,,64,0.989248,1.159,289.789,3.6224,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], diff --git a/benchmark/attention_inference/results/auto_regressive_dit/gb300/auto_regressive_dit_context.png b/benchmark/attention_inference/results/auto_regressive_dit/gb300/auto_regressive_dit_context.png new file mode 100644 index 000000000..08f20b040 Binary files /dev/null and b/benchmark/attention_inference/results/auto_regressive_dit/gb300/auto_regressive_dit_context.png differ diff --git a/benchmark/attention_inference/results/auto_regressive_dit/gb300/auto_regressive_dit_generation.png b/benchmark/attention_inference/results/auto_regressive_dit/gb300/auto_regressive_dit_generation.png new file mode 100644 index 000000000..2812d75ff Binary files /dev/null and b/benchmark/attention_inference/results/auto_regressive_dit/gb300/auto_regressive_dit_generation.png differ diff --git a/benchmark/attention_inference/results/auto_regressive_dit/h200/auto_regressive_dit_20260818_133441.csv b/benchmark/attention_inference/results/auto_regressive_dit/h200/auto_regressive_dit_20260818_133441.csv new file mode 100644 index 000000000..f99aad5d3 --- /dev/null +++ b/benchmark/attention_inference/results/auto_regressive_dit/h200/auto_regressive_dit_20260818_133441.csv @@ -0,0 +1,11 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,8192,8192,9,9,128,128,gqa,,64,0.527392,586.353,143.152,2.9823,20,True,,NVIDIA H200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,62208,62208,9,9,128,128,gqa,,64,28.938208,616.216,19.811,0.4127,20,True,,NVIDIA H200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,985,62208,9,9,128,128,gqa,,64,0.743296,379.868,391.76,8.1617,20,True,,NVIDIA H200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,2048,62208,9,9,128,128,gqa,,64,1.487552,394.654,199.046,4.1468,20,True,,NVIDIA H200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,4096,62208,9,9,128,128,gqa,,64,2.224064,527.924,137.374,2.862,20,True,,NVIDIA H200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,8192,62208,9,9,128,128,gqa,,64,3.962112,592.682,81.876,1.7058,20,True,,NVIDIA H200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,1,62208,9,9,128,128,gqa,,64,0.099072,2.893,2893.442,60.28,20,True,,NVIDIA H200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,2,62208,9,9,128,128,gqa,,64,0.74272,0.772,385.965,8.0409,20,True,,NVIDIA H200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,3,62208,9,9,128,128,gqa,,64,0.74416,1.156,385.224,8.0255,20,True,,NVIDIA H200,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,4,62208,9,9,128,128,gqa,,64,0.741632,1.546,386.543,8.053,20,True,,NVIDIA H200,cudnn 9.26.0, diff --git a/benchmark/attention_inference/results/auto_regressive_dit/h200/auto_regressive_dit_context.png b/benchmark/attention_inference/results/auto_regressive_dit/h200/auto_regressive_dit_context.png new file mode 100644 index 000000000..2037774e0 Binary files /dev/null and b/benchmark/attention_inference/results/auto_regressive_dit/h200/auto_regressive_dit_context.png differ diff --git a/benchmark/attention_inference/results/auto_regressive_dit/h200/auto_regressive_dit_generation.png b/benchmark/attention_inference/results/auto_regressive_dit/h200/auto_regressive_dit_generation.png new file mode 100644 index 000000000..a00bbe167 Binary files /dev/null and b/benchmark/attention_inference/results/auto_regressive_dit/h200/auto_regressive_dit_generation.png differ diff --git a/benchmark/attention_inference/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_20260818_134936.csv b/benchmark/attention_inference/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_20260818_134936.csv new file mode 100644 index 000000000..bd2ea60a3 --- /dev/null +++ b/benchmark/attention_inference/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_20260818_134936.csv @@ -0,0 +1,21 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,8192,8192,9,9,128,128,gqa,,64,1.142784,270.6,66.065,3.6866,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,62208,62208,9,9,128,128,gqa,,64,48.970783,364.14,11.707,0.6533,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,985,62208,9,9,128,128,gqa,,64,1.983968,142.318,146.773,8.1905,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,2048,62208,9,9,128,128,gqa,,64,2.014176,291.468,147.004,8.2033,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,4096,62208,9,9,128,128,gqa,,64,4.009664,292.827,76.198,4.2521,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn,bfloat16,bfloat16,1,8192,62208,9,9,128,128,gqa,,64,7.96848,294.695,40.711,2.2718,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,1,62208,9,9,128,128,gqa,,64,0.204864,1.399,1399.265,78.084,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,2,62208,9,9,128,128,gqa,,64,1.143264,0.501,250.741,13.9923,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,3,62208,9,9,128,128,gqa,,64,1.143744,0.752,250.64,13.9866,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +auto_regressive_dit,ar_dit,generation,cudnn,bfloat16,bfloat16,1,4,62208,9,9,128,128,gqa,,64,1.174624,0.976,244.055,13.6191,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,9,9,128,128,gqa,,64,1.13792,271.757,66.347,3.7024,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,62208,62208,9,9,128,128,gqa,,64,45.043297,395.89,12.728,0.7103,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,985,62208,9,9,128,128,gqa,,64,1.416576,199.322,205.561,11.4711,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,2048,62208,9,9,128,128,gqa,,64,2.450688,239.552,120.82,6.7422,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,4096,62208,9,9,128,128,gqa,,64,3.974304,295.432,76.876,4.29,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +auto_regressive_dit,ar_dit,context,cudnn_oss,bfloat16,bfloat16,1,8192,62208,9,9,128,128,gqa,,64,7.507008,312.811,43.213,2.4115,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,1,62208,9,9,128,128,gqa,,64,1.386624,0.207,206.732,11.5364,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,2,62208,9,9,128,128,gqa,,64,1.432032,0.4,200.18,11.1707,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,3,62208,9,9,128,128,gqa,,64,1.398112,0.615,205.04,11.4419,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +auto_regressive_dit,ar_dit,generation,cudnn_oss,bfloat16,bfloat16,1,4,62208,9,9,128,128,gqa,,64,1.438976,0.797,199.22,11.1172,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], diff --git a/benchmark/attention_inference/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_context.png b/benchmark/attention_inference/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_context.png new file mode 100644 index 000000000..673394e33 Binary files /dev/null and b/benchmark/attention_inference/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_context.png differ diff --git a/benchmark/attention_inference/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_generation.png b/benchmark/attention_inference/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_generation.png new file mode 100644 index 000000000..dd1972ea0 Binary files /dev/null and b/benchmark/attention_inference/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_generation.png differ diff --git a/benchmark/attention_inference/results/deepseek_v4/b300/deepseek_v4_20260817_140126.csv b/benchmark/attention_inference/results/deepseek_v4/b300/deepseek_v4_20260817_140126.csv deleted file mode 100644 index e6d3cbcdf..000000000 --- a/benchmark/attention_inference/results/deepseek_v4/b300/deepseek_v4_20260817_140126.csv +++ /dev/null @@ -1,1761 +0,0 @@ -config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version -deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,0.549056,500.637,492.723,6.159,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,8.900352,494.143,121.583,1.5198,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,8.831904,497.973,15.197,0.19,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,17.95232,489.97,11.215,0.1402,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,16.63936,528.632,12.099,0.1512,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,34.043488,516.756,7.885,0.0986,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,0.39136,43.898,343.287,4.2911,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,37.441826,58.732,459.29,5.7411,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,10.663296,3.222,12.611,0.1576,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,1315.338745,3.344,13.087,0.1636,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,10.642976,4.843,12.648,0.1581,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,1313.978638,5.021,13.113,0.1639,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,10.637408,6.46,12.667,0.1583,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,1313.105225,6.699,13.134,0.1642,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,0.641792,428.297,421.527,5.2691,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,4.296672,1023.594,251.853,3.1482,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,2.856032,1539.915,46.994,0.5874,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,6.15424,1429.274,32.713,0.4089,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,6.272352,1402.36,32.097,0.4012,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,12.426624,1415.685,21.602,0.27,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,2.720864,6.314,49.377,0.6172,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,356.01236,6.177,48.304,0.6038,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,2.743232,12.525,49.022,0.6128,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,359.084839,12.248,47.937,0.5992,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,2.747136,18.761,49.0,0.6125,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,359.111969,18.371,47.98,0.5998,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,2.74272,25.055,49.127,0.6141,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,359.237946,24.485,48.01,0.6001,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,0.29168,471.198,467.344,5.8418,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,4.3008,511.306,126.781,1.5848,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,3.957376,555.677,25.437,0.318,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,8.830752,498.038,18.999,0.2375,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,8.813184,499.03,15.229,0.1904,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,17.876545,492.047,11.262,0.1408,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,0.22464,38.239,597.771,7.4721,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,21.766369,50.514,789.67,9.8709,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,10.749952,1.598,12.498,0.1562,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,643.538879,3.417,26.722,0.334,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,10.732992,2.401,12.523,0.1565,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,642.983887,5.13,26.758,0.3345,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,10.727776,3.203,12.536,0.1567,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,642.24408,6.848,26.802,0.335,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,0.37024,371.216,368.18,4.6022,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,2.199328,999.861,247.921,3.099,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,1.456672,1509.621,69.105,0.8638,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,2.797024,1572.402,59.982,0.7498,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,3.023616,1454.565,44.39,0.5549,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,6.298976,1396.432,31.962,0.3995,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,1.400416,6.134,95.888,1.1986,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,177.681793,6.188,96.736,1.2092,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,1.42544,12.052,94.251,1.1781,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,178.158081,12.343,96.525,1.2066,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,1.423936,18.098,94.396,1.18,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,178.137955,18.517,96.583,1.2073,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,1.421824,24.166,94.583,1.1823,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,178.495193,24.64,96.436,1.2055,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.161472,425.581,428.595,5.3574,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,1.881824,584.28,147.104,1.8388,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,1.89904,578.983,44.173,0.5522,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,3.800928,578.549,39.726,0.4966,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,4.23584,519.147,23.765,0.2971,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,8.725568,504.041,19.228,0.2403,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,0.136672,31.425,982.282,12.2785,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,11.614816,47.332,1479.495,18.4937,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,10.679328,0.804,12.574,0.1572,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,317.801819,3.46,54.085,0.6761,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,10.652576,1.21,12.609,0.1576,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,318.435852,5.179,53.99,0.6749,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,10.64736,1.614,12.618,0.1577,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,317.758636,6.92,54.119,0.6765,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.224608,305.953,308.119,3.8515,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,1.179776,931.966,234.641,2.933,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,0.778752,1411.889,107.719,1.3465,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,1.397728,1573.284,108.029,1.3504,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,1.417152,1551.72,71.032,0.8879,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,2.950528,1490.596,56.862,0.7108,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,1.398656,3.071,95.985,1.1998,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,89.72451,6.127,191.52,2.394,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,1.418656,6.055,94.655,1.1832,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,89.774628,12.247,191.46,2.3933,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,1.422272,9.059,94.438,1.1805,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,89.791199,18.368,191.471,2.3934,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,1.420576,12.094,94.573,1.1822,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,89.856163,24.473,191.38,2.3922,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,512,512,mla_absorbed,,64,0.095328,360.437,373.989,4.6749,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,512,512,mla_absorbed,,64,0.944512,582.053,150.984,1.8873,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,512,512,mla_absorbed,,64,1.77776,309.241,42.468,0.5308,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,512,512,mla_absorbed,,64,3.546784,310.002,40.207,0.5026,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,512,512,mla_absorbed,,64,1.866112,589.199,44.952,0.5619,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,512,512,mla_absorbed,,64,3.710496,592.649,40.694,0.5087,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,512,512,mla_absorbed,,64,0.13632,15.753,984.699,12.3087,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,512,512,mla_absorbed,,64,11.6168,23.662,1479.062,18.4883,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,512,512,mla_absorbed,,64,10.658784,0.403,12.595,0.1574,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,512,512,mla_absorbed,,64,160.321503,3.429,107.185,1.3398,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,512,512,mla_absorbed,,64,10.635008,0.606,12.625,0.1578,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,512,512,mla_absorbed,,64,160.087067,5.151,107.355,1.3419,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,512,512,mla_absorbed,,64,10.62352,0.809,12.64,0.158,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,512,512,mla_absorbed,,64,160.008102,6.872,107.421,1.3428,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,512,512,mla_absorbed,,64,0.163648,209.961,217.855,2.7232,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,512,512,mla_absorbed,,64,0.666912,824.33,213.831,2.6729,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,512,512,mla_absorbed,,64,0.76944,714.488,98.12,1.2265,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,512,512,mla_absorbed,,64,1.426272,770.899,99.985,1.2498,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,512,512,mla_absorbed,,64,0.777792,1413.632,107.852,1.3481,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,512,512,mla_absorbed,,64,1.439968,1527.133,104.86,1.3107,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,8,1,512,512,mla_absorbed,,64,1.399584,1.534,95.91,1.1989,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,8,1,512,512,mla_absorbed,,64,45.753922,6.008,375.53,4.6941,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,512,512,mla_absorbed,,64,1.419424,3.026,94.581,1.1823,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,512,512,mla_absorbed,,64,45.820801,11.998,375.028,4.6878,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,512,512,mla_absorbed,,64,1.418624,4.541,94.646,1.1831,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,512,512,mla_absorbed,,64,45.678143,18.053,376.245,4.7031,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,512,512,mla_absorbed,,64,1.42096,6.045,94.502,1.1813,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,512,512,mla_absorbed,,64,45.87085,23.97,374.71,4.6839,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,8,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,2048,2048,128,1,512,512,mla_absorbed,,64,1.071456,513.092,503.024,6.2878,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,8192,8192,128,1,512,512,mla_absorbed,,64,17.802912,494.082,121.097,1.5137,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,512,65536,128,1,512,512,mla_absorbed,,64,16.733919,525.645,12.031,0.1504,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,512,131072,128,1,512,512,mla_absorbed,,64,34.20771,514.275,7.847,0.0981,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,1024,65536,128,1,512,512,mla_absorbed,,64,33.949951,518.18,9.883,0.1235,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,1024,131072,128,1,512,512,mla_absorbed,,64,68.48288,513.769,5.88,0.0735,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,1,131072,128,1,512,512,mla_absorbed,,64,0.697728,49.245,192.74,2.4092,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,1,131072,128,1,512,512,mla_absorbed,,64,72.595711,60.583,237.114,2.9639,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,2,131072,128,1,512,512,mla_absorbed,,64,23.722656,2.897,5.68,0.071,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,2,131072,128,1,512,512,mla_absorbed,,64,2752.263672,3.196,6.266,0.0783,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,3,131072,128,1,512,512,mla_absorbed,,64,23.694944,4.35,5.698,0.0712,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,3,131072,128,1,512,512,mla_absorbed,,64,2750.384033,4.797,6.283,0.0785,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,4,131072,128,1,512,512,mla_absorbed,,64,23.688065,5.802,5.71,0.0714,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,4,131072,128,1,512,512,mla_absorbed,,64,2748.631836,6.4,6.299,0.0787,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,128,1,512,512,mla_absorbed,,64,1.186432,463.369,454.276,5.6785,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,128,1,512,512,mla_absorbed,,64,8.72592,1008.042,247.065,3.0883,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,128,1,512,512,mla_absorbed,,64,6.152544,1429.668,32.722,0.409,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,128,1,512,512,mla_absorbed,,64,12.56432,1400.17,21.365,0.2671,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,128,1,512,512,mla_absorbed,,64,12.644896,1391.248,26.536,0.3317,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,128,1,512,512,mla_absorbed,,64,25.698175,1369.139,15.669,0.1959,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,128,1,512,512,mla_absorbed,,64,5.448224,6.307,24.683,0.3085,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,128,1,512,512,mla_absorbed,,64,716.095276,6.142,24.038,0.3005,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,128,1,512,512,mla_absorbed,,64,5.48352,12.532,24.572,0.3072,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,128,1,512,512,mla_absorbed,,64,722.045471,12.182,23.886,0.2986,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,128,1,512,512,mla_absorbed,,64,5.48528,18.792,24.612,0.3077,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,128,1,512,512,mla_absorbed,,64,722.421265,18.264,23.92,0.299,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,128,1,512,512,mla_absorbed,,64,5.490048,25.034,24.638,0.308,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,128,1,512,512,mla_absorbed,,64,723.953369,24.3,23.916,0.299,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,128,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,0.552096,497.881,490.01,6.1251,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,8.911744,493.511,121.427,1.5178,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,8.81776,498.771,15.221,0.1903,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,17.969889,489.491,11.204,0.14,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,16.574335,530.706,12.147,0.1518,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,33.993313,517.519,7.897,0.0987,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,0.392,43.826,342.727,4.2841,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,37.2952,58.963,461.095,5.7637,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,10.667136,3.221,12.607,0.1576,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,1314.764771,3.345,13.092,0.1637,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,10.644128,4.842,12.646,0.1581,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,1313.325806,5.023,13.12,0.164,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,10.638144,6.46,12.666,0.1583,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,1313.046143,6.699,13.135,0.1642,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,0.643232,427.339,420.583,5.2573,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,4.351616,1010.67,248.673,3.1084,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,2.97488,1478.395,45.117,0.564,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,6.255136,1406.219,32.186,0.4023,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,6.318784,1392.055,31.862,0.3983,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,12.496736,1407.743,21.48,0.2685,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,2.719584,6.317,49.4,0.6175,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,356.009064,6.177,48.304,0.6038,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,2.744192,12.521,49.005,0.6126,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,358.620453,12.264,47.999,0.6,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,2.7448,18.777,49.042,0.613,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,359.4039,18.356,47.941,0.5993,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,2.741792,25.064,49.144,0.6143,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,359.597504,24.461,47.962,0.5995,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,64,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,0.291488,471.508,467.652,5.8456,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,4.20768,522.621,129.587,1.6198,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,3.81648,576.191,26.376,0.3297,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,8.79536,500.042,19.075,0.2384,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,8.7632,501.877,15.316,0.1915,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,17.927937,490.636,11.23,0.1404,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,0.227424,37.771,590.453,7.3807,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,21.801249,50.433,788.407,9.8551,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,10.745888,1.599,12.502,0.1563,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,643.737793,3.416,26.714,0.3339,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,10.73136,2.401,12.525,0.1566,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,643.04541,5.13,26.756,0.3344,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,10.724384,3.204,12.54,0.1567,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,642.14679,6.849,26.806,0.3351,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,0.368448,373.021,369.97,4.6246,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,2.195616,1001.552,248.34,3.1043,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,1.458944,1507.271,68.997,0.8625,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,2.928608,1501.753,57.287,0.7161,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,2.982976,1474.382,44.995,0.5624,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,6.311904,1393.572,31.896,0.3987,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,1.401568,6.129,95.809,1.1976,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,177.838562,6.183,96.651,1.2081,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,1.42304,12.073,94.41,1.1801,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,178.098785,12.347,96.557,1.207,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,1.4248,18.087,94.339,1.1792,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,178.246597,18.505,96.524,1.2065,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,1.422976,24.146,94.506,1.1813,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,178.475906,24.642,96.447,1.2056,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,32,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.161184,426.342,429.36,5.367,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,1.876448,585.954,147.526,1.8441,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,1.896288,579.823,44.237,0.553,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,4.016064,547.557,37.598,0.47,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,3.846048,571.762,26.173,0.3272,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,8.735264,503.482,19.206,0.2401,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,0.13632,31.507,984.819,12.3102,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,11.61824,47.318,1479.059,18.4882,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,10.676992,0.805,12.577,0.1572,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,318.624481,3.451,53.945,0.6743,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,10.655808,1.209,12.605,0.1576,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,318.776764,5.174,53.933,0.6742,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,10.649056,1.613,12.616,0.1577,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,317.787323,6.92,54.114,0.6764,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.222016,309.525,311.716,3.8965,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,1.18128,930.78,234.342,2.9293,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,0.778784,1411.831,107.714,1.3464,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,1.442176,1524.795,104.699,1.3087,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,1.454816,1511.547,69.193,0.8649,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,3.008576,1461.837,55.765,0.6971,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,1.399392,3.069,95.935,1.1992,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,89.607681,6.135,191.77,2.3971,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,1.420928,6.045,94.504,1.1813,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,89.910301,12.229,191.171,2.3896,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,1.420768,9.069,94.538,1.1817,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,89.799072,18.366,191.455,2.3932,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,1.420192,12.097,94.599,1.1825,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,89.795677,24.489,191.509,2.3939,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,16,1,512,512,mla_absorbed,,64,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, diff --git a/benchmark/attention_inference/results/deepseek_v4/b300/deepseek_v4_context.png b/benchmark/attention_inference/results/deepseek_v4/b300/deepseek_v4_context.png deleted file mode 100644 index 899146486..000000000 Binary files a/benchmark/attention_inference/results/deepseek_v4/b300/deepseek_v4_context.png and /dev/null differ diff --git a/benchmark/attention_inference/results/deepseek_v4/b300/deepseek_v4_generation.png b/benchmark/attention_inference/results/deepseek_v4/b300/deepseek_v4_generation.png deleted file mode 100644 index 7bade7ed4..000000000 Binary files a/benchmark/attention_inference/results/deepseek_v4/b300/deepseek_v4_generation.png and /dev/null differ diff --git a/benchmark/attention_inference/results/deepseek_v4/gb200/deepseek_v4_20260818_130603.csv b/benchmark/attention_inference/results/deepseek_v4/gb200/deepseek_v4_20260818_130603.csv new file mode 100644 index 000000000..060996b4c --- /dev/null +++ b/benchmark/attention_inference/results/deepseek_v4/gb200/deepseek_v4_20260818_130603.csv @@ -0,0 +1,225 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,0.58544,469.524,462.101,5.7763,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,8.91232,493.479,121.42,1.5177,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,8.745376,502.9,15.347,0.1918,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,17.902529,491.332,11.246,0.1406,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,16.805887,523.394,11.98,0.1497,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,34.038719,516.829,7.886,0.0986,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,0.387328,44.355,346.861,4.3358,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,38.478306,57.15,446.918,5.5865,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,10.527904,3.264,12.774,0.1597,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,1264.319702,3.479,13.615,0.1702,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,10.507776,4.905,12.811,0.1601,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,1262.240112,5.226,13.65,0.1706,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,10.501792,6.544,12.83,0.1604,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,1261.393921,6.973,13.673,0.1709,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,0.593536,463.119,455.798,5.6975,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,3.943968,1115.132,274.376,3.4297,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,2.824832,1556.923,47.514,0.5939,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,6.154944,1429.11,32.71,0.4089,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,6.083488,1445.896,33.094,0.4137,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,12.182304,1444.077,22.035,0.2754,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,2.6568,6.466,50.568,0.6321,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,321.818939,6.833,53.436,0.6679,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,2.695232,12.748,49.895,0.6237,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,322.061829,13.656,53.448,0.6681,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,2.699328,19.093,49.868,0.6234,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,322.499268,20.456,53.427,0.6678,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,2.702016,25.433,49.867,0.6233,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,322.130676,27.306,53.54,0.6693,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,0.314944,436.392,432.823,5.4103,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,4.273632,514.556,127.587,1.5948,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,4.265408,515.548,23.6,0.295,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,8.769984,501.489,19.13,0.2391,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,8.762432,501.921,15.317,0.1915,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,17.629601,498.939,11.42,0.1427,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,0.228256,37.633,588.301,7.3538,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,22.333696,49.231,769.611,9.6201,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,10.584448,1.623,12.693,0.1587,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,616.182922,3.569,27.908,0.3489,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,10.567232,2.439,12.72,0.159,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,615.440735,5.36,27.956,0.3494,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,10.560192,3.254,12.735,0.1592,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,615.218201,7.149,27.979,0.3497,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,0.330432,415.937,412.535,5.1567,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,1.991968,1103.945,273.729,3.4216,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,1.468064,1497.907,68.569,0.8571,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,2.848704,1543.876,58.894,0.7362,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,2.944544,1493.626,45.582,0.5698,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,5.98928,1468.639,33.614,0.4202,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,1.355808,6.336,99.043,1.238,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,160.919128,6.833,106.813,1.3352,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,1.397344,12.295,96.146,1.2018,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,161.492767,13.617,106.486,1.3311,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,1.40576,18.332,95.617,1.1952,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,161.01236,20.486,106.855,1.3357,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,1.397344,24.589,96.24,1.203,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,161.447586,27.241,106.619,1.3327,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.17392,395.121,397.919,4.974,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,1.92256,571.9,143.987,1.7998,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,1.951104,563.533,42.994,0.5374,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,4.098656,536.523,36.84,0.4605,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,4.102016,536.084,24.54,0.3067,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,8.888576,494.798,18.875,0.2359,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,0.13472,31.881,996.515,12.4564,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,11.741728,46.821,1463.504,18.2938,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,10.521024,0.816,12.763,0.1595,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,311.232666,3.533,55.226,0.6903,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,10.506464,1.226,12.784,0.1598,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,311.148712,5.301,55.255,0.6907,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,10.493984,1.637,12.802,0.16,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,310.647064,7.079,55.358,0.692,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.197536,347.883,350.346,4.3793,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,1.05824,1039.0,261.589,3.2699,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,0.791776,1388.665,105.947,1.3243,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,1.441568,1525.438,104.744,1.3093,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,1.495648,1470.281,67.304,0.8413,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,2.806496,1567.095,59.78,0.7472,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,1.351072,3.179,99.366,1.2421,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,81.012131,6.786,212.117,2.6515,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,1.393696,6.163,96.35,1.2044,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,81.427681,13.503,211.086,2.6386,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,1.391328,9.261,96.538,1.2067,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,81.959709,20.123,209.767,2.6221,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,1.395968,12.307,96.241,1.203,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,82.159615,26.765,209.308,2.6163,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,512,512,mla_absorbed,,64,0.1032,332.943,345.461,4.3183,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,512,512,mla_absorbed,,64,0.949088,579.246,150.256,1.8782,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,512,512,mla_absorbed,,64,1.791808,306.816,42.135,0.5267,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,512,512,mla_absorbed,,64,3.645728,301.589,39.116,0.489,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,512,512,mla_absorbed,,64,1.870624,587.778,44.844,0.5605,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,512,512,mla_absorbed,,64,4.349088,505.629,34.719,0.434,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,512,512,mla_absorbed,,64,0.133056,16.14,1008.854,12.6107,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,512,512,mla_absorbed,,64,11.743136,23.408,1463.15,18.2894,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,512,512,mla_absorbed,,64,10.517472,0.408,12.765,0.1596,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,512,512,mla_absorbed,,64,158.423294,3.47,108.469,1.3559,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,512,512,mla_absorbed,,64,10.501216,0.613,12.786,0.1598,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,512,512,mla_absorbed,,64,158.292633,5.21,108.572,1.3572,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,512,512,mla_absorbed,,64,10.490112,0.819,12.801,0.16,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,512,512,mla_absorbed,,64,158.119553,6.954,108.704,1.3588,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,512,512,mla_absorbed,,64,0.151744,226.432,234.946,2.9368,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,512,512,mla_absorbed,,64,0.559168,983.168,255.033,3.1879,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,512,512,mla_absorbed,,64,0.758336,724.95,99.557,1.2445,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,512,512,mla_absorbed,,64,1.398688,786.102,101.957,1.2745,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,512,512,mla_absorbed,,64,0.785184,1400.323,106.836,1.3355,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,512,512,mla_absorbed,,64,1.440128,1526.964,104.848,1.3106,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,8,1,512,512,mla_absorbed,,64,1.348704,1.592,99.528,1.2441,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,8,1,512,512,mla_absorbed,,64,41.669086,6.597,412.343,5.1543,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,512,512,mla_absorbed,,64,1.39232,3.085,96.422,1.2053,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,512,512,mla_absorbed,,64,41.977825,13.096,409.36,5.117,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,512,512,mla_absorbed,,64,1.395232,4.617,96.233,1.2029,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,512,512,mla_absorbed,,64,42.002335,19.633,409.172,5.1146,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,512,512,mla_absorbed,,64,1.392256,6.17,96.45,1.2056,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,512,512,mla_absorbed,,64,42.002911,26.177,409.216,5.1152,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,2048,2048,128,1,512,512,mla_absorbed,,64,1.122336,489.832,480.22,6.0027,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,8192,8192,128,1,512,512,mla_absorbed,,64,17.89024,491.67,120.505,1.5063,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,512,65536,128,1,512,512,mla_absorbed,,64,16.913536,520.062,11.903,0.1488,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,512,131072,128,1,512,512,mla_absorbed,,64,34.155521,515.061,7.859,0.0982,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,1024,65536,128,1,512,512,mla_absorbed,,64,34.215744,514.155,9.807,0.1226,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,1024,131072,128,1,512,512,mla_absorbed,,64,68.596642,512.917,5.87,0.0734,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,1,131072,128,1,512,512,mla_absorbed,,64,0.71184,48.269,188.919,2.3615,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,1,131072,128,1,512,512,mla_absorbed,,64,72.559875,60.613,237.231,2.9654,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,2,131072,128,1,512,512,mla_absorbed,,64,23.605728,2.911,5.708,0.0714,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,2,131072,128,1,512,512,mla_absorbed,,64,2667.791504,3.297,6.465,0.0808,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,3,131072,128,1,512,512,mla_absorbed,,64,23.590401,4.37,5.723,0.0715,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,3,131072,128,1,512,512,mla_absorbed,,64,2663.615967,4.953,6.488,0.0811,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,4,131072,128,1,512,512,mla_absorbed,,64,23.57056,5.831,5.739,0.0717,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,4,131072,128,1,512,512,mla_absorbed,,64,2661.652344,6.609,6.505,0.0813,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,128,1,512,512,mla_absorbed,,64,1.10896,495.74,486.012,6.0752,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,128,1,512,512,mla_absorbed,,64,7.995264,1100.163,269.644,3.3705,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,128,1,512,512,mla_absorbed,,64,5.971392,1473.039,33.715,0.4214,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,128,1,512,512,mla_absorbed,,64,12.37232,1421.899,21.696,0.2712,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,128,1,512,512,mla_absorbed,,64,11.800512,1490.798,28.435,0.3554,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,128,1,512,512,mla_absorbed,,64,23.63184,1488.855,17.039,0.213,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,128,1,512,512,mla_absorbed,,64,5.241568,6.555,25.656,0.3207,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,128,1,512,512,mla_absorbed,,64,643.842957,6.831,26.735,0.3342,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,128,1,512,512,mla_absorbed,,64,5.323968,12.908,25.309,0.3164,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,128,1,512,512,mla_absorbed,,64,647.710876,13.58,26.628,0.3328,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,128,1,512,512,mla_absorbed,,64,5.293088,19.474,25.506,0.3188,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,128,1,512,512,mla_absorbed,,64,646.527893,20.408,26.728,0.3341,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,128,1,512,512,mla_absorbed,,64,5.302592,25.919,25.509,0.3189,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,128,1,512,512,mla_absorbed,,64,649.327881,27.093,26.665,0.3333,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,0.5768,476.557,469.023,5.8628,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,8.901824,494.061,121.563,1.5195,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,8.83936,497.553,15.184,0.1898,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,17.553568,501.1,11.469,0.1434,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,16.840193,522.327,11.955,0.1494,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,33.932255,518.45,7.911,0.0989,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,0.38864,44.205,345.69,4.3211,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,38.517025,57.092,446.469,5.5809,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,10.523008,3.265,12.78,0.1597,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,1263.834351,3.48,13.62,0.1702,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,10.51456,4.902,12.802,0.16,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,1262.46582,5.226,13.648,0.1706,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,10.504768,6.542,12.827,0.1603,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,1261.344238,6.974,13.673,0.1709,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,0.592352,464.045,456.709,5.7089,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,3.92672,1120.031,275.581,3.4448,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,2.831168,1553.439,47.407,0.5926,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,5.993024,1467.722,33.593,0.4199,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,5.819264,1511.547,34.597,0.4325,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,11.92,1475.855,22.52,0.2815,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,2.660128,6.458,50.505,0.6313,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,321.155518,6.847,53.546,0.6693,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,2.697952,12.735,49.845,0.6231,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,321.720825,13.67,53.504,0.6688,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,2.699744,19.091,49.861,0.6233,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,321.58429,20.514,53.579,0.6697,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,2.700064,25.451,49.903,0.6238,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,322.900818,27.241,53.413,0.6677,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,0.314112,437.548,433.969,5.4246,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,3.82032,575.612,142.726,1.7841,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,4.16128,528.449,24.19,0.3024,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,8.825856,498.314,19.009,0.2376,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,8.568064,513.307,15.665,0.1958,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,17.556032,501.03,11.468,0.1433,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,0.229184,37.481,585.919,7.324,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,22.329248,49.241,769.764,9.6221,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,10.584544,1.623,12.693,0.1587,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,616.499634,3.567,27.894,0.3487,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,10.567136,2.439,12.72,0.159,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,615.499146,5.359,27.953,0.3494,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,10.559104,3.254,12.736,0.1592,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,614.701233,7.155,28.003,0.35,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,0.33008,416.381,412.975,5.1622,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,1.99456,1102.51,273.373,3.4172,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,1.482528,1483.293,67.9,0.8487,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,2.888032,1522.852,58.092,0.7262,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,2.86448,1535.373,46.856,0.5857,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,6.094432,1443.3,33.035,0.4129,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,1.356992,6.33,98.957,1.237,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,160.382553,6.856,107.17,1.3396,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,1.40016,12.27,95.952,1.1994,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,160.724121,13.682,106.995,1.3374,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,1.397376,18.442,96.191,1.2024,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,160.791962,20.514,107.002,1.3375,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,1.398272,24.573,96.176,1.2022,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,161.117691,27.297,106.838,1.3355,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.177824,386.447,389.183,4.8648,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,1.916032,573.848,144.478,1.806,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,1.908384,576.148,43.957,0.5495,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,3.914144,561.815,38.577,0.4822,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,4.056224,542.136,24.817,0.3102,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,8.664352,507.602,19.363,0.242,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,0.132928,32.31,1009.949,12.6244,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,11.738208,46.835,1463.943,18.2993,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,10.517152,0.817,12.768,0.1596,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,311.520935,3.529,55.175,0.6897,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,10.505056,1.227,12.786,0.1598,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,311.573639,5.293,55.179,0.6897,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,10.497792,1.637,12.798,0.16,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,310.68396,7.078,55.351,0.6919,20,True,,NVIDIA GB200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.198912,345.477,347.923,4.349,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,1.0512,1045.959,263.341,3.2918,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,0.789024,1393.508,106.316,1.329,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,1.441376,1525.642,104.758,1.3095,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,1.468192,1497.776,68.563,0.857,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,2.836992,1550.25,59.137,0.7392,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,1.349824,3.182,99.458,1.2432,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,81.346786,6.758,211.245,2.6406,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,1.396544,6.151,96.154,1.2019,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,81.109856,13.556,211.913,2.6489,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,1.397088,9.223,96.14,1.2017,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,81.605217,20.21,210.678,2.6335,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,1.397728,12.291,96.119,1.2015,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,81.62307,26.941,210.684,2.6335,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], diff --git a/benchmark/attention_inference/results/deepseek_v4/gb200/deepseek_v4_context.png b/benchmark/attention_inference/results/deepseek_v4/gb200/deepseek_v4_context.png new file mode 100644 index 000000000..ca2c380eb Binary files /dev/null and b/benchmark/attention_inference/results/deepseek_v4/gb200/deepseek_v4_context.png differ diff --git a/benchmark/attention_inference/results/deepseek_v4/gb200/deepseek_v4_generation.png b/benchmark/attention_inference/results/deepseek_v4/gb200/deepseek_v4_generation.png new file mode 100644 index 000000000..7ca2b9368 Binary files /dev/null and b/benchmark/attention_inference/results/deepseek_v4/gb200/deepseek_v4_generation.png differ diff --git a/benchmark/attention_inference/results/deepseek_v4/gb300/deepseek_v4_20260818_131234.csv b/benchmark/attention_inference/results/deepseek_v4/gb300/deepseek_v4_20260818_131234.csv new file mode 100644 index 000000000..03b0df8df --- /dev/null +++ b/benchmark/attention_inference/results/deepseek_v4/gb300/deepseek_v4_20260818_131234.csv @@ -0,0 +1,225 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,0.499232,550.602,541.898,6.7737,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,7.333728,599.701,147.555,1.8444,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,7.557984,581.907,17.758,0.222,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,15.237344,577.272,13.213,0.1652,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,13.971104,629.592,14.41,0.1801,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,27.949535,629.427,9.604,0.1201,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,0.380032,45.206,353.52,4.419,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,37.917919,57.994,453.523,5.669,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,10.456576,3.286,12.861,0.1608,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,1244.747559,3.533,13.829,0.1729,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,10.436128,4.939,12.899,0.1612,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,1244.266235,5.302,13.848,0.1731,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,10.426336,6.591,12.923,0.1615,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,1243.691528,7.073,13.868,0.1733,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,0.58096,473.144,465.665,5.8208,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,3.824768,1149.886,282.927,3.5366,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,2.738848,1605.802,49.005,0.6126,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,5.363392,1640.024,37.537,0.4692,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,5.488032,1602.777,36.685,0.4586,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,10.8536,1620.862,24.732,0.3092,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,2.642816,6.501,50.835,0.6354,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,300.115112,7.327,57.3,0.7163,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,2.681248,12.815,50.156,0.6269,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,300.765839,14.623,57.232,0.7154,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,2.688192,19.173,50.075,0.6259,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,300.883087,21.926,57.265,0.7158,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,2.686976,25.575,50.146,0.6268,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,301.090546,29.214,57.282,0.716,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,0.2672,514.367,510.16,6.377,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,3.513568,625.866,155.187,1.9398,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,3.636128,604.771,27.684,0.3461,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,7.524416,584.503,22.297,0.2787,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,7.509312,585.679,17.874,0.2234,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,15.117248,581.858,13.318,0.1665,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,0.221568,38.769,606.059,7.5757,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,21.907455,50.189,784.585,9.8073,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,10.485792,1.638,12.812,0.1602,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,606.523804,3.626,28.353,0.3544,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,10.46416,2.463,12.845,0.1606,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,606.406677,5.439,28.372,0.3547,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,10.464448,3.283,12.851,0.1606,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,605.588928,7.262,28.424,0.3553,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,0.319808,429.755,426.24,5.328,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,1.951328,1126.937,279.43,3.4929,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,1.426688,1541.348,70.557,0.882,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,2.718432,1617.862,61.717,0.7715,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,2.726432,1613.114,49.228,0.6154,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,5.403168,1627.951,37.261,0.4658,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,1.352544,6.351,99.282,1.241,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,150.154526,7.323,114.47,1.4309,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,1.39024,12.357,96.637,1.208,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,150.422562,14.619,114.322,1.429,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,1.392576,18.505,96.522,1.2065,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,150.446976,21.925,114.359,1.4295,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,1.399296,24.555,96.105,1.2013,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,150.498398,29.223,114.376,1.4297,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.149696,459.06,462.31,5.7789,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,1.659232,662.663,166.839,2.0855,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,1.797536,611.677,46.667,0.5833,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,3.676224,598.174,41.073,0.5134,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,3.625408,606.559,27.766,0.3471,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,7.461216,589.454,22.486,0.2811,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,0.131008,32.784,1024.75,12.8094,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,11.413248,48.168,1505.624,18.8203,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,10.419104,0.824,12.888,0.1611,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,307.551941,3.575,55.887,0.6986,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,10.39792,1.239,12.918,0.1615,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,307.616852,5.361,55.889,0.6986,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,10.39744,1.652,12.921,0.1615,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,306.854156,7.166,56.042,0.7005,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.192576,356.843,359.37,4.4921,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,1.016736,1081.413,272.267,3.4033,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,0.764096,1438.97,109.785,1.3723,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,1.400512,1570.157,107.814,1.3477,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,1.423712,1544.57,70.705,0.8838,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,2.728736,1611.752,61.483,0.7685,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,1.349344,3.183,99.493,1.2437,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,75.172546,7.313,228.595,2.8574,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,1.394208,6.161,96.315,1.2039,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,75.363167,14.59,228.072,2.8509,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,1.392736,9.252,96.44,1.2055,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,75.419037,21.868,227.959,2.8495,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,1.396768,12.3,96.185,1.2023,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,75.442047,29.149,227.945,2.8493,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,512,512,mla_absorbed,,64,0.088864,386.655,401.193,5.0149,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,512,512,mla_absorbed,,64,0.847744,648.493,168.219,2.1027,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,512,512,mla_absorbed,,64,1.70768,321.931,44.211,0.5526,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,512,512,mla_absorbed,,64,3.469856,316.875,41.099,0.5137,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,512,512,mla_absorbed,,64,1.768992,621.547,47.42,0.5928,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,512,512,mla_absorbed,,64,3.562048,617.348,42.39,0.5299,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,512,512,mla_absorbed,,64,0.13152,16.328,1020.636,12.758,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,512,512,mla_absorbed,,64,11.40352,24.105,1506.725,18.8341,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,512,512,mla_absorbed,,64,10.409824,0.413,12.897,0.1612,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,512,512,mla_absorbed,,64,156.845093,3.505,109.561,1.3695,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,512,512,mla_absorbed,,64,10.390624,0.62,12.922,0.1615,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,512,512,mla_absorbed,,64,156.631714,5.265,109.723,1.3715,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,512,512,mla_absorbed,,64,10.382656,0.827,12.933,0.1617,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,512,512,mla_absorbed,,64,156.417892,7.029,109.887,1.3736,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,512,512,mla_absorbed,,64,0.15616,220.029,228.302,2.8538,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,512,512,mla_absorbed,,64,0.538848,1020.243,264.65,3.3081,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,512,512,mla_absorbed,,64,0.750816,732.211,100.554,1.2569,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,512,512,mla_absorbed,,64,1.394592,788.411,102.257,1.2782,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,512,512,mla_absorbed,,64,0.761088,1444.658,110.219,1.3777,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,512,512,mla_absorbed,,64,1.40592,1564.117,107.399,1.3425,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,8,1,512,512,mla_absorbed,,64,1.349568,1.591,99.465,1.2433,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,8,1,512,512,mla_absorbed,,64,37.830078,7.266,454.188,5.6773,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,512,512,mla_absorbed,,64,1.39248,3.084,96.411,1.2051,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,512,512,mla_absorbed,,64,37.883297,14.512,453.605,5.6701,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,512,512,mla_absorbed,,64,1.385248,4.651,96.926,1.2116,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,512,512,mla_absorbed,,64,37.870399,21.775,453.815,5.6727,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,512,512,mla_absorbed,,64,1.380128,6.224,97.298,1.2162,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_flash-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,512,512,mla_absorbed,,64,37.872543,29.032,453.845,5.6731,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,2048,2048,128,1,512,512,mla_absorbed,,64,0.965696,569.285,558.114,6.9764,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,8192,8192,128,1,512,512,mla_absorbed,,64,14.645376,600.605,147.205,1.8401,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,512,65536,128,1,512,512,mla_absorbed,,64,14.01936,627.425,14.361,0.1795,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,512,131072,128,1,512,512,mla_absorbed,,64,27.989344,628.532,9.591,0.1199,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,1024,65536,128,1,512,512,mla_absorbed,,64,28.068417,626.761,11.955,0.1494,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,1024,131072,128,1,512,512,mla_absorbed,,64,56.594913,621.688,7.115,0.0889,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,1,131072,128,1,512,512,mla_absorbed,,64,0.701888,48.953,191.597,2.395,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,1,131072,128,1,512,512,mla_absorbed,,64,70.960449,61.979,242.578,3.0322,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,2,131072,128,1,512,512,mla_absorbed,,64,23.306528,2.949,5.781,0.0723,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,2,131072,128,1,512,512,mla_absorbed,,64,2618.563477,3.359,6.586,0.0823,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,3,131072,128,1,512,512,mla_absorbed,,64,23.274944,4.429,5.8,0.0725,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,3,131072,128,1,512,512,mla_absorbed,,64,2615.818359,5.044,6.606,0.0826,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,4,131072,128,1,512,512,mla_absorbed,,64,23.2752,5.905,5.812,0.0726,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,4,131072,128,1,512,512,mla_absorbed,,64,2614.961182,6.728,6.621,0.0828,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,128,1,512,512,mla_absorbed,,64,1.09248,503.218,493.344,6.1668,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,128,1,512,512,mla_absorbed,,64,7.587616,1159.27,284.13,3.5516,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,128,1,512,512,mla_absorbed,,64,5.422624,1622.11,37.127,0.4641,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,128,1,512,512,mla_absorbed,,64,10.87952,1617.0,24.673,0.3084,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,128,1,512,512,mla_absorbed,,64,10.526016,1671.305,31.878,0.3985,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,128,1,512,512,mla_absorbed,,64,20.704224,1699.381,19.448,0.2431,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,128,1,512,512,mla_absorbed,,64,5.193888,6.615,25.892,0.3236,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,128,1,512,512,mla_absorbed,,64,602.45636,7.3,28.572,0.3572,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,128,1,512,512,mla_absorbed,,64,5.258816,13.067,25.622,0.3203,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,128,1,512,512,mla_absorbed,,64,605.099121,14.537,28.503,0.3563,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,128,1,512,512,mla_absorbed,,64,5.254656,19.617,25.692,0.3212,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,128,1,512,512,mla_absorbed,,64,605.268738,21.799,28.55,0.3569,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,128,1,512,512,mla_absorbed,,64,5.249728,26.18,25.766,0.3221,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,128,1,512,512,mla_absorbed,,64,605.571411,29.051,28.591,0.3574,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,0.499488,550.319,541.62,6.7702,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,7.291264,603.194,148.415,1.8552,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,7.591968,579.303,17.679,0.221,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,15.257536,576.508,13.195,0.1649,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,13.969472,629.665,14.412,0.1801,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,28.010559,628.056,9.583,0.1198,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,0.380832,45.111,352.777,4.4097,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,37.91581,57.998,453.548,5.6694,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,10.454016,3.287,12.864,0.1608,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,1245.676025,3.531,13.819,0.1727,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,10.432032,4.941,12.904,0.1613,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,1244.362915,5.302,13.847,0.1731,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,10.421056,6.594,12.93,0.1616,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,1243.388184,7.074,13.871,0.1734,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,0.576352,476.927,469.388,5.8673,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,3.822688,1150.511,283.081,3.5385,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,2.738336,1606.102,49.014,0.6127,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,5.39888,1629.244,37.29,0.4661,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,5.448736,1614.336,36.949,0.4619,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,10.82832,1624.646,24.79,0.3099,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,2.6392,6.509,50.905,0.6363,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,299.948822,7.331,57.332,0.7166,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,2.690688,12.77,49.98,0.6247,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,299.889984,14.666,57.399,0.7175,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,2.684064,19.202,50.152,0.6269,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,299.520233,22.025,57.526,0.7191,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,2.685344,25.591,50.177,0.6272,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,299.761688,29.344,57.536,0.7192,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,0.267136,514.491,510.283,6.3785,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,3.535104,622.053,154.241,1.928,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,3.700224,594.295,27.205,0.3401,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,7.506912,585.866,22.349,0.2794,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,7.565152,581.356,17.742,0.2218,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,15.115488,581.926,13.319,0.1665,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,0.221952,38.702,605.01,7.5626,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,21.888832,50.232,785.252,9.8157,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,10.483968,1.639,12.815,0.1602,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,606.831238,3.624,28.338,0.3542,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,10.464448,2.463,12.845,0.1606,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,606.071167,5.442,28.388,0.3548,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,10.46144,3.284,12.855,0.1607,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,605.916443,7.259,28.409,0.3551,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,0.320768,428.468,424.964,5.3121,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,1.951392,1126.9,279.421,3.4928,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,1.421376,1547.109,70.821,0.8853,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,2.719424,1617.271,61.694,0.7712,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,2.733568,1608.903,49.1,0.6137,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,5.421824,1622.349,37.133,0.4642,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,1.350432,6.361,99.437,1.243,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,149.930084,7.333,114.642,1.433,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,1.391488,12.346,96.55,1.2069,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,150.377441,14.623,114.357,1.4295,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,1.39984,18.409,96.021,1.2003,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,150.438873,21.926,114.366,1.4296,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,1.396384,24.606,96.306,1.2038,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,150.506912,29.222,114.37,1.4296,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.150432,456.814,460.048,5.7506,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,1.669184,658.712,165.844,2.073,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,1.80016,610.785,46.599,0.5825,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,3.60176,610.541,41.923,0.524,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,3.592224,612.162,28.023,0.3503,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,7.539296,583.35,22.253,0.2782,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,0.13168,32.617,1019.521,12.744,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,11.40112,48.219,1507.226,18.8403,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,10.417792,0.825,12.89,0.1611,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,307.552246,3.575,55.887,0.6986,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,10.399488,1.239,12.916,0.1614,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,307.677856,5.36,55.878,0.6985,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,10.390176,1.653,12.93,0.1616,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,307.19812,7.158,55.979,0.6997,20,True,,NVIDIA GB300,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.195968,350.667,353.15,4.4144,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,1.01376,1084.588,273.067,3.4133,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,0.768224,1431.238,109.195,1.3649,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,1.404768,1565.4,107.487,1.3436,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,1.421568,1546.9,70.811,0.8851,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,2.722688,1615.333,61.62,0.7703,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,1.349152,3.183,99.507,1.2438,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,75.156769,7.315,228.643,2.858,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,1.397376,6.147,96.097,1.2012,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,75.259453,14.61,228.387,2.8548,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,1.391712,9.258,96.511,1.2064,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,75.35405,21.887,228.156,2.8519,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,1.39072,12.353,96.604,1.2075,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +deepseek_v4,dsv4_pro-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,75.34816,29.185,228.229,2.8529,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d512[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], diff --git a/benchmark/attention_inference/results/deepseek_v4/gb300/deepseek_v4_context.png b/benchmark/attention_inference/results/deepseek_v4/gb300/deepseek_v4_context.png new file mode 100644 index 000000000..5b00e1cbf Binary files /dev/null and b/benchmark/attention_inference/results/deepseek_v4/gb300/deepseek_v4_context.png differ diff --git a/benchmark/attention_inference/results/deepseek_v4/gb300/deepseek_v4_generation.png b/benchmark/attention_inference/results/deepseek_v4/gb300/deepseek_v4_generation.png new file mode 100644 index 000000000..7d88ce722 Binary files /dev/null and b/benchmark/attention_inference/results/deepseek_v4/gb300/deepseek_v4_generation.png differ diff --git a/benchmark/attention_inference/results/deepseek_v4/h200/deepseek_v4_20260818_135743.csv b/benchmark/attention_inference/results/deepseek_v4/h200/deepseek_v4_20260818_135743.csv new file mode 100644 index 000000000..09ade9aa4 --- /dev/null +++ b/benchmark/attention_inference/results/deepseek_v4/h200/deepseek_v4_20260818_135743.csv @@ -0,0 +1,113 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,2.940544,93.479,92.001,1.9167,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,43.415329,101.302,24.925,0.5193,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,49.576,88.713,2.707,0.0564,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,100.397728,87.612,2.005,0.0418,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,96.960098,90.719,2.076,0.0433,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,198.092987,88.808,1.355,0.0282,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,0.440128,39.034,305.249,6.3594,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,44.833088,49.049,383.57,7.9911,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,10.892416,3.154,12.346,0.2572,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,1522.686768,2.888,11.305,0.2355,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,10.89344,4.731,12.357,0.2574,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,1523.757446,4.329,11.308,0.2356,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,10.920768,6.293,12.338,0.257,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,1523.969482,5.772,11.317,0.2358,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,1.530304,89.812,89.077,1.8558,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,21.85984,100.596,24.943,0.5197,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,24.751553,88.844,4.067,0.0847,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,49.655998,88.57,3.379,0.0704,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,48.867329,90.0,2.747,0.0572,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,99.551903,88.357,2.022,0.0421,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,0.244736,35.099,548.686,11.431,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,23.144705,47.506,742.643,15.4717,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,10.408448,1.651,12.908,0.2689,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,740.466187,2.97,23.224,0.4838,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,10.401728,2.477,12.922,0.2692,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,739.674255,4.459,23.26,0.4846,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,10.397984,3.304,12.933,0.2694,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,737.853149,5.961,23.329,0.486,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.829536,82.841,83.427,1.7381,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,11.039584,99.597,25.076,0.5224,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,12.238688,89.839,6.854,0.1428,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,24.441055,89.973,6.178,0.1287,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,24.483232,89.818,4.112,0.0857,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,49.496769,88.855,3.39,0.0706,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,0.142016,30.243,945.32,19.6942,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,12.349184,44.518,1391.514,28.9899,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,10.30464,0.834,13.031,0.2715,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,364.844757,3.014,47.111,0.9815,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,10.325536,1.248,13.008,0.271,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,365.180939,4.516,47.079,0.9808,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,10.316672,1.665,13.022,0.2713,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,364.567505,6.032,47.17,0.9827,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,512,512,mla_absorbed,,64,0.478752,71.769,74.468,1.5514,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,512,512,mla_absorbed,,64,5.718144,96.142,24.939,0.5196,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,512,512,mla_absorbed,,64,5.550624,99.044,13.602,0.2834,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,512,512,mla_absorbed,,64,11.01232,99.844,12.95,0.2698,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,512,512,mla_absorbed,,64,12.206048,90.079,6.873,0.1432,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,512,512,mla_absorbed,,64,24.410879,90.084,6.186,0.1289,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,512,512,mla_absorbed,,64,0.141632,15.162,947.767,19.7451,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,512,512,mla_absorbed,,64,12.319584,22.312,1394.687,29.056,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,512,512,mla_absorbed,,64,10.26704,0.418,13.076,0.2724,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,512,512,mla_absorbed,,64,183.655777,2.993,93.567,1.9493,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,512,512,mla_absorbed,,64,10.289248,0.626,13.049,0.2719,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,512,512,mla_absorbed,,64,183.560486,4.492,93.627,1.9506,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,512,512,mla_absorbed,,64,10.364448,0.829,12.956,0.2699,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,512,512,mla_absorbed,,64,184.38327,5.963,93.22,1.9421,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,2048,2048,128,1,512,512,mla_absorbed,,64,5.749856,95.612,93.736,1.9528,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,8192,8192,128,1,512,512,mla_absorbed,,64,87.396355,100.646,24.668,0.5139,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,512,65536,128,1,512,512,mla_absorbed,,64,99.764702,88.168,2.018,0.042,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,512,131072,128,1,512,512,mla_absorbed,,64,200.149277,87.895,1.341,0.0279,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,1024,65536,128,1,512,512,mla_absorbed,,64,193.401505,90.962,1.735,0.0361,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,1024,131072,128,1,512,512,mla_absorbed,,64,395.77713,88.899,1.017,0.0212,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,1,131072,128,1,512,512,mla_absorbed,,64,0.769536,44.65,174.754,3.6407,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,1,131072,128,1,512,512,mla_absorbed,,64,87.832161,50.073,195.981,4.0829,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,2,131072,128,1,512,512,mla_absorbed,,64,24.629057,2.79,5.471,0.114,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,2,131072,128,1,512,512,mla_absorbed,,64,3184.618652,2.762,5.416,0.1128,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,3,131072,128,1,512,512,mla_absorbed,,64,24.614208,4.188,5.485,0.1143,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,3,131072,128,1,512,512,mla_absorbed,,64,3170.215088,4.162,5.451,0.1136,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,4,131072,128,1,512,512,mla_absorbed,,64,24.47056,5.617,5.528,0.1152,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,4,131072,128,1,512,512,mla_absorbed,,64,3170.186768,5.549,5.462,0.1138,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,2.936416,93.61,92.13,1.9194,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,43.463074,101.19,24.898,0.5187,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,49.597729,88.674,2.706,0.0564,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,100.353218,87.651,2.006,0.0418,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,96.970016,90.709,2.076,0.0433,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,198.196701,88.761,1.354,0.0282,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,0.438464,39.182,306.408,6.3835,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,45.13232,48.724,381.027,7.9381,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,10.871008,3.161,12.371,0.2577,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,1522.545654,2.889,11.306,0.2355,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,10.923616,4.718,12.323,0.2567,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,1516.224487,4.351,11.364,0.2367,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,10.948992,6.276,12.306,0.2564,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,1521.641724,5.781,11.334,0.2361,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,1.53456,89.562,88.83,1.8506,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,21.858433,100.603,24.945,0.5197,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,24.756161,88.827,4.066,0.0847,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,49.666401,88.552,3.378,0.0704,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,48.627232,90.444,2.76,0.0575,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,98.992386,88.856,2.034,0.0424,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,0.243488,35.279,551.498,11.4896,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,23.134144,47.528,742.982,15.4788,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,10.412736,1.65,12.902,0.2688,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,739.211487,2.975,23.263,0.4847,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,10.422432,2.473,12.897,0.2687,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,737.283264,4.474,23.336,0.4862,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,10.389664,3.307,12.944,0.2697,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,736.510986,5.971,23.372,0.4869,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.826624,83.133,83.721,1.7442,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,11.041984,99.576,25.07,0.5223,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,12.2488,89.765,6.849,0.1427,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,24.456127,89.917,6.174,0.1286,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,24.501375,89.751,4.108,0.0856,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,49.49181,88.864,3.39,0.0706,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,0.142688,30.1,940.867,19.6014,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,12.360096,44.478,1390.286,28.9643,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,10.303616,0.834,13.033,0.2715,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,365.529114,3.008,47.023,0.9796,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,10.3048,1.25,13.034,0.2715,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,364.886108,4.52,47.117,0.9816,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,10.246848,1.677,13.111,0.2732,20,True,,NVIDIA H200,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,365.036407,6.024,47.109,0.9814,20,True,,NVIDIA H200,cudnn 9.26.0, diff --git a/benchmark/attention_inference/results/deepseek_v4/h200/deepseek_v4_context.png b/benchmark/attention_inference/results/deepseek_v4/h200/deepseek_v4_context.png new file mode 100644 index 000000000..01c341581 Binary files /dev/null and b/benchmark/attention_inference/results/deepseek_v4/h200/deepseek_v4_context.png differ diff --git a/benchmark/attention_inference/results/deepseek_v4/h200/deepseek_v4_generation.png b/benchmark/attention_inference/results/deepseek_v4/h200/deepseek_v4_generation.png new file mode 100644 index 000000000..832436803 Binary files /dev/null and b/benchmark/attention_inference/results/deepseek_v4/h200/deepseek_v4_generation.png differ diff --git a/benchmark/attention_inference/results/deepseek_v4/rtxpro6000/deepseek_v4_20260818_140843.csv b/benchmark/attention_inference/results/deepseek_v4/rtxpro6000/deepseek_v4_20260818_140843.csv new file mode 100644 index 000000000..28341933e --- /dev/null +++ b/benchmark/attention_inference/results/deepseek_v4/rtxpro6000/deepseek_v4_20260818_140843.csv @@ -0,0 +1,113 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,3.152512,87.193,85.815,4.7888,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,46.747265,94.081,23.149,1.2918,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,49.045025,89.674,2.737,0.1527,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,98.164192,89.606,2.051,0.1144,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,92.054047,95.554,2.187,0.122,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,184.666809,95.264,1.454,0.0811,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,0.625248,27.477,214.873,11.9907,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,68.170334,32.258,252.26,14.077,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,12.38096,2.775,10.862,0.6061,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,1061.547363,4.143,16.215,0.9049,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,12.396352,4.158,10.859,0.606,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,1074.984497,6.137,16.028,0.8944,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,12.391456,5.546,10.874,0.6068,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,1082.197998,8.128,15.937,0.8893,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,1.63136,84.248,83.559,4.6629,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,23.570816,93.294,23.133,1.2909,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,24.575071,89.482,4.096,0.2286,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,49.269215,89.266,3.405,0.19,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,48.844994,90.041,2.748,0.1533,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,97.96582,89.787,2.055,0.1147,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,0.329088,26.102,408.047,22.7705,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,36.53392,30.096,470.474,26.2541,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,9.796896,1.754,13.713,0.7653,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,533.176575,4.124,32.253,1.7998,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,9.808832,2.627,13.703,0.7647,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,539.776306,6.111,31.874,1.7787,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,9.810176,3.502,13.708,0.765,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,541.378662,8.124,31.796,1.7743,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.877536,78.31,78.864,4.4009,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,12.001248,91.616,23.066,1.2872,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,16.131008,68.161,5.2,0.2902,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,32.483711,67.696,4.648,0.2594,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,24.42416,90.035,4.121,0.23,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,49.004383,89.748,3.424,0.191,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,0.22112,19.424,607.139,33.8805,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,18.438911,29.815,931.946,52.0059,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,9.768,0.879,13.747,0.7671,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,267.556946,4.109,64.241,3.5849,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,9.745184,1.322,13.783,0.7691,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,269.226776,6.126,63.859,3.5635,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,9.757504,1.761,13.769,0.7683,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,270.881073,8.118,63.484,3.5426,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,512,512,mla_absorbed,,64,0.50656,67.83,70.38,3.9274,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,512,512,mla_absorbed,,64,6.2136,88.476,22.951,1.2807,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,512,512,mla_absorbed,,64,7.35856,74.71,10.26,0.5725,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,512,512,mla_absorbed,,64,14.691872,74.838,9.706,0.5417,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,512,512,mla_absorbed,,64,16.161856,68.031,5.19,0.2896,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,512,512,mla_absorbed,,64,32.410816,67.848,4.659,0.26,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,512,512,mla_absorbed,,64,0.215904,9.946,621.731,34.6948,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,512,512,mla_absorbed,,64,18.169312,15.129,945.659,52.7711,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,512,512,mla_absorbed,,64,9.803136,0.438,13.695,0.7642,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,512,512,mla_absorbed,,64,135.232315,4.065,127.071,7.091,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,512,512,mla_absorbed,,64,9.802144,0.657,13.698,0.7644,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,512,512,mla_absorbed,,64,135.997986,6.064,126.371,7.0519,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,512,512,mla_absorbed,,64,9.79232,0.877,13.713,0.7652,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_flash-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,512,512,mla_absorbed,,64,136.518051,8.054,125.905,7.0259,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,2048,2048,128,1,512,512,mla_absorbed,,64,6.157472,89.283,87.531,4.8845,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,8192,8192,128,1,512,512,mla_absorbed,,64,93.065056,94.516,23.165,1.2927,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,512,65536,128,1,512,512,mla_absorbed,,64,92.355011,95.242,2.18,0.1216,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,512,131072,128,1,512,512,mla_absorbed,,64,184.985352,95.1,1.451,0.081,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,1024,65536,128,1,512,512,mla_absorbed,,64,183.194748,96.03,1.832,0.1022,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro,context,cudnn,bfloat16,bfloat16,1,1024,131072,128,1,512,512,mla_absorbed,,64,370.236786,95.032,1.088,0.0607,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,1,131072,128,1,512,512,mla_absorbed,,64,1.200512,28.621,112.019,6.251,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,1,131072,128,1,512,512,mla_absorbed,,64,135.988129,32.341,126.58,7.0636,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,2,131072,128,1,512,512,mla_absorbed,,64,21.219135,3.239,6.35,0.3544,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,2,131072,128,1,512,512,mla_absorbed,,64,2143.854248,4.103,8.045,0.4489,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,3,131072,128,1,512,512,mla_absorbed,,64,21.241632,4.853,6.356,0.3547,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,3,131072,128,1,512,512,mla_absorbed,,64,2280.630371,5.785,7.577,0.4228,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,1,4,131072,128,1,512,512,mla_absorbed,,64,21.251936,6.467,6.365,0.3552,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro,generation,cudnn,bfloat16,bfloat16,128,4,131072,128,1,512,512,mla_absorbed,,64,2295.632324,7.663,7.542,0.4209,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,1,512,512,mla_absorbed,,64,3.14368,87.438,86.056,4.8022,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,1,512,512,mla_absorbed,,64,46.745983,94.084,23.149,1.2918,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,64,1,512,512,mla_absorbed,,64,49.011646,89.735,2.738,0.1528,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,64,1,512,512,mla_absorbed,,64,98.141922,89.626,2.051,0.1145,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,1,512,512,mla_absorbed,,64,92.045853,95.562,2.187,0.1221,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,1,512,512,mla_absorbed,,64,185.612961,94.779,1.446,0.0807,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,1,512,512,mla_absorbed,,64,0.624416,27.513,215.159,12.0066,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,1,512,512,mla_absorbed,,64,68.137184,32.273,252.383,14.0839,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,1,512,512,mla_absorbed,,64,12.38368,2.775,10.859,0.606,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,1,512,512,mla_absorbed,,64,1068.296875,4.117,16.113,0.8992,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,1,512,512,mla_absorbed,,64,12.39184,4.159,10.863,0.6062,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,1,512,512,mla_absorbed,,64,1079.147461,6.113,15.966,0.891,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,1,512,512,mla_absorbed,,64,12.387424,5.548,10.877,0.607,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,1,512,512,mla_absorbed,,64,1107.02063,7.946,15.58,0.8694,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,1,512,512,mla_absorbed,,64,1.633952,84.114,83.426,4.6555,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,1,512,512,mla_absorbed,,64,23.560863,93.334,23.143,1.2914,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,32,1,512,512,mla_absorbed,,64,24.69376,89.052,4.076,0.2275,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,32,1,512,512,mla_absorbed,,64,49.302177,89.206,3.403,0.1899,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,1,512,512,mla_absorbed,,64,48.820606,90.086,2.749,0.1534,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,1,512,512,mla_absorbed,,64,97.902367,89.846,2.056,0.1148,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,1,512,512,mla_absorbed,,64,0.329312,26.084,407.769,22.755,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,1,512,512,mla_absorbed,,64,36.571522,30.065,469.99,26.2271,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,1,512,512,mla_absorbed,,64,9.801888,1.753,13.706,0.7649,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,1,512,512,mla_absorbed,,64,533.418335,4.123,32.239,1.799,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,1,512,512,mla_absorbed,,64,9.79392,2.631,13.724,0.7659,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,1,512,512,mla_absorbed,,64,538.994385,6.12,31.921,1.7813,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,1,512,512,mla_absorbed,,64,9.79984,3.506,13.723,0.7658,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,1,512,512,mla_absorbed,,64,542.323608,8.11,31.74,1.7712,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,512,512,mla_absorbed,,64,0.877184,78.341,78.896,4.4027,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,512,512,mla_absorbed,,64,11.992672,91.682,23.083,1.2881,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,512,512,mla_absorbed,,64,16.149696,68.082,5.194,0.2899,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,512,512,mla_absorbed,,64,32.478432,67.707,4.649,0.2594,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,512,512,mla_absorbed,,64,24.427649,90.022,4.121,0.23,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,512,512,mla_absorbed,,64,48.89613,89.947,3.431,0.1915,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,512,512,mla_absorbed,,64,0.2232,19.243,601.481,33.5648,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,512,512,mla_absorbed,,64,18.414944,29.854,933.159,52.0736,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,512,512,mla_absorbed,,64,9.75472,0.881,13.766,0.7682,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,512,512,mla_absorbed,,64,267.116638,4.116,64.347,3.5908,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,512,512,mla_absorbed,,64,9.745216,1.322,13.783,0.7691,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,512,512,mla_absorbed,,64,268.723694,6.137,63.978,3.5702,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,512,512,mla_absorbed,,64,9.747968,1.762,13.782,0.7691,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +deepseek_v4,dsv4_pro-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,512,512,mla_absorbed,,64,270.309692,8.135,63.618,3.5501,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, diff --git a/benchmark/attention_inference/results/deepseek_v4/rtxpro6000/deepseek_v4_context.png b/benchmark/attention_inference/results/deepseek_v4/rtxpro6000/deepseek_v4_context.png new file mode 100644 index 000000000..dac268bf1 Binary files /dev/null and b/benchmark/attention_inference/results/deepseek_v4/rtxpro6000/deepseek_v4_context.png differ diff --git a/benchmark/attention_inference/results/deepseek_v4/rtxpro6000/deepseek_v4_generation.png b/benchmark/attention_inference/results/deepseek_v4/rtxpro6000/deepseek_v4_generation.png new file mode 100644 index 000000000..9ad59afe4 Binary files /dev/null and b/benchmark/attention_inference/results/deepseek_v4/rtxpro6000/deepseek_v4_generation.png differ diff --git a/benchmark/attention_inference/results/gpt_oss/b300/gpt_oss_20260817_134457.csv b/benchmark/attention_inference/results/gpt_oss/b300/gpt_oss_20260817_134457.csv deleted file mode 100644 index b2a2452af..000000000 --- a/benchmark/attention_inference/results/gpt_oss/b300/gpt_oss_20260817_134457.csv +++ /dev/null @@ -1,1849 +0,0 @@ -config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version -gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,128.0,16,0.060544,567.517,623.493,7.7937,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,128.0,16,0.133312,4123.828,1132.643,14.158,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,128.0,16,0.427936,20554.693,1411.379,17.6422,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,128.0,16,0.027808,19769.7,348.796,4.36,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,128.0,16,0.040736,26991.153,238.102,2.9763,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,128.0,16,0.048096,22860.77,397.882,4.9735,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,128.0,16,0.048672,45180.456,393.173,4.9147,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,64,8,64,64,gqa,128.0,16,0.98448,2.181,0.151,0.0019,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,64,8,64,64,gqa,128.0,16,52.131329,5.273,0.365,0.0046,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,128.0,16,0.039616,108.415,7.548,0.0943,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,64,8,64,64,gqa,128.0,16,0.037152,115.605,4.465,0.0558,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,128.0,16,0.306848,1791.623,124.73,1.5591,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,64,8,64,64,gqa,128.0,16,0.502784,1093.423,42.232,0.5279,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,128.0,16,0.040544,158.9,7.83,0.0979,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,64,8,64,64,gqa,128.0,16,0.056672,113.68,3.234,0.0404,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,128.0,16,0.308128,2676.27,131.868,1.6484,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,64,8,64,64,gqa,128.0,16,0.521696,1580.679,44.972,0.5622,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,128.0,16,0.040608,211.533,8.271,0.1034,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,64,8,64,64,gqa,128.0,16,0.056576,151.83,3.548,0.0443,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,128.0,16,0.3088,3560.595,139.222,1.7403,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,64,8,64,64,gqa,128.0,16,0.503488,2183.789,51.024,0.6378,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,128.0,16,0.202016,170.084,186.86,2.3358,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,128.0,16,0.495008,1110.6,305.035,3.8129,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,128.0,16,1.690624,5202.868,357.253,4.4657,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,64,8,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,128.0,16,0.04768,360.316,395.855,4.9482,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,128.0,16,0.084032,3271.11,898.437,11.2305,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,128.0,16,0.233696,18819.52,1292.234,16.1529,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,128.0,16,0.041056,6695.195,118.123,1.4765,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,128.0,16,0.04032,13634.816,120.279,1.5035,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,128.0,16,0.040032,13732.909,239.015,2.9877,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,128.0,16,0.040064,27443.881,238.824,2.9853,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,32,4,64,64,gqa,128.0,16,0.98448,1.091,0.075,0.0009,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,32,4,64,64,gqa,128.0,16,26.106848,5.264,0.364,0.0045,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,128.0,16,0.039744,54.033,3.762,0.047,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,32,4,64,64,gqa,128.0,16,0.05568,38.568,1.49,0.0186,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,128.0,16,0.1696,1620.742,112.833,1.4104,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,32,4,64,64,gqa,128.0,16,0.286464,959.555,37.062,0.4633,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,128.0,16,0.039968,80.595,3.971,0.0496,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,32,4,64,64,gqa,128.0,16,0.056832,56.68,1.613,0.0202,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,128.0,16,0.1696,2431.114,119.789,1.4974,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,32,4,64,64,gqa,128.0,16,0.286656,1438.368,40.923,0.5115,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,128.0,16,0.039456,108.855,4.256,0.0532,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,32,4,64,64,gqa,128.0,16,0.056672,75.786,1.771,0.0221,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,128.0,16,0.169472,3243.933,126.84,1.5855,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,32,4,64,64,gqa,128.0,16,0.288192,1907.603,44.571,0.5571,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,128.0,16,0.169312,101.469,111.477,1.3935,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,128.0,16,0.250528,1097.194,301.353,3.7669,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,128.0,16,0.904192,4864.063,333.989,4.1749,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,32,4,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,128.0,16,0.039104,219.669,241.336,3.0167,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,128.0,16,0.061024,2252.212,618.588,7.7324,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,128.0,16,0.133824,16432.203,1128.31,14.1039,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,128.0,16,0.040128,3425.014,60.427,0.7553,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,128.0,16,0.04048,6790.462,59.902,0.7488,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,128.0,16,0.039328,6989.369,121.647,1.5206,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,128.0,16,0.039264,14001.523,121.845,1.5231,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,16,2,64,64,gqa,128.0,16,0.978656,0.549,0.038,0.0005,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,16,2,64,64,gqa,128.0,16,13.071424,5.257,0.363,0.0045,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,128.0,16,0.039488,27.192,1.893,0.0237,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,16,2,64,64,gqa,128.0,16,0.056288,19.076,0.737,0.0092,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,128.0,16,0.10192,1348.498,93.88,1.1735,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,16,2,64,64,gqa,128.0,16,0.172448,796.988,30.783,0.3848,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,128.0,16,0.03968,40.59,2.0,0.025,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,16,2,64,64,gqa,128.0,16,0.05552,29.01,0.825,0.0103,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,128.0,16,0.103168,1998.279,98.462,1.2308,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,16,2,64,64,gqa,128.0,16,0.169984,1212.811,34.506,0.4313,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,128.0,16,0.039776,53.989,2.111,0.0264,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,16,2,64,64,gqa,128.0,16,0.037568,57.163,1.336,0.0167,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,128.0,16,0.101248,2714.897,106.154,1.3269,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,16,2,64,64,gqa,128.0,16,0.170624,1611.015,37.641,0.4705,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,128.0,16,0.14704,58.419,64.181,0.8023,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,128.0,16,0.206016,667.128,183.232,2.2904,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,128.0,16,0.495872,4434.659,304.504,3.8063,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,16,2,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,128.0,16,0.040224,106.776,117.308,1.4663,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,128.0,16,0.047232,1454.935,399.61,4.9951,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,128.0,16,0.0832,13215.284,907.422,11.3428,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,128.0,16,0.039616,1734.639,30.604,0.3826,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,128.0,16,0.038784,3543.702,31.261,0.3908,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,128.0,16,0.039904,3444.24,59.945,0.7493,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,128.0,16,0.039968,6877.45,59.849,0.7481,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,8,1,64,64,gqa,128.0,16,0.977664,0.275,0.019,0.0002,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,8,1,64,64,gqa,128.0,16,6.60176,5.205,0.36,0.0045,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,128.0,16,0.041088,13.066,0.91,0.0114,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,8,1,64,64,gqa,128.0,16,0.056384,9.522,0.368,0.0046,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,128.0,16,0.067584,1016.801,70.788,0.8848,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,8,1,64,64,gqa,128.0,16,0.111136,618.337,23.883,0.2985,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,128.0,16,0.039264,20.51,1.011,0.0126,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,8,1,64,64,gqa,128.0,16,0.055456,14.522,0.413,0.0052,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,128.0,16,0.067552,1525.924,75.187,0.9398,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,8,1,64,64,gqa,128.0,16,0.111008,928.575,26.419,0.3302,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,128.0,16,0.0392,27.391,1.071,0.0134,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,8,1,64,64,gqa,128.0,16,0.055584,19.317,0.451,0.0056,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,128.0,16,0.068384,2009.812,78.585,0.9823,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,8,1,64,64,gqa,128.0,16,0.111264,1235.251,28.862,0.3608,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,128.0,16,0.124608,34.468,37.867,0.4733,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,128.0,16,0.150272,457.301,125.601,1.57,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,128.0,16,0.26768,4107.56,282.044,3.5255,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,8,1,64,64,gqa,128.0,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,,16,0.07296,470.939,517.389,6.4674,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,,16,0.495264,1110.026,304.878,3.811,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,,16,7.73072,1137.81,78.127,0.9766,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,,16,0.527616,1041.962,270.284,3.3786,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,,16,1.0056,1093.389,275.282,3.441,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,,16,1.016128,1082.06,148.598,1.8575,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,,16,1.992064,1103.892,143.174,1.7897,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,8,64,64,gqa,,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,64,8,64,64,gqa,,16,0.828128,2.593,162.093,2.0262,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,8,64,64,gqa,,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,64,8,64,64,gqa,,16,44.444767,6.185,386.591,4.8324,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,,16,2.502528,1.716,107.279,1.341,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,64,8,64,64,gqa,,16,0.964096,4.455,139.25,1.7406,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,,16,113.020515,4.864,304.05,3.8006,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,64,8,64,64,gqa,,16,51.867264,10.599,331.308,4.1414,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,,16,2.514176,2.562,106.788,1.3349,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,64,8,64,64,gqa,,16,0.98336,6.551,136.539,1.7067,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,,16,113.02211,7.296,304.065,3.8008,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,64,8,64,64,gqa,,16,51.854176,15.903,331.433,4.1429,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,,16,2.51488,3.416,106.765,1.3346,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,64,8,64,64,gqa,,16,0.984448,8.726,136.405,1.7051,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,,16,113.007454,9.73,304.123,3.8015,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,64,8,64,64,gqa,,16,51.857471,21.203,331.452,4.1431,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,,16,0.157888,217.621,239.086,2.9886,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,,16,0.914176,601.368,165.171,2.0646,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,,16,11.125728,790.608,54.287,0.6786,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,,16,0.886304,620.279,160.9,2.0113,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,,16,1.676832,655.708,165.088,2.0636,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,,16,1.517696,724.461,99.49,1.2436,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,,16,2.972544,739.778,95.949,1.1994,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,64,8,64,64,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,64,8,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,64,8,64,64,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,64,8,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,,16,1.679712,2.557,159.83,1.9979,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,64,8,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,,16,187.596542,2.931,183.18,2.2897,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,64,8,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,,16,1.680064,3.835,159.806,1.9976,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,64,8,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,,16,187.692932,4.394,183.097,2.2887,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,64,8,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,,16,1.644096,5.225,163.312,2.0414,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,64,8,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,,16,187.736191,5.857,183.066,2.2883,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,64,8,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,,16,0.054016,318.051,349.422,4.3678,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,,16,0.267264,1028.488,282.483,3.531,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,,16,3.417728,1286.833,88.36,1.1045,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,,16,0.526816,521.772,135.347,1.6918,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,,16,1.014816,541.73,136.391,1.7049,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,,16,0.51504,1067.404,146.586,1.8323,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,,16,1.0136,1084.759,140.693,1.7587,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,4,64,64,gqa,,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,32,4,64,64,gqa,,16,0.824256,1.303,81.427,1.0178,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,4,64,64,gqa,,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,32,4,64,64,gqa,,16,22.243135,6.179,386.231,4.8279,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,,16,2.509824,0.856,53.483,0.6685,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,32,4,64,64,gqa,,16,0.98096,2.189,68.428,0.8554,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,,16,56.675999,4.85,303.161,3.7895,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,32,4,64,64,gqa,,16,25.974655,10.583,330.785,4.1348,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,,16,2.496,1.291,53.783,0.6723,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,32,4,64,64,gqa,,16,0.982944,3.277,68.298,0.8537,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,,16,56.67683,7.275,303.175,3.7897,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,32,4,64,64,gqa,,16,25.969536,15.877,330.891,4.1361,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,,16,2.497152,1.72,53.761,0.672,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,32,4,64,64,gqa,,16,0.982272,4.372,68.353,0.8544,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,,16,56.661537,9.702,303.276,3.7909,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,32,4,64,64,gqa,,16,25.971807,21.167,330.902,4.1363,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,,16,0.180736,95.055,104.431,1.3054,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,,16,0.515904,532.808,146.34,1.8293,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,,16,5.611808,783.713,53.813,0.6727,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,,16,0.76,361.681,93.82,1.1727,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,,16,1.483008,370.703,93.332,1.1666,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,,16,0.802528,685.03,94.075,1.1759,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,,16,1.518816,723.927,93.893,1.1737,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,4,64,64,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,32,4,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,4,64,64,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,32,4,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,,16,1.519712,1.413,88.329,1.1041,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,32,4,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,,16,93.078461,2.953,184.597,2.3075,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,32,4,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,,16,1.51632,2.124,88.532,1.1066,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,32,4,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,,16,93.424416,4.413,183.924,2.2991,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,32,4,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,,16,1.478784,2.904,90.784,1.1348,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,32,4,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,,16,93.329117,5.891,184.123,2.3015,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,32,4,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,,16,0.049728,172.738,189.776,2.3722,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,,16,0.1536,894.785,245.76,3.072,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,,16,1.731776,1269.808,87.191,1.0899,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,,16,0.525536,261.521,67.839,0.848,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,,16,1.013568,271.198,68.28,0.8535,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,,16,0.527104,521.487,71.615,0.8952,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,,16,1.000896,549.264,71.239,0.8905,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,2,64,64,gqa,,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,16,2,64,64,gqa,,16,0.841344,0.638,39.887,0.4986,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,2,64,64,gqa,,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,16,2,64,64,gqa,,16,11.163904,6.156,384.766,4.8096,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,,16,2.334688,0.46,28.748,0.3593,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,16,2,64,64,gqa,,16,0.973248,1.103,34.485,0.4311,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,,16,28.544832,4.815,300.965,3.7621,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,16,2,64,64,gqa,,16,13.019904,10.556,329.958,4.1245,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,,16,2.334112,0.69,28.757,0.3595,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,16,2,64,64,gqa,,16,0.97312,1.655,34.494,0.4312,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,,16,28.536863,7.224,301.067,3.7633,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,16,2,64,64,gqa,,16,13.019872,15.834,329.999,4.125,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,,16,2.335424,0.92,28.742,0.3593,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,16,2,64,64,gqa,,16,0.9728,2.208,34.509,0.4314,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,,16,28.535872,9.633,301.096,3.7637,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,16,2,64,64,gqa,,16,13.021408,21.11,330.0,4.125,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,,16,0.152384,56.37,61.93,0.7741,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,,16,0.373376,368.098,101.101,1.2638,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,,16,2.806208,783.628,53.807,0.6726,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,,16,0.75632,181.721,47.138,0.5892,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,,16,1.425824,192.785,48.538,0.6067,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,,16,0.757536,362.858,49.831,0.6229,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,,16,1.392256,394.867,51.214,0.6402,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,2,64,64,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,16,2,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,2,64,64,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,16,2,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,,16,1.423136,0.754,47.161,0.5895,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,16,2,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,,16,46.487904,2.956,184.8,2.31,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,16,2,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,,16,1.387168,1.161,48.387,0.6048,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,16,2,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,,16,46.308128,4.452,185.529,2.3191,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,16,2,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,,16,1.424448,1.508,47.124,0.589,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,16,2,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,,16,46.523457,5.908,184.682,2.3085,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,16,2,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,,16,0.037312,115.11,126.463,1.5808,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,,16,0.095584,718.943,197.464,2.4683,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,,16,0.890848,1234.23,84.748,1.0593,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,,16,0.521056,131.885,34.211,0.4276,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,,16,1.008256,136.314,34.32,0.429,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,,16,0.524832,261.872,35.963,0.4495,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,,16,1.013312,271.267,35.183,0.4398,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,64,64,gqa,,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,8,1,64,64,gqa,,16,0.839168,0.32,19.995,0.2499,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,64,64,gqa,,16,inf,0.0,0.0,,20,False,"Traceback (most recent call last): - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 744, in - main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,8,1,64,64,gqa,,16,5.605696,6.129,383.136,4.7892,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,,16,2.313696,0.232,14.504,0.1813,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,8,1,64,64,gqa,,16,0.972256,0.552,17.26,0.2158,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,,16,14.447552,4.756,297.316,3.7165,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,8,1,64,64,gqa,,16,6.588832,10.43,326.007,4.0751,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,,16,2.31296,0.348,14.51,0.1814,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,8,1,64,64,gqa,,16,0.971488,0.829,17.276,0.2159,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,,16,14.4304,7.143,297.688,3.7211,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,8,1,64,64,gqa,,16,6.590368,15.641,325.971,4.0746,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,,16,2.326656,0.461,14.425,0.1803,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,8,1,64,64,gqa,,16,0.971872,1.105,17.271,0.2159,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,,16,14.380608,9.557,298.737,3.7342,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,8,1,64,64,gqa,,16,6.592512,20.848,325.905,4.0738,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,,16,0.129184,33.247,36.526,0.4566,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,,16,0.270272,254.26,69.835,0.8729,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,,16,1.538528,714.652,49.071,0.6134,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,,16,0.730144,94.118,24.414,0.3052,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,,16,1.351872,101.666,25.596,0.32,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,,16,0.737376,186.389,25.597,0.32,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,,16,1.361792,201.85,26.18,0.3272,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,8,1,64,64,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,8,1,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,8,1,64,64,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 169, in setup_cudnn - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: decode only mode, i.e. s_q == 1, not supported with sink_token",,, -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,8,1,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,,16,1.302688,0.412,25.761,0.322,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,8,1,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,,16,18.027937,3.812,238.269,2.9784,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,8,1,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,,16,1.30256,0.618,25.765,0.3221,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,8,1,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,,16,18.028929,5.717,238.27,2.9784,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,8,1,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,,16,1.35104,0.795,24.842,0.3105,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,8,1,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,,16,18.064417,7.608,237.816,2.9727,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,8,1,64,64,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, diff --git a/benchmark/attention_inference/results/gpt_oss/b300/gpt_oss_context.png b/benchmark/attention_inference/results/gpt_oss/b300/gpt_oss_context.png deleted file mode 100644 index 78e86a9e6..000000000 Binary files a/benchmark/attention_inference/results/gpt_oss/b300/gpt_oss_context.png and /dev/null differ diff --git a/benchmark/attention_inference/results/gpt_oss/b300/gpt_oss_generation.png b/benchmark/attention_inference/results/gpt_oss/b300/gpt_oss_generation.png deleted file mode 100644 index 2b3d3667a..000000000 Binary files a/benchmark/attention_inference/results/gpt_oss/b300/gpt_oss_generation.png and /dev/null differ diff --git a/benchmark/attention_inference/results/gpt_oss/gb200/gpt_oss_20260818_125939.csv b/benchmark/attention_inference/results/gpt_oss/gb200/gpt_oss_20260818_125939.csv new file mode 100644 index 000000000..c033e94ab --- /dev/null +++ b/benchmark/attention_inference/results/gpt_oss/gb200/gpt_oss_20260818_125939.csv @@ -0,0 +1,209 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,128.0,16,0.062464,550.073,604.328,7.5541,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,128.0,16,0.139648,3936.725,1081.254,13.5157,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,128.0,16,0.442176,19892.741,1365.926,17.0741,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,128.0,16,0.043552,12622.975,222.707,2.7838,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,128.0,16,0.042656,25776.247,227.385,2.8423,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,128.0,16,0.051424,21381.293,372.132,4.6516,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,128.0,16,0.049792,44164.189,384.329,4.8041,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,128.0,16,0.040064,107.203,7.463,0.0933,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,128.0,16,0.295264,1861.913,129.623,1.6203,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,128.0,16,0.038496,167.354,8.246,0.1031,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,128.0,16,0.29616,2784.42,137.197,1.715,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,128.0,16,0.037248,230.615,9.017,0.1127,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,128.0,16,0.296448,3708.953,145.022,1.8128,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,128.0,16,0.23952,143.452,157.602,1.97,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,128.0,16,0.514496,1068.533,293.481,3.6685,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,128.0,16,1.83696,4788.397,328.793,4.1099,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,128.0,16,0.267264,2056.977,36.291,0.4536,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,128.0,16,0.41856,2626.891,23.173,0.2897,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,128.0,16,0.288512,3810.974,66.328,0.8291,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,128.0,16,0.444352,4948.832,43.066,0.5383,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,128.0,16,0.407936,10.529,0.733,0.0092,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,128.0,16,41.985344,13.094,0.912,0.0114,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,128.0,16,0.409152,15.746,0.776,0.0097,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,128.0,16,42.022335,19.624,0.967,0.0121,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,128.0,16,0.4096,20.972,0.82,0.0103,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,128.0,16,41.964993,26.201,1.024,0.0128,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,128.0,16,0.050176,342.392,376.163,4.702,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,128.0,16,0.08784,3129.302,859.489,10.7436,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,128.0,16,0.240896,18257.034,1253.611,15.6701,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,128.0,16,0.045664,6019.576,106.203,1.3275,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,128.0,16,0.046752,11758.98,103.732,1.2966,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,128.0,16,0.044608,12324.153,214.496,2.6812,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,128.0,16,0.047648,23075.713,200.811,2.5101,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,128.0,16,0.039328,54.604,3.801,0.0475,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,128.0,16,0.162528,1691.265,117.743,1.4718,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,128.0,16,0.037888,85.02,4.189,0.0524,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,128.0,16,0.163104,2527.938,124.56,1.557,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,128.0,16,0.038176,112.504,4.399,0.055,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,128.0,16,0.163296,3366.621,131.637,1.6455,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,128.0,16,0.181152,94.837,104.191,1.3024,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,128.0,16,0.319168,861.233,236.545,2.9568,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,128.0,16,0.96272,4568.355,313.684,3.9211,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,128.0,16,0.185376,1482.813,26.161,0.327,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,128.0,16,0.263776,2084.177,18.386,0.2298,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,128.0,16,0.18736,2934.222,51.069,0.6384,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,128.0,16,0.267776,4106.087,35.732,0.4467,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,128.0,16,0.2512,8.549,0.595,0.0074,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,128.0,16,21.58256,12.736,0.887,0.0111,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,128.0,16,0.252992,12.733,0.627,0.0078,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,128.0,16,21.291872,19.365,0.954,0.0119,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,128.0,16,0.25392,16.915,0.661,0.0083,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,128.0,16,21.286272,25.827,1.01,0.0126,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,128.0,16,0.04352,197.379,216.847,2.7106,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,128.0,16,0.062816,2187.961,600.941,7.5118,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,128.0,16,0.139584,15754.121,1081.75,13.5219,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,128.0,16,0.047776,2876.736,50.754,0.6344,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,128.0,16,0.043168,6367.631,56.172,0.7021,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,128.0,16,0.0448,6135.668,106.789,1.3349,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,128.0,16,0.044,12494.45,108.73,1.3591,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,128.0,16,0.039072,27.481,1.913,0.0239,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,128.0,16,0.0984,1396.737,97.238,1.2155,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,128.0,16,0.039008,41.289,2.034,0.0254,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,128.0,16,0.098656,2089.669,102.965,1.2871,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,128.0,16,0.042496,50.534,1.976,0.0247,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,128.0,16,0.097792,2810.842,109.906,1.3738,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,128.0,16,0.167296,51.346,56.41,0.7051,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,128.0,16,0.23136,594.048,163.16,2.0395,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,128.0,16,0.515168,4268.556,293.098,3.6637,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,128.0,16,0.158624,866.445,15.287,0.1911,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,128.0,16,0.1864,1474.667,13.009,0.1626,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,128.0,16,0.162944,1686.947,29.361,0.367,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,128.0,16,0.188416,2917.777,25.391,0.3174,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,128.0,16,0.177856,6.037,0.42,0.0053,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,128.0,16,10.816224,12.707,0.885,0.0111,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,128.0,16,0.175616,9.171,0.452,0.0056,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,128.0,16,10.815456,19.061,0.939,0.0117,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,128.0,16,0.186752,11.499,0.45,0.0056,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,128.0,16,10.8144,25.418,0.994,0.0124,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,128.0,16,0.03984,107.805,118.439,1.4805,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,128.0,16,0.050048,1373.071,377.125,4.7141,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,128.0,16,0.085664,12835.166,881.321,11.0165,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,128.0,16,0.041696,1648.107,29.078,0.3635,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,128.0,16,0.045888,2995.096,26.421,0.3303,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,128.0,16,0.044416,3094.357,53.856,0.6732,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,128.0,16,0.04272,6434.408,55.994,0.6999,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,128.0,16,0.039904,13.454,0.937,0.0117,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,128.0,16,0.065696,1046.022,72.822,0.9103,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,128.0,16,0.040096,20.084,0.99,0.0124,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,128.0,16,0.064928,1587.593,78.226,0.9778,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,128.0,16,0.040288,26.652,1.042,0.013,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,128.0,16,0.065088,2111.587,82.564,1.0321,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,128.0,16,0.141568,30.339,33.331,0.4166,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,128.0,16,0.160672,427.7,117.471,1.4684,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,128.0,16,0.290304,3787.449,260.063,3.2508,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,128.0,16,0.13792,498.256,8.791,0.1099,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,128.0,16,0.132832,1034.683,9.127,0.1141,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,128.0,16,0.13312,1032.444,17.969,0.2246,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,128.0,16,0.134048,2050.593,17.845,0.2231,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,128.0,16,0.1344,3.995,0.278,0.0035,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,128.0,16,0.258976,265.351,18.473,0.2309,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,128.0,16,0.137024,5.877,0.29,0.0036,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,128.0,16,0.26192,393.552,19.392,0.2424,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,128.0,16,0.13584,7.904,0.309,0.0039,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,128.0,16,0.257152,534.466,20.898,0.2612,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,,16,0.08192,419.43,460.8,5.76,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,,16,0.59104,930.15,255.473,3.1934,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,,16,8.57888,1025.319,70.403,0.88,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,,16,0.658208,835.231,216.658,2.7082,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,,16,1.27728,860.823,216.729,2.7091,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,,16,1.271904,864.461,118.716,1.4839,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,,16,2.495968,881.03,114.269,1.4284,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,,16,2.466272,1.741,108.856,1.3607,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,,16,101.962494,5.392,337.025,4.2128,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,,16,2.46528,2.613,108.906,1.3613,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,,16,101.985474,8.086,336.97,4.2121,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,,16,2.464416,3.486,108.951,1.3619,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,,16,101.970078,10.783,337.041,4.213,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,,16,0.210016,163.605,179.742,2.2468,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,,16,0.972512,565.295,155.263,1.9408,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,,16,11.731904,749.758,51.482,0.6435,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,,16,1.009952,544.339,141.201,1.765,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,,16,1.930976,569.407,143.36,1.792,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,,16,1.781824,617.071,84.742,1.0593,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,,16,3.469504,633.815,82.206,1.0276,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,,16,1.927392,2.228,139.291,1.7411,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,,16,206.710846,2.66,166.242,2.078,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,,16,1.921888,3.352,139.698,1.7462,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,,16,206.777374,3.988,166.198,2.0775,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,,16,1.921504,4.47,139.735,1.7467,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,,16,206.741058,5.318,166.238,2.078,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,,16,0.0592,290.2,318.824,3.9853,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,,16,0.315712,870.66,239.134,2.9892,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,,16,4.320448,1017.961,69.898,0.8737,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,,16,0.6552,419.533,108.827,1.3603,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,,16,1.267072,433.879,109.238,1.3655,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,,16,0.653728,840.955,115.488,1.4436,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,,16,1.269088,866.379,112.369,1.4046,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,,16,2.469504,0.87,54.357,0.6795,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,,16,52.818783,5.204,325.3,4.0663,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,,16,2.475552,1.301,54.227,0.6778,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,,16,52.835648,7.804,325.216,4.0652,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,,16,2.469344,1.739,54.367,0.6796,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,,16,52.817986,10.408,325.345,4.0668,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,,16,0.186304,92.214,101.31,1.2664,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,,16,0.546592,502.894,138.124,1.7265,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,,16,5.79552,758.87,52.107,0.6513,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,,16,0.930624,295.369,76.619,0.9577,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,,16,1.77008,310.582,78.195,0.9774,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,,16,0.931104,590.434,81.084,1.0135,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,,16,1.772704,620.245,80.446,1.0056,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,,16,1.7728,1.211,75.719,0.9465,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,,16,103.855743,2.647,165.441,2.068,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,,16,1.769568,1.82,75.862,0.9483,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,,16,103.597603,3.98,165.863,2.0733,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,,16,1.770016,2.427,75.847,0.9481,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,,16,103.553535,5.309,165.944,2.0743,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,,16,0.054752,156.888,172.362,2.1545,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,,16,0.179904,763.957,209.827,2.6228,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,,16,2.103136,1045.592,71.795,0.8974,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,,16,0.65424,210.074,54.493,0.6812,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,,16,1.27296,215.936,54.366,0.6796,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,,16,0.651424,421.965,57.948,0.7244,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,,16,1.27168,432.307,56.07,0.7009,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,,16,2.303712,0.466,29.134,0.3642,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,,16,26.43552,5.199,324.979,4.0622,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,,16,2.304096,0.699,29.131,0.3641,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,,16,26.43792,7.798,324.969,4.0621,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,,16,2.306752,0.931,29.099,0.3637,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,,16,26.434015,10.399,325.037,4.063,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,,16,0.168224,51.062,56.099,0.7012,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,,16,0.344448,399.012,109.592,1.3699,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,,16,2.935744,749.051,51.433,0.6429,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,,16,0.90384,152.061,39.445,0.4931,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,,16,1.695456,162.126,40.819,0.5102,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,,16,0.906688,303.167,41.634,0.5204,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,,16,1.69376,324.577,42.098,0.5262,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,,16,1.693696,0.634,39.628,0.4953,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,,16,51.938305,2.646,165.407,2.0676,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,,16,1.69728,0.949,39.546,0.4943,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,,16,51.927746,3.97,165.451,2.0681,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,,16,1.69552,1.267,39.59,0.4949,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,,16,51.941666,5.292,165.417,2.0677,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,,16,0.057408,74.815,82.194,1.0274,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,,16,0.115104,597.021,163.977,2.0497,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,,16,1.076896,1021.001,70.107,0.8763,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,,16,0.652832,105.264,27.305,0.3413,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,,16,1.26032,109.051,27.456,0.3432,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,,16,0.652416,210.662,28.93,0.3616,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,,16,1.267136,216.928,28.136,0.3517,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,,16,2.303872,0.233,14.566,0.1821,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,,16,15.111904,4.547,284.246,3.5531,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,,16,2.30272,0.35,14.574,0.1822,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,,16,15.11232,6.821,284.255,3.5532,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,,16,2.303072,0.466,14.573,0.1822,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,,16,15.112,9.095,284.278,3.5535,20,True,,NVIDIA GB200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,,16,0.135712,31.648,34.769,0.4346,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,,16,0.259904,264.403,72.621,0.9078,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,,16,1.498912,733.54,50.368,0.6296,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,,16,0.875232,78.516,20.367,0.2546,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,,16,1.633664,84.129,21.181,0.2648,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,,16,0.876864,156.739,21.525,0.2691,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,,16,1.639456,167.664,21.746,0.2718,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,,16,1.633728,0.329,20.541,0.2568,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,,16,21.57424,3.185,199.103,2.4888,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,,16,1.633664,0.493,20.543,0.2568,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,,16,21.559649,4.781,199.25,2.4906,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,,16,1.633792,0.657,20.543,0.2568,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,,16,21.560064,6.375,199.258,2.4907,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], diff --git a/benchmark/attention_inference/results/gpt_oss/gb200/gpt_oss_context.png b/benchmark/attention_inference/results/gpt_oss/gb200/gpt_oss_context.png new file mode 100644 index 000000000..759255621 Binary files /dev/null and b/benchmark/attention_inference/results/gpt_oss/gb200/gpt_oss_context.png differ diff --git a/benchmark/attention_inference/results/gpt_oss/gb200/gpt_oss_generation.png b/benchmark/attention_inference/results/gpt_oss/gb200/gpt_oss_generation.png new file mode 100644 index 000000000..1a29f2bc8 Binary files /dev/null and b/benchmark/attention_inference/results/gpt_oss/gb200/gpt_oss_generation.png differ diff --git a/benchmark/attention_inference/results/gpt_oss/gb300/gpt_oss_20260818_130614.csv b/benchmark/attention_inference/results/gpt_oss/gb300/gpt_oss_20260818_130614.csv new file mode 100644 index 000000000..7a8004e7a --- /dev/null +++ b/benchmark/attention_inference/results/gpt_oss/gb300/gpt_oss_20260818_130614.csv @@ -0,0 +1,209 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,128.0,16,0.059744,575.116,631.841,7.898,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,128.0,16,0.12848,4278.921,1175.241,14.6905,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,128.0,16,0.4096,21474.837,1474.56,18.432,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,128.0,16,0.04288,12820.798,226.197,2.8275,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,128.0,16,0.045376,24231.128,213.755,2.6719,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,128.0,16,0.047552,23122.3,402.433,5.0304,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,128.0,16,0.047392,46400.728,403.792,5.0474,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,128.0,16,0.033568,127.948,8.908,0.1113,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,128.0,16,0.293056,1875.941,130.6,1.6325,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,128.0,16,0.038784,166.111,8.185,0.1023,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,128.0,16,0.294368,2801.37,138.032,1.7254,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,128.0,16,0.039136,219.489,8.582,0.1073,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,128.0,16,0.293088,3751.473,146.685,1.8336,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,128.0,16,0.226976,151.38,166.312,2.0789,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,128.0,16,0.5128,1072.067,294.452,3.6806,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,128.0,16,1.836608,4789.314,328.856,4.1107,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,128.0,16,0.263296,2087.976,36.838,0.4605,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,128.0,16,0.417952,2630.713,23.207,0.2901,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,128.0,16,0.290688,3782.446,65.832,0.8229,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,128.0,16,0.444736,4944.559,43.029,0.5379,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,128.0,16,0.407424,10.542,0.734,0.0092,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,128.0,16,41.420544,13.273,0.924,0.0116,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,128.0,16,0.398528,16.166,0.797,0.01,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,128.0,16,41.404896,19.916,0.981,0.0123,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,128.0,16,0.401664,21.386,0.836,0.0105,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,128.0,16,41.411903,26.551,1.038,0.013,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,128.0,16,0.047616,360.8,396.387,4.9548,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,128.0,16,0.0832,3303.821,907.422,11.3428,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,128.0,16,0.225312,19519.806,1340.319,16.754,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,128.0,16,0.043296,6348.806,112.012,1.4001,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,128.0,16,0.043552,12622.975,111.353,1.3919,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,128.0,16,0.045504,12081.483,210.273,2.6284,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,128.0,16,0.042592,25814.98,224.649,2.8081,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,128.0,16,0.038624,55.6,3.871,0.0484,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,128.0,16,0.15872,1731.842,120.568,1.5071,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,128.0,16,0.0376,85.671,4.221,0.0528,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,128.0,16,0.160864,2563.139,126.294,1.5787,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,128.0,16,0.035232,121.905,4.767,0.0596,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,128.0,16,0.158272,3473.488,135.816,1.6977,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,128.0,16,0.191552,89.688,98.534,1.2317,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,128.0,16,0.30784,892.925,245.249,3.0656,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,128.0,16,0.959456,4583.896,314.751,3.9344,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,128.0,16,0.18704,1469.621,25.928,0.3241,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,128.0,16,0.263424,2086.962,18.41,0.2301,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,128.0,16,0.183872,2989.883,52.038,0.6505,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,128.0,16,0.273344,4022.446,35.004,0.4376,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,128.0,16,0.255424,8.408,0.585,0.0073,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,128.0,16,21.23296,12.946,0.901,0.0113,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,128.0,16,0.252224,12.771,0.629,0.0079,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,128.0,16,20.967903,19.664,0.969,0.0121,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,128.0,16,0.24592,17.465,0.683,0.0085,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,128.0,16,20.955999,26.234,1.026,0.0128,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,128.0,16,0.047424,181.131,198.996,2.4874,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,128.0,16,0.060128,2285.773,627.806,7.8476,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,128.0,16,0.127808,17205.677,1181.42,14.7678,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,128.0,16,0.035072,3918.766,69.139,0.8642,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,128.0,16,0.04304,6386.568,56.339,0.7042,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,128.0,16,0.045152,6087.834,105.956,1.3245,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,128.0,16,0.043488,12641.552,110.01,1.3751,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,128.0,16,0.038272,28.056,1.953,0.0244,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,128.0,16,0.096352,1426.425,99.305,1.2413,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,128.0,16,0.037152,43.352,2.136,0.0267,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,128.0,16,0.097728,2109.512,103.942,1.2993,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,128.0,16,0.037408,57.407,2.245,0.0281,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,128.0,16,0.100384,2738.264,107.068,1.3383,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,128.0,16,0.16032,53.58,58.865,0.7358,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,128.0,16,0.233344,588.997,161.773,2.0222,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,128.0,16,0.511776,4296.847,295.041,3.688,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,128.0,16,0.165536,830.266,14.648,0.1831,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,128.0,16,0.18928,1452.229,12.811,0.1601,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,128.0,16,0.162816,1688.273,29.384,0.3673,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,128.0,16,0.191488,2870.967,24.984,0.3123,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,128.0,16,0.183616,5.848,0.407,0.0051,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,128.0,16,10.665952,12.886,0.897,0.0112,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,128.0,16,0.178848,9.005,0.444,0.0055,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,128.0,16,10.66912,19.323,0.952,0.0119,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,128.0,16,0.18368,11.691,0.457,0.0057,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,128.0,16,10.65376,25.801,1.009,0.0126,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,128.0,16,0.043264,99.273,109.065,1.3633,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,128.0,16,0.047424,1449.044,397.992,4.9749,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,128.0,16,0.083328,13194.984,906.028,11.3253,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,128.0,16,0.043648,1574.401,27.777,0.3472,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,128.0,16,0.033696,4078.791,35.981,0.4498,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,128.0,16,0.041888,3281.106,57.106,0.7138,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,128.0,16,0.0416,6607.642,57.502,0.7188,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,128.0,16,0.038432,13.969,0.973,0.0122,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,128.0,16,0.063904,1075.355,74.864,0.9358,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,128.0,16,0.038816,20.747,1.022,0.0128,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,128.0,16,0.064064,1609.004,79.281,0.991,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,128.0,16,0.040448,26.546,1.038,0.013,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,128.0,16,0.063712,2157.191,84.348,1.0543,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,128.0,16,0.141856,30.277,33.263,0.4158,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,128.0,16,0.14992,458.374,125.896,1.5737,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,128.0,16,0.293216,3749.835,257.481,3.2185,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,128.0,16,0.134368,511.427,9.023,0.1128,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,128.0,16,0.1424,965.161,8.514,0.1064,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,128.0,16,0.133088,1032.692,17.974,0.2247,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,128.0,16,0.142848,1924.269,16.746,0.2093,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,128.0,16,0.138592,3.874,0.27,0.0034,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,128.0,16,0.259008,265.318,18.471,0.2309,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,128.0,16,0.129824,6.203,0.306,0.0038,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,128.0,16,0.264672,389.46,19.19,0.2399,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,128.0,16,0.137632,7.802,0.305,0.0038,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,128.0,16,0.257696,533.338,20.854,0.2607,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,,16,0.071104,483.232,530.895,6.6362,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,,16,0.470368,1168.778,321.014,4.0127,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,,16,6.593824,1333.99,91.598,1.145,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,,16,0.515264,1066.94,276.764,3.4595,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,,16,0.99712,1102.687,277.624,3.4703,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,,16,0.99584,1104.105,151.626,1.8953,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,,16,1.945056,1130.571,146.635,1.8329,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,,16,2.46496,1.742,108.914,1.3614,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,,16,106.212608,5.176,323.539,4.0442,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,,16,2.462816,2.616,109.015,1.3627,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,,16,106.210114,7.764,323.566,4.0446,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,,16,2.464416,3.486,108.951,1.3619,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,,16,106.218941,10.351,323.559,4.0445,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,,16,0.21728,158.136,173.733,2.1717,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,,16,0.848896,647.613,177.872,2.2234,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,,16,9.750272,902.138,61.945,0.7743,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,,16,0.882624,622.865,161.571,2.0196,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,,16,1.679104,654.82,164.864,2.0608,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,,16,1.534144,716.694,98.423,1.2303,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,,16,2.95872,743.235,96.397,1.205,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,,16,1.667296,2.576,161.02,2.0128,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,,16,177.047256,3.105,194.095,2.4262,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,,16,1.66256,3.875,161.489,2.0186,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,,16,177.063965,4.657,194.088,2.4261,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,,16,1.65664,5.185,162.076,2.0259,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,,16,177.049347,6.21,194.116,2.4265,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,,16,0.053536,320.903,352.555,4.4069,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,,16,0.253088,1086.096,298.305,3.7288,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,,16,3.24912,1353.612,92.945,1.1618,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,,16,0.514144,534.632,138.683,1.7335,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,,16,0.992576,553.868,139.447,1.7431,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,,16,0.509984,1077.986,148.039,1.8505,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,,16,0.991296,1109.166,143.858,1.7982,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,,16,2.462624,0.872,54.509,0.6814,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,,16,53.259521,5.161,322.608,4.0326,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,,16,2.461088,1.309,54.546,0.6818,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,,16,53.265217,7.741,322.594,4.0324,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,,16,2.460096,1.746,54.571,0.6821,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,,16,53.256321,10.323,322.667,4.0333,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,,16,0.196448,87.453,96.078,1.201,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,,16,0.491104,559.714,153.73,1.9216,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,,16,4.929792,892.136,61.258,0.7657,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,,16,0.79696,344.908,89.469,1.1184,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,,16,1.518432,362.055,91.155,1.1394,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,,16,0.808448,680.014,93.386,1.1673,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,,16,1.520448,723.15,93.792,1.1724,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,,16,1.51792,1.415,88.433,1.1054,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,,16,89.439072,3.073,192.108,2.4014,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,,16,1.516224,2.125,88.537,1.1067,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,,16,89.160416,4.624,192.72,2.409,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,,16,1.5144,2.836,88.649,1.1081,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,,16,89.194496,6.164,192.658,2.4082,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,,16,0.0496,173.184,190.266,2.3783,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,,16,0.146144,940.435,258.298,3.2287,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,,16,1.64752,1334.748,91.65,1.1456,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,,16,0.513632,267.583,69.411,0.8676,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,,16,0.991424,277.256,69.805,0.8726,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,,16,0.5152,533.536,73.27,0.9159,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,,16,0.988896,555.929,72.104,0.9013,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,,16,2.264512,0.474,29.639,0.3705,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,,16,27.704224,4.961,310.097,3.8762,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,,16,2.267456,0.71,29.602,0.37,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,,16,27.712992,7.439,310.017,3.8752,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,,16,2.267488,0.947,29.603,0.37,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,,16,27.707872,9.921,310.094,3.8762,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,,16,0.176928,48.55,53.339,0.6667,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,,16,0.31664,434.054,119.217,1.4902,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,,16,2.458848,894.331,61.409,0.7676,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,,16,0.787136,174.606,45.293,0.5662,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,,16,1.44288,190.506,47.964,0.5995,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,,16,0.783552,350.81,48.176,0.6022,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,,16,1.45568,377.663,48.983,0.6123,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,,16,1.437472,0.747,46.691,0.5836,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,,16,44.908928,3.06,191.298,2.3912,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,,16,1.438848,1.119,46.649,0.5831,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,,16,44.887711,4.593,191.4,2.3925,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,,16,1.428736,1.503,46.982,0.5873,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,,16,44.894943,6.123,191.381,2.3923,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,,16,0.049568,86.648,95.194,1.1899,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,,16,0.0936,734.182,201.649,2.5206,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,,16,0.848256,1296.203,89.003,1.1125,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,,16,0.514016,133.691,34.679,0.4335,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,,16,0.982112,139.942,35.233,0.4404,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,,16,0.51168,268.603,36.887,0.4611,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,,16,0.98592,278.803,36.161,0.452,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,,16,2.26464,0.237,14.818,0.1852,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,,16,13.999424,4.909,306.833,3.8354,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,,16,2.26752,0.355,14.801,0.185,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,,16,13.997856,7.364,306.887,3.8361,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,,16,2.265568,0.474,14.814,0.1852,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,,16,13.999168,9.818,306.877,3.836,20,True,,NVIDIA GB300,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,,16,0.153472,27.985,30.746,0.3843,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,,16,0.22592,304.176,83.544,1.0443,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,,16,1.277792,860.478,59.084,0.7386,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,,16,0.753824,91.161,23.647,0.2956,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,,16,1.389376,98.921,24.905,0.3113,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,,16,0.748512,183.616,25.216,0.3152,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,,16,1.385856,198.345,25.725,0.3216,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,,16,1.39088,0.386,24.128,0.3016,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,,16,17.866049,3.846,240.428,3.0053,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,,16,1.386432,0.581,24.206,0.3026,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,,16,17.844448,5.777,240.733,3.0092,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,,16,1.385056,0.775,24.232,0.3029,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,,16,17.85696,7.697,240.579,3.0072,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], diff --git a/benchmark/attention_inference/results/gpt_oss/gb300/gpt_oss_context.png b/benchmark/attention_inference/results/gpt_oss/gb300/gpt_oss_context.png new file mode 100644 index 000000000..4d278947e Binary files /dev/null and b/benchmark/attention_inference/results/gpt_oss/gb300/gpt_oss_context.png differ diff --git a/benchmark/attention_inference/results/gpt_oss/gb300/gpt_oss_generation.png b/benchmark/attention_inference/results/gpt_oss/gb300/gpt_oss_generation.png new file mode 100644 index 000000000..4044a883a Binary files /dev/null and b/benchmark/attention_inference/results/gpt_oss/gb300/gpt_oss_generation.png differ diff --git a/benchmark/attention_inference/results/gpt_oss/h200/gpt_oss_20260818_140856.csv b/benchmark/attention_inference/results/gpt_oss/h200/gpt_oss_20260818_140856.csv new file mode 100644 index 000000000..6e57fed69 --- /dev/null +++ b/benchmark/attention_inference/results/gpt_oss/h200/gpt_oss_20260818_140856.csv @@ -0,0 +1,105 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,128.0,16,0.064736,530.767,583.118,12.1483,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,128.0,16,0.163136,3369.923,925.577,19.2829,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,128.0,16,0.54912,16018.526,1099.905,22.9147,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,128.0,16,0.046016,11947.058,210.782,4.3913,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,128.0,16,0.043872,25061.808,221.082,4.6059,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,128.0,16,0.051296,21434.647,373.061,7.7721,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,128.0,16,0.051392,42789.213,372.364,7.7576,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,128.0,16,0.043232,99.347,6.916,0.1441,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,128.0,16,0.355744,1545.369,107.586,2.2414,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,128.0,16,0.045056,142.988,7.045,0.1468,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,128.0,16,0.35856,2299.849,113.321,2.3609,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,128.0,16,0.04416,194.518,7.606,0.1585,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,128.0,16,0.358112,3070.301,120.051,2.5011,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,128.0,16,0.048512,354.136,389.066,8.1055,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,128.0,16,0.099136,2772.735,761.555,15.8657,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,128.0,16,0.290848,15121.461,1038.308,21.6314,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,128.0,16,0.0424,6482.97,114.379,2.3829,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,128.0,16,0.043104,12754.172,112.511,2.344,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,128.0,16,0.044544,12341.86,214.805,4.4751,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,128.0,16,0.043424,25320.368,220.345,4.5905,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,128.0,16,0.044096,48.7,3.39,0.0706,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,128.0,16,0.200992,1367.606,95.21,1.9835,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,128.0,16,0.04336,74.29,3.661,0.0763,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,128.0,16,0.206656,1995.185,98.309,2.0481,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,128.0,16,0.043584,98.545,3.853,0.0803,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,128.0,16,0.199872,2750.539,107.548,2.2406,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,128.0,16,0.046496,184.746,202.968,4.2285,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,128.0,16,0.064864,2118.879,581.967,12.1243,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,128.0,16,0.16288,13500.879,927.032,19.3132,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,128.0,16,0.0416,3303.821,58.289,1.2144,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,128.0,16,0.045728,6011.151,53.027,1.1047,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,128.0,16,0.043776,6279.192,109.287,2.2768,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,128.0,16,0.043968,12503.544,108.809,2.2669,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,128.0,16,0.043808,24.51,1.706,0.0355,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,128.0,16,0.119104,1153.941,80.335,1.6737,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,128.0,16,0.043744,36.819,1.814,0.0378,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,128.0,16,0.120288,1713.874,84.448,1.7593,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,128.0,16,0.042464,50.572,1.977,0.0412,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,128.0,16,0.118816,2313.475,90.458,1.8845,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,128.0,16,0.041472,103.563,113.778,2.3704,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,128.0,16,0.048576,1414.68,388.553,8.0949,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,128.0,16,0.0984,11173.899,767.251,15.9844,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,128.0,16,0.0432,1590.729,28.065,0.5847,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,128.0,16,0.044416,3094.357,27.297,0.5687,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,128.0,16,0.044096,3116.812,54.247,1.1301,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,128.0,16,0.042752,6429.592,55.952,1.1657,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,128.0,16,0.043648,12.3,0.856,0.0178,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,128.0,16,0.075872,905.729,63.055,1.3137,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,128.0,16,0.04624,17.416,0.858,0.0179,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,128.0,16,0.075904,1358.021,66.914,1.394,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,128.0,16,0.043008,24.966,0.976,0.0203,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,128.0,16,0.075168,1828.424,71.493,1.4894,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,,16,0.127584,269.311,295.874,6.164,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,,16,1.238272,443.97,121.94,2.5404,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,,16,19.186016,458.464,31.48,0.6558,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,,16,1.182944,464.735,120.552,2.5115,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,,16,2.3264,472.624,118.992,2.479,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,,16,2.305312,476.947,65.499,1.3646,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,,16,4.593088,478.768,62.096,1.2937,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,,16,1.170912,3.668,229.281,4.7767,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,,16,71.568642,7.682,480.153,10.0032,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,,16,1.173056,5.492,228.876,4.7683,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,,16,71.564384,11.523,480.211,10.0044,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,,16,1.173312,7.321,228.84,4.7675,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,,16,71.710434,15.333,479.263,9.9846,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,,16,0.084928,202.287,222.24,4.63,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,,16,0.659552,416.765,114.468,2.3847,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,,16,9.691936,453.784,31.159,0.6491,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,,16,0.609472,451.01,116.992,2.4373,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,,16,1.184832,463.995,116.82,2.4337,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,,16,1.1784,466.527,64.068,1.3347,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,,16,2.317952,474.346,61.523,1.2817,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,,16,1.169504,1.836,114.779,2.3912,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,,16,36.356544,7.561,472.596,9.8458,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,,16,1.172384,2.748,114.504,2.3855,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,,16,36.091969,11.424,476.09,9.9185,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,,16,1.176576,3.65,114.103,2.3771,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,,16,36.360737,15.119,472.599,9.8458,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,,16,0.063168,135.986,149.398,3.1125,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,,16,0.36992,371.537,102.046,2.126,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,,16,4.822176,456.023,31.313,0.6523,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,,16,0.602432,228.14,59.179,1.2329,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,,16,1.170976,234.743,59.101,1.2313,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,,16,0.608672,451.603,62.018,1.292,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,,16,1.186464,463.357,60.097,1.252,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,,16,1.1704,0.917,57.345,1.1947,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,,16,18.084896,7.6,475.036,9.8966,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,,16,1.169152,1.378,57.41,1.196,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,,16,18.079103,11.403,475.218,9.9004,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,,16,1.171072,1.834,57.319,1.1942,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,,16,18.128384,15.163,473.955,9.8741,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,,16,0.049888,86.092,94.584,1.9705,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,,16,0.224128,306.608,84.212,1.7544,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,,16,2.421056,454.145,31.184,0.6497,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,,16,0.60272,114.016,29.576,0.6162,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,,16,1.167328,117.738,29.643,0.6176,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,,16,0.600928,228.711,31.409,0.6543,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,,16,1.16784,235.373,30.528,0.636,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,,16,1.167744,0.46,28.738,0.5987,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,,16,9.069568,7.577,473.616,9.867,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,,16,1.168704,0.689,28.716,0.5983,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,,16,9.14128,11.276,469.929,9.7902,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,,16,1.168832,0.919,28.715,0.5982,20,True,,NVIDIA H200,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,,16,9.078752,15.139,473.195,9.8582,20,True,,NVIDIA H200,cudnn 9.26.0, diff --git a/benchmark/attention_inference/results/gpt_oss/h200/gpt_oss_context.png b/benchmark/attention_inference/results/gpt_oss/h200/gpt_oss_context.png new file mode 100644 index 000000000..13a3f143d Binary files /dev/null and b/benchmark/attention_inference/results/gpt_oss/h200/gpt_oss_context.png differ diff --git a/benchmark/attention_inference/results/gpt_oss/h200/gpt_oss_generation.png b/benchmark/attention_inference/results/gpt_oss/h200/gpt_oss_generation.png new file mode 100644 index 000000000..55f2d10a6 Binary files /dev/null and b/benchmark/attention_inference/results/gpt_oss/h200/gpt_oss_generation.png differ diff --git a/benchmark/attention_inference/results/gpt_oss/rtxpro6000/gpt_oss_20260818_142442.csv b/benchmark/attention_inference/results/gpt_oss/rtxpro6000/gpt_oss_20260818_142442.csv new file mode 100644 index 000000000..054290278 --- /dev/null +++ b/benchmark/attention_inference/results/gpt_oss/rtxpro6000/gpt_oss_20260818_142442.csv @@ -0,0 +1,209 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,128.0,16,0.056192,611.47,671.781,37.4878,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,128.0,16,0.182464,3012.955,827.533,46.1793,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,128.0,16,0.658048,13366.948,917.835,51.2185,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,128.0,16,0.034976,15718.086,277.314,15.4751,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,128.0,16,0.035424,31038.606,273.807,15.2794,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,128.0,16,0.056224,19555.913,340.362,18.9934,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,128.0,16,0.056992,38584.77,335.775,18.7375,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,128.0,16,0.021376,200.925,13.988,0.7806,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,128.0,16,0.215744,2548.186,177.4,9.8996,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,128.0,16,0.021376,301.387,14.85,0.8287,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,128.0,16,0.218304,3777.456,186.127,10.3866,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,128.0,16,0.022144,387.913,15.168,0.8464,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,128.0,16,0.217792,5048.448,197.398,11.0155,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,128.0,16,0.116704,294.418,323.457,18.0501,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,128.0,16,0.384384,1430.226,392.823,21.9209,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,128.0,16,1.554848,5657.205,388.449,21.6769,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,128.0,16,0.270432,2032.88,35.866,2.0015,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,128.0,16,0.46464,2366.373,20.875,1.1649,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,128.0,16,0.290144,3789.538,65.955,3.6805,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,128.0,16,0.487424,4511.52,39.261,2.1909,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,128.0,16,0.452992,9.481,0.66,0.0368,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,128.0,16,51.771328,10.619,0.739,0.0413,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,128.0,16,0.451584,14.266,0.703,0.0392,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,128.0,16,51.776798,15.927,0.785,0.0438,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,128.0,16,0.451584,19.022,0.744,0.0415,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,128.0,16,51.774593,21.237,0.83,0.0463,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,128.0,16,0.037248,461.229,506.722,28.2769,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,128.0,16,0.097984,2805.335,770.508,42.9971,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,128.0,16,0.34672,12684.721,870.991,48.6044,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,128.0,16,0.03456,7953.643,140.326,7.8307,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,128.0,16,0.035232,15603.878,137.649,7.6813,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,128.0,16,0.035776,15366.609,267.449,14.9246,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,128.0,16,0.036256,30326.335,263.908,14.727,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,128.0,16,0.022496,95.461,6.646,0.3709,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,128.0,16,0.118784,2314.099,161.103,8.9901,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,128.0,16,0.021472,150.02,7.392,0.4125,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,128.0,16,0.123296,3344.122,164.775,9.1951,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,128.0,16,0.022528,190.65,7.455,0.416,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,128.0,16,0.123008,4469.269,174.751,9.7517,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,128.0,16,0.09216,186.414,204.8,11.4286,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,128.0,16,0.204896,1341.548,368.467,20.5618,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,128.0,16,0.783584,5612.731,385.396,21.5065,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,128.0,16,0.131424,2091.535,36.901,2.0592,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,128.0,16,0.255168,2154.486,19.006,1.0606,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,128.0,16,0.161024,3414.123,59.421,3.3159,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,128.0,16,0.262656,4186.128,36.429,2.0329,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,128.0,16,0.247552,8.675,0.604,0.0337,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,128.0,16,25.492577,10.783,0.751,0.0419,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,128.0,16,0.248352,12.97,0.639,0.0357,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,128.0,16,25.579329,16.119,0.794,0.0443,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,128.0,16,0.248288,17.298,0.676,0.0377,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,128.0,16,25.580544,21.491,0.84,0.0469,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,128.0,16,0.02944,291.778,320.557,17.8882,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,128.0,16,0.060128,2285.773,627.806,35.0338,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,128.0,16,0.188672,11655.27,800.304,44.6598,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,128.0,16,0.035776,3841.652,67.778,3.7823,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,128.0,16,0.035072,7837.532,69.139,3.8582,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,128.0,16,0.036,7635.498,132.892,7.4159,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,128.0,16,0.035136,15646.511,136.16,7.5982,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,128.0,16,0.02192,48.985,3.41,0.1903,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,128.0,16,0.071648,1918.252,133.545,7.4523,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,128.0,16,0.0216,74.565,3.674,0.205,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,128.0,16,0.072096,2859.499,140.897,7.8625,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,128.0,16,0.021568,99.568,3.893,0.2173,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,128.0,16,0.072512,3790.792,148.222,8.2713,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,128.0,16,0.075968,113.073,124.226,6.9322,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,128.0,16,0.116352,1181.234,324.436,18.1047,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,128.0,16,0.393792,5584.225,383.438,21.3972,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,128.0,16,0.071264,1928.589,34.026,1.8988,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,128.0,16,0.129024,2130.44,18.794,1.0488,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,128.0,16,0.078208,3514.703,61.172,3.4136,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,128.0,16,0.131072,4194.304,36.5,2.0368,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,128.0,16,0.108864,9.863,0.687,0.0383,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,128.0,16,12.682784,10.837,0.754,0.0421,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,128.0,16,0.109568,14.7,0.724,0.0404,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,128.0,16,12.684384,16.253,0.801,0.0447,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,128.0,16,0.10864,19.767,0.773,0.0431,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,128.0,16,12.68336,21.672,0.847,0.0473,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,128.0,16,0.022432,191.466,210.351,11.7383,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,128.0,16,0.037024,1856.079,509.787,28.448,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,128.0,16,0.097664,11258.106,773.033,43.138,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,128.0,16,0.035328,1945.184,34.319,1.9151,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,128.0,16,0.0352,3904.516,34.444,1.9221,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,128.0,16,0.03488,3940.337,68.58,3.827,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,128.0,16,0.035936,7649.095,66.565,3.7145,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,128.0,16,0.02256,23.797,1.657,0.0925,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,128.0,16,0.045248,1518.73,105.731,5.9002,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,128.0,16,0.021952,36.685,1.808,0.1009,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,128.0,16,0.045792,2251.031,110.915,6.1895,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,128.0,16,0.02144,50.081,1.958,0.1093,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,128.0,16,0.045632,3011.898,117.767,6.5718,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,128.0,16,0.061024,70.382,77.324,4.3149,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,128.0,16,0.07952,864.179,237.354,13.2452,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,128.0,16,0.186144,5906.78,405.586,22.6332,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,128.0,16,0.059456,1155.804,20.392,1.1379,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,128.0,16,0.06016,2284.557,20.153,1.1246,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,128.0,16,0.058624,2344.414,40.803,2.277,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,128.0,16,0.059488,4620.729,40.211,2.2439,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,128.0,16,0.071808,7.476,0.52,0.029,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,128.0,16,0.085856,800.404,55.723,3.1095,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,128.0,16,0.060544,13.301,0.655,0.0366,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,128.0,16,0.085728,1202.398,59.246,3.3061,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,128.0,16,0.059936,17.915,0.7,0.0391,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_swa-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,128.0,16,0.086432,1590.14,62.175,3.4696,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,,16,0.164128,209.347,229.996,12.8346,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,,16,1.715168,320.526,88.035,4.9127,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,,16,25.462591,345.452,23.72,1.3237,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,,16,2.337824,235.157,61.0,3.404,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,,16,4.648992,236.505,59.545,3.3228,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,,16,4.64096,236.915,32.535,1.8156,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,,16,9.269792,237.225,30.768,1.717,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,,16,2.492224,1.723,107.722,6.0113,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,,16,98.148384,5.601,350.122,19.5381,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,,16,2.492544,2.585,107.715,6.0109,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,,16,98.141312,8.403,350.169,19.5407,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,,16,2.492448,3.446,107.726,6.0115,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,,16,98.14032,11.203,350.194,19.5421,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,8,64,64,gqa,,16,0.18176,189.039,207.685,11.5895,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,8,64,64,gqa,,16,1.626944,337.907,92.809,5.1791,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,64,8,64,64,gqa,,16,23.576639,373.085,25.618,1.4296,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,8,64,64,gqa,,16,2.115712,259.844,67.403,3.7614,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,8,64,64,gqa,,16,4.346528,252.963,63.689,3.554,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,8,64,64,gqa,,16,3.819616,287.859,39.531,2.206,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,8,64,64,gqa,,16,7.482912,293.873,38.115,2.127,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,8,64,64,gqa,,16,1.5872,2.706,169.146,9.4389,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,8,64,64,gqa,,16,136.240677,4.035,252.23,14.0753,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,8,64,64,gqa,,16,1.587232,4.059,169.153,9.4393,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,8,64,64,gqa,,16,136.232346,6.053,252.26,14.077,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,8,64,64,gqa,,16,1.58752,5.411,169.132,9.4382,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,8,64,64,gqa,,16,136.230011,8.071,252.28,14.0781,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,,16,0.097408,176.37,193.766,10.8128,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,,16,0.889984,308.857,84.83,4.7338,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,,16,12.827712,342.855,23.542,1.3137,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,,16,2.331008,117.922,30.589,1.707,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,,16,4.644,118.38,29.804,1.6632,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,,16,2.33376,235.567,32.35,1.8053,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,,16,4.652736,236.315,30.65,1.7104,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,,16,2.4912,0.862,53.883,3.0069,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,,16,49.181664,5.589,349.357,19.4954,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,,16,2.490656,1.293,53.898,3.0077,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,,16,49.17728,8.384,349.41,19.4983,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,,16,2.490656,1.724,53.902,3.0079,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,,16,49.181599,11.178,349.4,19.4978,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,4,64,64,gqa,,16,0.134464,127.766,140.367,7.833,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,4,64,64,gqa,,16,0.824896,333.227,91.524,5.1073,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,4,64,64,gqa,,16,11.641536,377.789,25.941,1.4476,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,4,64,64,gqa,,16,1.24112,221.476,57.451,3.206,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,4,64,64,gqa,,16,2.587936,212.43,53.484,2.9846,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,4,64,64,gqa,,16,1.8192,302.196,41.5,2.3159,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,4,64,64,gqa,,16,3.673408,299.316,38.821,2.1664,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,4,64,64,gqa,,16,1.37472,1.562,97.645,5.4489,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,4,64,64,gqa,,16,67.753441,4.057,253.595,14.1515,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,4,64,64,gqa,,16,1.373152,2.346,97.762,5.4555,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,4,64,64,gqa,,16,67.838333,6.078,253.294,14.1347,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,4,64,64,gqa,,16,1.370752,3.133,97.939,5.4654,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,4,64,64,gqa,,16,67.833984,8.104,253.325,14.1365,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,,16,0.069216,124.103,136.344,7.6085,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,,16,0.501632,273.984,75.252,4.1993,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,,16,6.3592,345.802,23.744,1.325,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,,16,2.33024,58.981,15.3,0.8538,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,,16,4.642784,59.205,14.906,0.8318,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,,16,2.330816,117.932,16.196,0.9038,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,,16,4.645152,118.35,15.35,0.8566,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,,16,2.568256,0.418,26.133,1.4583,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,,16,24.688416,5.567,347.976,19.4183,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,,16,2.492064,0.646,26.934,1.503,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,,16,24.690559,8.35,347.967,19.4178,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,,16,2.489952,0.862,26.958,1.5044,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,,16,24.691681,11.132,347.973,19.4181,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,2,64,64,gqa,,16,0.099456,86.369,94.888,5.2951,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,2,64,64,gqa,,16,0.435488,315.598,86.681,4.8371,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,2,64,64,gqa,,16,5.516864,398.6,27.37,1.5273,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,2,64,64,gqa,,16,0.63232,217.357,56.382,3.1463,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,2,64,64,gqa,,16,1.237568,222.111,55.921,3.1206,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,2,64,64,gqa,,16,1.191136,230.77,31.691,1.7685,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,2,64,64,gqa,,16,2.353024,233.638,30.303,1.691,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,2,64,64,gqa,,16,1.263808,0.85,53.107,2.9636,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,2,64,64,gqa,,16,33.871521,4.058,253.634,14.1537,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,2,64,64,gqa,,16,1.255872,1.282,53.446,2.9825,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,2,64,64,gqa,,16,33.870529,6.087,253.657,14.155,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,2,64,64,gqa,,16,1.263712,1.699,53.118,2.9641,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,2,64,64,gqa,,16,33.867359,8.116,253.697,14.1572,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,,16,0.053536,80.226,88.139,4.9185,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,,16,0.320704,214.277,58.853,3.2842,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,,16,3.3472,328.487,22.555,1.2587,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,,16,2.33088,29.482,7.648,0.4268,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,,16,4.643904,29.596,7.451,0.4158,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,,16,2.329632,58.996,8.102,0.4521,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,,16,4.642144,59.214,7.68,0.4286,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,,16,2.529216,0.212,13.268,0.7404,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,,16,13.530496,5.079,317.467,17.7158,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,,16,2.48992,0.323,13.479,0.7522,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,,16,13.530912,7.618,317.477,17.7164,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,,16,2.490176,0.431,13.478,0.7521,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,,16,13.532032,10.157,317.47,17.716,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,64,64,gqa,,16,0.070208,61.175,67.209,3.7505,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,64,64,gqa,,16,0.28496,241.155,66.235,3.6962,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,64,64,gqa,,16,2.771232,396.759,27.243,1.5203,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,64,64,gqa,,16,0.626784,109.638,28.44,1.5871,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,64,64,gqa,,16,1.204576,114.097,28.726,1.603,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,64,64,gqa,,16,0.641632,214.202,29.416,1.6415,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,64,64,gqa,,16,1.221184,225.091,29.194,1.6291,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,64,64,gqa,,16,1.20032,0.447,27.958,1.5602,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,64,64,gqa,,16,11.66496,5.891,368.239,20.549,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,64,64,gqa,,16,1.200928,0.671,27.946,1.5595,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,64,64,gqa,,16,11.66368,8.838,368.302,20.5526,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,64,64,gqa,,16,1.199456,0.895,27.982,1.5615,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +gpt_oss,gpt_oss_full-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,64,64,gqa,,16,11.664704,11.782,368.292,20.552,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], diff --git a/benchmark/attention_inference/results/gpt_oss/rtxpro6000/gpt_oss_context.png b/benchmark/attention_inference/results/gpt_oss/rtxpro6000/gpt_oss_context.png new file mode 100644 index 000000000..b05935fb9 Binary files /dev/null and b/benchmark/attention_inference/results/gpt_oss/rtxpro6000/gpt_oss_context.png differ diff --git a/benchmark/attention_inference/results/gpt_oss/rtxpro6000/gpt_oss_generation.png b/benchmark/attention_inference/results/gpt_oss/rtxpro6000/gpt_oss_generation.png new file mode 100644 index 000000000..356c44e16 Binary files /dev/null and b/benchmark/attention_inference/results/gpt_oss/rtxpro6000/gpt_oss_generation.png differ diff --git a/benchmark/attention_inference/results/kimi_k3/b300/kimi_k3_20260817_132655.csv b/benchmark/attention_inference/results/kimi_k3/b300/kimi_k3_20260817_132655.csv deleted file mode 100644 index 5c4c72c9e..000000000 --- a/benchmark/attention_inference/results/kimi_k3/b300/kimi_k3_20260817_132655.csv +++ /dev/null @@ -1,473 +0,0 @@ -config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version -kimi_k3,kimi_k3,context,cudnn,bfloat16,bfloat16,1,2048,2048,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3,context,cudnn,bfloat16,bfloat16,1,8192,8192,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3,context,cudnn,bfloat16,bfloat16,1,32768,32768,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3,context,cudnn,bfloat16,bfloat16,1,512,65536,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3,context,cudnn,bfloat16,bfloat16,1,512,131072,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3,context,cudnn,bfloat16,bfloat16,1,1024,65536,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3,context,cudnn,bfloat16,bfloat16,1,1024,131072,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,1,131072,96,1,576,512,mla_absorbed,,32,0.648128,42.245,233.293,2.9162,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,1,131072,96,1,576,512,mla_absorbed,,32,68.145439,51.43,284.012,3.5501,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,2,131072,96,1,576,512,mla_absorbed,,32,28.62064,1.913,5.29,0.0661,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,2,131072,96,1,576,512,mla_absorbed,,32,2475.784424,2.831,7.828,0.0979,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,3,131072,96,1,576,512,mla_absorbed,,32,28.606976,2.871,5.3,0.0663,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,3,131072,96,1,576,512,mla_absorbed,,32,2473.067139,4.251,7.848,0.0981,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,4,131072,96,1,576,512,mla_absorbed,,32,28.599936,3.829,5.309,0.0664,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,4,131072,96,1,576,512,mla_absorbed,,32,2473.15918,5.668,7.858,0.0982,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,96,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,48,1,576,512,mla_absorbed,,32,0.361248,37.897,418.271,5.2284,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,48,1,576,512,mla_absorbed,,32,37.721249,46.455,512.728,6.4091,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,48,1,576,512,mla_absorbed,,32,12.899424,2.123,11.722,0.1465,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,48,1,576,512,mla_absorbed,,32,1188.61853,2.949,16.283,0.2035,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,48,1,576,512,mla_absorbed,,32,12.894688,3.185,11.734,0.1467,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,48,1,576,512,mla_absorbed,,32,1185.102295,4.436,16.342,0.2043,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,48,1,576,512,mla_absorbed,,32,12.884736,4.25,11.751,0.1469,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,48,1,576,512,mla_absorbed,,32,1186.050903,5.91,16.341,0.2043,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,48,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,24,1,576,512,mla_absorbed,,32,0.258336,26.497,584.693,7.3087,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,24,1,576,512,mla_absorbed,,32,26.152897,33.502,739.269,9.2409,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,24,1,576,512,mla_absorbed,,32,12.821792,1.068,11.785,0.1473,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,24,1,576,512,mla_absorbed,,32,580.829529,3.017,33.298,0.4162,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,24,1,576,512,mla_absorbed,,32,12.814176,1.603,11.796,0.1474,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,24,1,576,512,mla_absorbed,,32,579.37915,4.537,33.393,0.4174,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,24,1,576,512,mla_absorbed,,32,12.809984,2.137,11.804,0.1475,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,24,1,576,512,mla_absorbed,,32,579.739563,6.045,33.384,0.4173,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,24,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,12,1,576,512,mla_absorbed,,32,0.154336,22.176,978.521,12.2315,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,12,1,576,512,mla_absorbed,,32,13.870304,31.585,1393.675,17.4209,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,12,1,576,512,mla_absorbed,,32,12.800768,0.535,11.8,0.1475,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,12,1,576,512,mla_absorbed,,32,289.056396,3.031,66.887,0.8361,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,12,1,576,512,mla_absorbed,,32,12.793088,0.803,11.809,0.1476,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,12,1,576,512,mla_absorbed,,32,289.190216,4.545,66.867,0.8358,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,12,1,576,512,mla_absorbed,,32,12.790688,1.07,11.813,0.1477,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,12,1,576,512,mla_absorbed,,32,289.183289,6.06,66.88,0.836,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -kimi_k3,kimi_k3-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False,SKIPPED: context phase for latent-absorbed MLA runs unabsorbed (see training suite prefill),,, -kimi_k3,kimi_k3-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -kimi_k3,kimi_k3-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,12,1,576,512,mla_absorbed,,32,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 173, in setup_cudnn - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, diff --git a/benchmark/attention_inference/results/kimi_k3/b300/kimi_k3_context.png b/benchmark/attention_inference/results/kimi_k3/b300/kimi_k3_context.png deleted file mode 100644 index a4baf876c..000000000 Binary files a/benchmark/attention_inference/results/kimi_k3/b300/kimi_k3_context.png and /dev/null differ diff --git a/benchmark/attention_inference/results/kimi_k3/b300/kimi_k3_generation.png b/benchmark/attention_inference/results/kimi_k3/b300/kimi_k3_generation.png deleted file mode 100644 index b7dc9f5ba..000000000 Binary files a/benchmark/attention_inference/results/kimi_k3/b300/kimi_k3_generation.png and /dev/null differ diff --git a/benchmark/attention_inference/results/kimi_k3/gb200/kimi_k3_20260818_124034.csv b/benchmark/attention_inference/results/kimi_k3/gb200/kimi_k3_20260818_124034.csv new file mode 100644 index 000000000..139aefe8f --- /dev/null +++ b/benchmark/attention_inference/results/kimi_k3/gb200/kimi_k3_20260818_124034.csv @@ -0,0 +1,33 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,1,131072,96,1,576,512,mla_absorbed,,32,0.645952,42.388,234.079,2.926,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,1,131072,96,1,576,512,mla_absorbed,,32,68.169281,51.412,283.912,3.5489,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,2,131072,96,1,576,512,mla_absorbed,,32,28.54096,1.919,5.305,0.0663,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,2,131072,96,1,576,512,mla_absorbed,,32,2382.912109,2.942,8.133,0.1017,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,3,131072,96,1,576,512,mla_absorbed,,32,28.53664,2.878,5.313,0.0664,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,3,131072,96,1,576,512,mla_absorbed,,32,2382.477783,4.413,8.146,0.1018,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,4,131072,96,1,576,512,mla_absorbed,,32,28.532896,3.838,5.321,0.0665,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,4,131072,96,1,576,512,mla_absorbed,,32,2380.615479,5.889,8.164,0.102,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,48,1,576,512,mla_absorbed,,32,0.359776,38.052,419.982,5.2498,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,48,1,576,512,mla_absorbed,,32,37.622913,46.577,514.068,6.4258,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,48,1,576,512,mla_absorbed,,32,12.794208,2.14,11.818,0.1477,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,48,1,576,512,mla_absorbed,,32,1133.448608,3.092,17.075,0.2134,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,48,1,576,512,mla_absorbed,,32,12.788096,3.212,11.832,0.1479,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,48,1,576,512,mla_absorbed,,32,1133.631104,4.637,17.084,0.2136,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,48,1,576,512,mla_absorbed,,32,12.77776,4.286,11.85,0.1481,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,48,1,576,512,mla_absorbed,,32,1132.288574,6.19,17.117,0.214,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,24,1,576,512,mla_absorbed,,32,0.25904,26.425,583.104,7.2888,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,24,1,576,512,mla_absorbed,,32,26.585793,32.956,727.232,9.0904,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,24,1,576,512,mla_absorbed,,32,12.748736,1.074,11.852,0.1482,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,24,1,576,512,mla_absorbed,,32,561.145264,3.123,34.467,0.4308,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,24,1,576,512,mla_absorbed,,32,12.733792,1.613,11.87,0.1484,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,24,1,576,512,mla_absorbed,,32,561.032715,4.685,34.485,0.4311,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,24,1,576,512,mla_absorbed,,32,12.732192,2.15,11.876,0.1484,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,24,1,576,512,mla_absorbed,,32,560.009644,6.258,34.56,0.432,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,12,1,576,512,mla_absorbed,,32,0.152672,22.418,989.186,12.3648,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,12,1,576,512,mla_absorbed,,32,14.000192,31.291,1380.745,17.2593,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,12,1,576,512,mla_absorbed,,32,12.723584,0.538,11.871,0.1484,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,12,1,576,512,mla_absorbed,,32,285.567413,3.068,67.704,0.8463,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,12,1,576,512,mla_absorbed,,32,12.710304,0.808,11.886,0.1486,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,12,1,576,512,mla_absorbed,,32,285.075317,4.61,67.833,0.8479,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,12,1,576,512,mla_absorbed,,32,12.710304,1.077,11.888,0.1486,20,True,,NVIDIA GB200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,12,1,576,512,mla_absorbed,,32,285.309204,6.142,67.789,0.8474,20,True,,NVIDIA GB200,cudnn 9.26.0, diff --git a/benchmark/attention_inference/results/kimi_k3/gb200/kimi_k3_context.png b/benchmark/attention_inference/results/kimi_k3/gb200/kimi_k3_context.png new file mode 100644 index 000000000..d5c5da866 Binary files /dev/null and b/benchmark/attention_inference/results/kimi_k3/gb200/kimi_k3_context.png differ diff --git a/benchmark/attention_inference/results/kimi_k3/gb200/kimi_k3_generation.png b/benchmark/attention_inference/results/kimi_k3/gb200/kimi_k3_generation.png new file mode 100644 index 000000000..a6709ab3f Binary files /dev/null and b/benchmark/attention_inference/results/kimi_k3/gb200/kimi_k3_generation.png differ diff --git a/benchmark/attention_inference/results/kimi_k3/gb300/kimi_k3_20260818_124631.csv b/benchmark/attention_inference/results/kimi_k3/gb300/kimi_k3_20260818_124631.csv new file mode 100644 index 000000000..3bb2b36b6 --- /dev/null +++ b/benchmark/attention_inference/results/kimi_k3/gb300/kimi_k3_20260818_124631.csv @@ -0,0 +1,33 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,1,131072,96,1,576,512,mla_absorbed,,32,0.632864,43.264,238.92,2.9865,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,1,131072,96,1,576,512,mla_absorbed,,32,66.739746,52.513,289.993,3.6249,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,2,131072,96,1,576,512,mla_absorbed,,32,28.193184,1.942,5.371,0.0671,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,2,131072,96,1,576,512,mla_absorbed,,32,2350.77002,2.982,8.244,0.1031,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,3,131072,96,1,576,512,mla_absorbed,,32,28.179457,2.915,5.381,0.0673,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,3,131072,96,1,576,512,mla_absorbed,,32,2348.12915,4.478,8.265,0.1033,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,4,131072,96,1,576,512,mla_absorbed,,32,28.180672,3.886,5.388,0.0673,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,4,131072,96,1,576,512,mla_absorbed,,32,2346.813721,5.974,8.281,0.1035,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,48,1,576,512,mla_absorbed,,32,0.353024,38.78,428.014,5.3502,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,48,1,576,512,mla_absorbed,,32,37.026817,47.326,522.344,6.5293,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,48,1,576,512,mla_absorbed,,32,12.802208,2.139,11.811,0.1476,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,48,1,576,512,mla_absorbed,,32,1118.749146,3.133,17.3,0.2162,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,48,1,576,512,mla_absorbed,,32,12.79728,3.209,11.823,0.1478,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,48,1,576,512,mla_absorbed,,32,1119.354614,4.696,17.302,0.2163,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,48,1,576,512,mla_absorbed,,32,12.793632,4.28,11.835,0.1479,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,48,1,576,512,mla_absorbed,,32,1118.605103,6.266,17.326,0.2166,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,24,1,576,512,mla_absorbed,,32,0.256928,26.642,587.897,7.3487,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,24,1,576,512,mla_absorbed,,32,26.131872,33.529,739.864,9.2483,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,24,1,576,512,mla_absorbed,,32,12.738976,1.075,11.861,0.1483,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,24,1,576,512,mla_absorbed,,32,553.474365,3.166,34.944,0.4368,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,24,1,576,512,mla_absorbed,,32,12.72672,1.614,11.877,0.1485,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,24,1,576,512,mla_absorbed,,32,552.815613,4.755,34.998,0.4375,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,24,1,576,512,mla_absorbed,,32,12.71552,2.153,11.891,0.1486,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,24,1,576,512,mla_absorbed,,32,552.73291,6.341,35.015,0.4377,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,12,1,576,512,mla_absorbed,,32,0.149344,22.917,1011.23,12.6404,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,12,1,576,512,mla_absorbed,,32,13.573472,32.275,1424.153,17.8019,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,12,1,576,512,mla_absorbed,,32,12.710304,0.539,11.884,0.1485,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,12,1,576,512,mla_absorbed,,32,281.895721,3.108,68.586,0.8573,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,12,1,576,512,mla_absorbed,,32,12.697728,0.809,11.898,0.1487,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,12,1,576,512,mla_absorbed,,32,281.512268,4.669,68.691,0.8586,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,12,1,576,512,mla_absorbed,,32,12.690432,1.079,11.907,0.1488,20,True,,NVIDIA GB300,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,12,1,576,512,mla_absorbed,,32,281.619934,6.222,68.677,0.8585,20,True,,NVIDIA GB300,cudnn 9.26.0, diff --git a/benchmark/attention_inference/results/kimi_k3/gb300/kimi_k3_context.png b/benchmark/attention_inference/results/kimi_k3/gb300/kimi_k3_context.png new file mode 100644 index 000000000..d5c5da866 Binary files /dev/null and b/benchmark/attention_inference/results/kimi_k3/gb300/kimi_k3_context.png differ diff --git a/benchmark/attention_inference/results/kimi_k3/gb300/kimi_k3_generation.png b/benchmark/attention_inference/results/kimi_k3/gb300/kimi_k3_generation.png new file mode 100644 index 000000000..4551c878a Binary files /dev/null and b/benchmark/attention_inference/results/kimi_k3/gb300/kimi_k3_generation.png differ diff --git a/benchmark/attention_inference/results/kimi_k3/h200/kimi_k3_20260818_141810.csv b/benchmark/attention_inference/results/kimi_k3/h200/kimi_k3_20260818_141810.csv new file mode 100644 index 000000000..e9fe698e8 --- /dev/null +++ b/benchmark/attention_inference/results/kimi_k3/h200/kimi_k3_20260818_141810.csv @@ -0,0 +1,33 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,1,131072,96,1,576,512,mla_absorbed,,32,0.6832,40.077,221.317,4.6108,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,1,131072,96,1,576,512,mla_absorbed,,32,79.962273,43.829,242.04,5.0425,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,2,131072,96,1,576,512,mla_absorbed,,32,28.968609,1.89,5.227,0.1089,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,2,131072,96,1,576,512,mla_absorbed,,32,2812.320068,2.492,6.891,0.1436,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,3,131072,96,1,576,512,mla_absorbed,,32,28.948896,2.837,5.238,0.1091,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,3,131072,96,1,576,512,mla_absorbed,,32,2814.22168,3.736,6.896,0.1437,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,4,131072,96,1,576,512,mla_absorbed,,32,28.935841,3.785,5.247,0.1093,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,4,131072,96,1,576,512,mla_absorbed,,32,2814.230469,4.981,6.906,0.1439,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,48,1,576,512,mla_absorbed,,32,0.392096,34.915,385.363,8.0284,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,48,1,576,512,mla_absorbed,,32,42.462208,41.268,455.481,9.4892,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,48,1,576,512,mla_absorbed,,32,13.101344,2.09,11.541,0.2404,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,48,1,576,512,mla_absorbed,,32,1348.157104,2.6,14.356,0.2991,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,48,1,576,512,mla_absorbed,,32,13.013568,3.156,11.627,0.2422,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,48,1,576,512,mla_absorbed,,32,1347.359375,3.902,14.374,0.2995,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,48,1,576,512,mla_absorbed,,32,12.9944,4.214,11.652,0.2428,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,48,1,576,512,mla_absorbed,,32,1347.975342,5.2,14.378,0.2995,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,24,1,576,512,mla_absorbed,,32,0.277056,24.707,545.186,11.3581,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,24,1,576,512,mla_absorbed,,32,27.491327,31.871,703.278,14.6516,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,24,1,576,512,mla_absorbed,,32,12.841728,1.066,11.766,0.2451,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,24,1,576,512,mla_absorbed,,32,657.905518,2.664,29.397,0.6124,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,24,1,576,512,mla_absorbed,,32,12.82224,1.602,11.788,0.2456,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,24,1,576,512,mla_absorbed,,32,657.248169,3.999,29.437,0.6133,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,24,1,576,512,mla_absorbed,,32,12.814048,2.137,11.8,0.2458,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,24,1,576,512,mla_absorbed,,32,656.560425,5.338,29.478,0.6141,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,12,1,576,512,mla_absorbed,,32,0.161504,21.192,935.092,19.4811,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,12,1,576,512,mla_absorbed,,32,14.892832,29.416,1297.987,27.0414,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,12,1,576,512,mla_absorbed,,32,12.773504,0.536,11.825,0.2464,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,12,1,576,512,mla_absorbed,,32,332.638824,2.634,58.123,1.2109,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,12,1,576,512,mla_absorbed,,32,12.760416,0.805,11.839,0.2467,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,12,1,576,512,mla_absorbed,,32,332.633331,3.951,58.134,1.2111,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,12,1,576,512,mla_absorbed,,32,12.742208,1.074,11.858,0.247,20,True,,NVIDIA H200,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,12,1,576,512,mla_absorbed,,32,332.625031,5.268,58.146,1.2114,20,True,,NVIDIA H200,cudnn 9.26.0, diff --git a/benchmark/attention_inference/results/kimi_k3/h200/kimi_k3_context.png b/benchmark/attention_inference/results/kimi_k3/h200/kimi_k3_context.png new file mode 100644 index 000000000..d5c5da866 Binary files /dev/null and b/benchmark/attention_inference/results/kimi_k3/h200/kimi_k3_context.png differ diff --git a/benchmark/attention_inference/results/kimi_k3/h200/kimi_k3_generation.png b/benchmark/attention_inference/results/kimi_k3/h200/kimi_k3_generation.png new file mode 100644 index 000000000..dd4ec943f Binary files /dev/null and b/benchmark/attention_inference/results/kimi_k3/h200/kimi_k3_generation.png differ diff --git a/benchmark/attention_inference/results/kimi_k3/rtxpro6000/kimi_k3_20260818_143146.csv b/benchmark/attention_inference/results/kimi_k3/rtxpro6000/kimi_k3_20260818_143146.csv new file mode 100644 index 000000000..6d62c5a82 --- /dev/null +++ b/benchmark/attention_inference/results/kimi_k3/rtxpro6000/kimi_k3_20260818_143146.csv @@ -0,0 +1,33 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,1,131072,96,1,576,512,mla_absorbed,,32,1.144512,23.923,132.112,7.3723,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,1,131072,96,1,576,512,mla_absorbed,,32,119.706367,29.277,161.68,9.0223,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,2,131072,96,1,576,512,mla_absorbed,,32,27.3848,2.0,5.529,0.3085,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,2,131072,96,1,576,512,mla_absorbed,,32,1917.439697,3.656,10.108,0.564,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,3,131072,96,1,576,512,mla_absorbed,,32,27.369761,3.001,5.54,0.3091,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,3,131072,96,1,576,512,mla_absorbed,,32,2022.694336,5.198,9.595,0.5354,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,1,4,131072,96,1,576,512,mla_absorbed,,32,27.380352,4.0,5.545,0.3094,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3,generation,cudnn,bfloat16,bfloat16,128,4,131072,96,1,576,512,mla_absorbed,,32,2073.32373,6.761,9.374,0.5231,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,48,1,576,512,mla_absorbed,,32,0.562752,24.327,268.501,14.9833,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,48,1,576,512,mla_absorbed,,32,62.271042,28.141,310.589,17.332,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,48,1,576,512,mla_absorbed,,32,15.601344,1.755,9.692,0.5408,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,48,1,576,512,mla_absorbed,,32,963.890137,3.636,20.079,1.1205,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,48,1,576,512,mla_absorbed,,32,15.604832,2.632,9.696,0.5411,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,48,1,576,512,mla_absorbed,,32,981.849365,5.354,19.725,1.1008,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,48,1,576,512,mla_absorbed,,32,15.61264,3.507,9.698,0.5412,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,48,1,576,512,mla_absorbed,,32,1006.457642,6.964,19.256,1.0746,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,24,1,576,512,mla_absorbed,,32,0.36672,18.666,411.887,22.9848,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,24,1,576,512,mla_absorbed,,32,41.286304,21.222,468.292,26.1324,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,24,1,576,512,mla_absorbed,,32,13.09776,1.045,11.536,0.6438,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,24,1,576,512,mla_absorbed,,32,483.153564,3.627,40.03,2.2338,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,24,1,576,512,mla_absorbed,,32,13.100096,1.568,11.538,0.6439,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,24,1,576,512,mla_absorbed,,32,486.625458,5.402,39.758,2.2187,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,24,1,576,512,mla_absorbed,,32,13.098656,2.09,11.543,0.6442,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,24,1,576,512,mla_absorbed,,32,489.718292,7.157,39.521,2.2054,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,12,1,576,512,mla_absorbed,,32,0.252832,13.537,597.318,33.3325,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,12,1,576,512,mla_absorbed,,32,21.435328,20.438,901.815,50.3245,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,12,1,576,512,mla_absorbed,,32,13.18128,0.519,11.459,0.6395,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,12,1,576,512,mla_absorbed,,32,243.404739,3.6,79.432,4.4326,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,12,1,576,512,mla_absorbed,,32,13.18144,0.779,11.461,0.6396,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,12,1,576,512,mla_absorbed,,32,244.031036,5.386,79.241,4.422,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,12,1,576,512,mla_absorbed,,32,13.181248,1.039,11.463,0.6397,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +kimi_k3,kimi_k3-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,12,1,576,512,mla_absorbed,,32,245.65271,7.133,78.732,4.3935,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, diff --git a/benchmark/attention_inference/results/kimi_k3/rtxpro6000/kimi_k3_context.png b/benchmark/attention_inference/results/kimi_k3/rtxpro6000/kimi_k3_context.png new file mode 100644 index 000000000..d5c5da866 Binary files /dev/null and b/benchmark/attention_inference/results/kimi_k3/rtxpro6000/kimi_k3_context.png differ diff --git a/benchmark/attention_inference/results/kimi_k3/rtxpro6000/kimi_k3_generation.png b/benchmark/attention_inference/results/kimi_k3/rtxpro6000/kimi_k3_generation.png new file mode 100644 index 000000000..f5d607584 Binary files /dev/null and b/benchmark/attention_inference/results/kimi_k3/rtxpro6000/kimi_k3_generation.png differ diff --git a/benchmark/attention_inference/results/llama3.1/b300/llama3.1_20260817_133324.csv b/benchmark/attention_inference/results/llama3.1/b300/llama3.1_20260817_133324.csv deleted file mode 100644 index 5dc35e873..000000000 --- a/benchmark/attention_inference/results/llama3.1/b300/llama3.1_20260817_133324.csv +++ /dev/null @@ -1,537 +0,0 @@ -config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version -llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,128,128,gqa,,16,0.08192,838.861,921.6,11.52,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,128,128,gqa,,16,0.657536,1672.17,459.275,5.7409,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,128,128,gqa,,16,11.398048,1543.438,105.98,1.3247,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,128,128,gqa,,16,0.661024,1663.346,431.471,5.3934,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,128,128,gqa,,16,1.284288,1712.251,431.093,5.3887,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,128,128,gqa,,16,1.302368,1688.481,231.878,2.8985,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,128,128,gqa,,16,2.787072,1578.017,204.668,2.5584,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,8,128,128,gqa,,16,0.108288,39.662,4958.109,61.9764,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,64,8,128,128,gqa,,16,0.104096,41.26,2579.045,32.2381,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,8,128,128,gqa,,16,9.359328,58.739,7342.8,91.785,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,64,8,128,128,gqa,,16,inf,0.0,0.0,,20,False," File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 266, in setup_cudnn_fp8 - graph.build_plans() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1454, in build_plans - raise cudnn_graph_not_supported(""no plan in the list could be built:\n "" + ""\n "".join(failures or [""the plan list is empty""])) -cudnn._compiled_module.cudnnGraphNotSupportedError: no plan in the list could be built: - [0] eng10_k24=1_k27=0_k38=0_k40=3_k41=2: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor FailedEncountered runtime kernel compilation failure at: compilationResult != NVRTC_SUCCESS cudnn_status: CUDNN_STATUS_INTERNAL_ERROR_COMPILATION_FAILED",,, -llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,128,128,gqa,,16,2.192864,3.917,244.856,3.0607,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,64,8,128,128,gqa,,16,0.991232,8.666,270.876,3.386,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,8,128,128,gqa,,16,88.651169,12.403,775.262,9.6908,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,64,8,128,128,gqa,,16,59.048962,18.62,582.028,7.2753,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,128,128,gqa,,16,2.191456,5.88,245.029,3.0629,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,64,8,128,128,gqa,,16,0.990432,13.009,271.128,3.3891,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,8,128,128,gqa,,16,88.639648,18.606,775.41,9.6926,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,64,8,128,128,gqa,,16,58.972191,27.967,582.856,7.2857,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,128,128,gqa,,16,2.190848,7.842,245.111,3.0639,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,64,8,128,128,gqa,,16,0.9888,17.374,271.609,3.3951,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,8,128,128,gqa,,16,88.640579,24.808,775.449,9.6931,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,64,8,128,128,gqa,,16,58.998016,37.273,582.672,7.2834,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,8,128,128,gqa,,16,0.197184,348.504,382.878,4.786,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,8,128,128,gqa,,16,1.117056,984.294,270.344,3.3793,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,64,8,128,128,gqa,,16,13.07632,1345.347,92.378,1.1547,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,8,128,128,gqa,,16,1.049824,1047.329,271.677,3.396,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,8,128,128,gqa,,16,1.997888,1100.674,277.117,3.464,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,8,128,128,gqa,,16,1.724416,1275.228,175.126,2.1891,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,8,128,128,gqa,,16,3.465984,1268.917,164.578,2.0572,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,64,8,128,128,gqa,,16,1.920896,2.236,279.507,3.4938,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,64,8,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,64,8,128,128,gqa,,16,242.285095,2.269,283.648,3.5456,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,64,8,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,8,128,128,gqa,,16,1.935072,4.439,277.476,3.4685,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,64,8,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,8,128,128,gqa,,16,243.727966,4.511,281.986,3.5248,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,64,8,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,8,128,128,gqa,,16,1.934624,6.66,277.557,3.4695,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,64,8,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,8,128,128,gqa,,16,242.851715,6.791,283.021,3.5378,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,64,8,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,8,128,128,gqa,,16,1.9368,8.87,277.262,3.4658,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,64,8,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,8,128,128,gqa,,16,244.216537,9.004,281.456,3.5182,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,64,8,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,128,128,gqa,,16,0.056256,610.775,671.017,8.3877,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,128,128,gqa,,16,0.339136,1621.048,445.234,5.5654,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,128,128,gqa,,16,5.728928,1535.382,105.426,1.3178,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,128,128,gqa,,16,0.616256,892.09,231.408,2.8926,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,128,128,gqa,,16,1.194304,920.63,231.787,2.8973,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,128,128,gqa,,16,0.656256,1675.431,230.085,2.8761,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,128,128,gqa,,16,1.27376,1726.403,223.914,2.7989,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,4,128,128,gqa,,16,0.072064,29.8,3725.187,46.5648,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,32,4,128,128,gqa,,16,0.081248,26.431,1652.153,20.6519,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,4,128,128,gqa,,16,4.711648,58.34,7292.955,91.1619,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,32,4,128,128,gqa,,16,2.409664,114.073,7130.441,89.1305,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,128,128,gqa,,16,2.11472,2.031,126.952,1.5869,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,32,4,128,128,gqa,,16,0.989536,4.34,135.67,1.6959,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,128,128,gqa,,16,44.480289,12.36,772.565,9.6571,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,32,4,128,128,gqa,,16,29.41552,18.689,584.184,7.3023,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,128,128,gqa,,16,2.11376,3.048,127.018,1.5877,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,32,4,128,128,gqa,,16,0.988352,6.518,135.849,1.6981,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,128,128,gqa,,16,44.502079,18.53,772.234,9.6529,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,32,4,128,128,gqa,,16,29.42448,28.025,584.077,7.301,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,128,128,gqa,,16,2.117856,4.056,126.78,1.5847,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,32,4,128,128,gqa,,16,0.988928,8.686,135.787,1.6973,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,128,128,gqa,,16,44.503902,24.706,772.25,9.6531,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,32,4,128,128,gqa,,16,29.438272,37.35,583.875,7.2984,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,4,128,128,gqa,,16,0.179328,191.603,210.501,2.6313,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,4,128,128,gqa,,16,0.649216,846.8,232.58,2.9073,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,4,128,128,gqa,,16,6.66176,1320.386,90.664,1.1333,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,4,128,128,gqa,,16,0.868032,633.336,164.287,2.0536,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,4,128,128,gqa,,16,1.640832,670.094,168.71,2.1089,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,4,128,128,gqa,,16,0.895744,1227.484,168.569,2.1071,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,4,128,128,gqa,,16,1.69392,1298.186,168.374,2.1047,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,4,128,128,gqa,,16,1.6184,1.327,165.875,2.0734,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,32,4,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,4,128,128,gqa,,16,119.587616,2.299,287.336,3.5917,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,32,4,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,4,128,128,gqa,,16,1.636128,2.625,164.088,2.0511,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,32,4,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,4,128,128,gqa,,16,120.077988,4.578,286.18,3.5773,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,32,4,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,4,128,128,gqa,,16,1.633088,3.945,164.403,2.055,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,32,4,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,4,128,128,gqa,,16,121.572289,6.783,282.68,3.5335,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,32,4,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,4,128,128,gqa,,16,1.6336,5.258,164.362,2.0545,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,32,4,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,4,128,128,gqa,,16,121.112671,9.078,283.77,3.5471,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,32,4,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,128,128,gqa,,16,0.0528,325.376,357.469,4.4684,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,128,128,gqa,,16,0.18224,1508.329,414.275,5.1784,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,128,128,gqa,,16,2.379008,1848.689,126.939,1.5867,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,128,128,gqa,,16,0.615104,446.88,115.921,1.449,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,128,128,gqa,,16,1.192128,461.155,116.105,1.4513,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,128,128,gqa,,16,0.616128,892.275,122.535,1.5317,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,128,128,gqa,,16,1.195712,919.546,119.265,1.4908,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,2,128,128,gqa,,16,0.053952,19.902,2487.877,31.0985,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,16,2,128,128,gqa,,16,0.07056,15.217,951.205,11.8901,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,2,128,128,gqa,,16,2.385152,57.623,7203.28,90.041,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,16,2,128,128,gqa,,16,1.278528,107.498,6719.433,83.9929,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,128,128,gqa,,16,2.105216,1.02,63.763,0.797,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,16,2,128,128,gqa,,16,0.98432,2.182,68.195,0.8524,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,128,128,gqa,,16,22.392223,12.276,767.318,9.5915,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,16,2,128,128,gqa,,16,14.346752,19.16,598.883,7.486,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,128,128,gqa,,16,2.106048,1.53,63.741,0.7968,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,16,2,128,128,gqa,,16,0.983008,3.277,68.294,0.8537,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,128,128,gqa,,16,22.392672,18.413,767.35,9.5919,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,16,2,128,128,gqa,,16,14.366464,28.7,598.135,7.4767,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,128,128,gqa,,16,2.106208,2.039,63.74,0.7968,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,16,2,128,128,gqa,,16,0.98304,4.369,68.3,0.8538,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,128,128,gqa,,16,22.394079,24.549,767.349,9.5919,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,16,2,128,128,gqa,,16,14.31216,38.412,600.477,7.506,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,2,128,128,gqa,,16,0.148544,115.655,127.062,1.5883,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,2,128,128,gqa,,16,0.390144,704.555,193.512,2.4189,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,2,128,128,gqa,,16,3.144864,1398.485,96.026,1.2003,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,2,128,128,gqa,,16,0.7912,347.419,90.12,1.1265,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,2,128,128,gqa,,16,1.49104,368.706,92.829,1.1604,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,2,128,128,gqa,,16,0.797792,689.097,94.633,1.1829,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,2,128,128,gqa,,16,1.495392,735.267,95.364,1.192,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,2,128,128,gqa,,16,1.466176,0.732,91.548,1.1444,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,16,2,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,2,128,128,gqa,,16,60.026432,2.29,286.223,3.5778,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,16,2,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,2,128,128,gqa,,16,1.485184,1.446,90.382,1.1298,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,16,2,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,2,128,128,gqa,,16,59.546272,4.616,288.548,3.6069,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,16,2,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,2,128,128,gqa,,16,1.487296,2.166,90.259,1.1282,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,16,2,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,2,128,128,gqa,,16,59.659294,6.911,288.019,3.6002,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,16,2,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,2,128,128,gqa,,16,1.485408,2.891,90.38,1.1297,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,16,2,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,2,128,128,gqa,,16,60.03088,9.158,286.254,3.5782,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,16,2,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,128,128,gqa,,16,0.051232,167.667,184.205,2.3026,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,128,128,gqa,,16,0.111488,1232.769,338.59,4.2324,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,128,128,gqa,,16,1.195584,1839.288,126.294,1.5787,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,128,128,gqa,,16,0.6128,224.28,58.178,0.7272,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,128,128,gqa,,16,1.190976,230.801,58.109,0.7264,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,128,128,gqa,,16,0.613728,447.882,61.507,0.7688,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,128,128,gqa,,16,1.19232,461.081,59.802,0.7475,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,128,128,gqa,,16,0.048384,11.096,1387.09,17.3386,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,8,1,128,128,gqa,,16,0.068768,7.807,487.996,6.1,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,128,128,gqa,,16,1.207648,56.904,7113.38,88.9172,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,8,1,128,128,gqa,,16,inf,0.0,0.0,,20,False," File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 266, in setup_cudnn_fp8 - graph.build_plans() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1454, in build_plans - raise cudnn_graph_not_supported(""no plan in the list could be built:\n "" + ""\n "".join(failures or [""the plan list is empty""])) -cudnn._compiled_module.cudnnGraphNotSupportedError: no plan in the list could be built: - [0] eng10_k24=1_k27=0_k38=0_k40=3_k41=2: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor FailedEncountered runtime kernel compilation failure at: compilationResult != NVRTC_SUCCESS cudnn_status: CUDNN_STATUS_INTERNAL_ERROR_COMPILATION_FAILED",,, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,128,128,gqa,,16,1.986176,0.541,33.792,0.4224,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,8,1,128,128,gqa,,16,0.981312,1.094,34.202,0.4275,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,128,128,gqa,,16,11.41712,12.038,752.465,9.4058,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,8,1,128,128,gqa,,16,7.200256,19.088,596.648,7.4581,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,128,128,gqa,,16,1.986208,0.811,33.794,0.4224,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,8,1,128,128,gqa,,16,0.98208,1.64,34.179,0.4272,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,128,128,gqa,,16,11.41392,18.062,752.722,9.409,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,8,1,128,128,gqa,,16,7.1384,28.88,601.891,7.5236,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,128,128,gqa,,16,1.984576,1.082,33.823,0.4228,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,8,1,128,128,gqa,,16,0.981568,2.188,34.201,0.4275,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,128,128,gqa,,16,11.417472,24.075,752.534,9.4067,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,8,1,128,128,gqa,,16,7.14352,38.479,601.533,7.5192,20,True,,NVIDIA B300 SXM6 AC,cudnn fp8 9.25.0, -llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,128,128,gqa,,16,0.12384,69.363,76.205,0.9526,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,128,128,gqa,,16,0.269888,509.244,139.868,1.7484,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,128,128,gqa,,16,1.696832,1295.958,88.986,1.1123,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,128,128,gqa,,16,0.726464,189.189,49.076,0.6134,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,128,128,gqa,,16,1.35392,203.024,51.115,0.6389,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,128,128,gqa,,16,0.728704,377.215,51.803,0.6475,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,128,128,gqa,,16,1.350976,406.932,52.779,0.6597,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,8,1,128,128,gqa,,16,1.318624,0.407,50.896,0.6362,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,8,1,128,128,gqa,,16,1.380448,0.389,24.31,0.3039,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss fp8 plan=sdpa_fwd_prefill_sm100_d128_fp8[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,8,1,128,128,gqa,,16,19.924801,3.449,431.144,5.3893,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,8,1,128,128,gqa,,16,15.409888,4.459,278.749,3.4844,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss fp8 plan=sdpa_fwd_prefill_sm100_d128_fp8[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,128,128,gqa,,16,1.348416,0.796,49.775,0.6222,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,8,1,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,128,128,gqa,,16,19.821217,6.934,433.424,5.4178,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,8,1,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,128,128,gqa,,16,1.3448,1.198,49.912,0.6239,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,8,1,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,128,128,gqa,,16,20.035776,10.29,428.808,5.3601,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,8,1,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,128,128,gqa,,16,1.345376,1.596,49.893,0.6237,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,8,1,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,128,128,gqa,,16,20.054785,13.706,428.428,5.3554,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,8,1,128,128,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 265, in setup_cudnn_fp8 - graph.check_support() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 1389, in check_support - self._lowered_graph.check_support() -cudnn._compiled_module.cudnnGraphNotSupportedError: [cudnn_frontend] Error: No execution plans support the graph.",,, diff --git a/benchmark/attention_inference/results/llama3.1/b300/llama3.1_context.png b/benchmark/attention_inference/results/llama3.1/b300/llama3.1_context.png deleted file mode 100644 index c0cb756c2..000000000 Binary files a/benchmark/attention_inference/results/llama3.1/b300/llama3.1_context.png and /dev/null differ diff --git a/benchmark/attention_inference/results/llama3.1/b300/llama3.1_generation.png b/benchmark/attention_inference/results/llama3.1/b300/llama3.1_generation.png deleted file mode 100644 index 5ec92ceab..000000000 Binary files a/benchmark/attention_inference/results/llama3.1/b300/llama3.1_generation.png and /dev/null differ diff --git a/benchmark/attention_inference/results/llama3.1/gb200/llama3.1_20260818_130938.csv b/benchmark/attention_inference/results/llama3.1/gb200/llama3.1_20260818_130938.csv new file mode 100644 index 000000000..cbbfe40eb --- /dev/null +++ b/benchmark/attention_inference/results/llama3.1/gb200/llama3.1_20260818_130938.csv @@ -0,0 +1,148 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,128,128,gqa,,16,0.091392,751.92,826.084,10.3261,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,128,128,gqa,,16,0.737408,1491.049,409.529,5.1191,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,128,128,gqa,,16,12.67456,1387.992,95.306,1.1913,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,128,128,gqa,,16,0.785952,1398.955,362.888,4.5361,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,128,128,gqa,,16,1.472096,1493.804,376.095,4.7012,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,128,128,gqa,,16,1.50656,1459.632,200.45,2.5056,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,128,128,gqa,,16,3.061088,1436.759,186.347,2.3293,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,8,128,128,gqa,,16,0.103648,41.438,5180.068,64.7508,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,64,8,128,128,gqa,,16,0.100768,42.622,2664.058,33.3007,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,8,128,128,gqa,,16,9.005376,61.048,7631.405,95.3926,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,128,128,gqa,,16,2.19424,3.915,244.703,3.0588,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,64,8,128,128,gqa,,16,1.262176,6.806,212.703,2.6588,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,8,128,128,gqa,,16,83.90387,13.104,819.126,10.2391,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,128,128,gqa,,16,2.193664,5.874,244.782,3.0598,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,64,8,128,128,gqa,,16,1.25936,10.231,213.191,2.6649,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,8,128,128,gqa,,16,83.903748,19.657,819.177,10.2397,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,128,128,gqa,,16,2.193088,7.834,244.861,3.0608,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,64,8,128,128,gqa,,16,1.26128,13.621,212.88,2.661,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,8,128,128,gqa,,16,83.911263,26.207,819.154,10.2394,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,8,128,128,gqa,,16,0.228768,300.389,330.018,4.1252,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,8,128,128,gqa,,16,1.160672,947.306,260.185,3.2523,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,64,8,128,128,gqa,,16,13.677056,1286.255,88.32,1.104,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,8,128,128,gqa,,16,1.163296,945.169,245.176,3.0647,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,8,128,128,gqa,,16,2.24048,981.497,247.111,3.0889,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,8,128,128,gqa,,16,1.964544,1119.356,153.72,1.9215,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,8,128,128,gqa,,16,3.844672,1143.933,148.368,1.8546,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,64,8,128,128,gqa,,16,2.177376,1.973,246.583,3.0823,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,64,8,128,128,gqa,,16,249.974686,2.199,274.923,3.4365,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,8,128,128,gqa,,16,2.217312,3.874,242.156,3.027,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,8,128,128,gqa,,16,251.968643,4.364,272.764,3.4095,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,8,128,128,gqa,,16,2.213152,5.822,242.626,3.0328,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,8,128,128,gqa,,16,251.305283,6.563,273.5,3.4188,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,8,128,128,gqa,,16,2.217952,7.746,242.116,3.0265,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,8,128,128,gqa,,16,251.351486,8.749,273.467,3.4183,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,128,128,gqa,,16,0.059328,579.149,636.272,7.9534,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,128,128,gqa,,16,0.400224,1373.62,377.276,4.716,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,128,128,gqa,,16,6.34944,1385.334,95.123,1.189,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,128,128,gqa,,16,0.72016,763.38,198.02,2.4753,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,128,128,gqa,,16,1.399552,785.617,197.795,2.4724,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,128,128,gqa,,16,0.768832,1430.106,196.395,2.4549,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,128,128,gqa,,16,1.477248,1488.595,193.07,2.4134,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,4,128,128,gqa,,16,0.068544,31.33,3916.489,48.9561,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,32,4,128,128,gqa,,16,0.074208,28.939,1808.78,22.6097,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,4,128,128,gqa,,16,4.52352,60.766,7596.26,94.9533,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,32,4,128,128,gqa,,16,2.299808,119.522,7470.588,93.3823,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,128,128,gqa,,16,2.19168,1.96,122.494,1.5312,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,32,4,128,128,gqa,,16,1.2592,3.411,106.603,1.3325,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,128,128,gqa,,16,42.249569,13.012,813.356,10.1669,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,32,4,128,128,gqa,,16,33.183521,16.567,517.786,6.4723,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,128,128,gqa,,16,2.189856,2.942,122.604,1.5325,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,32,4,128,128,gqa,,16,1.25888,5.118,106.636,1.333,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,128,128,gqa,,16,42.243103,19.521,813.53,10.1691,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,32,4,128,128,gqa,,16,33.042271,24.957,520.031,6.5004,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,128,128,gqa,,16,2.189824,3.923,122.613,1.5327,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,32,4,128,128,gqa,,16,1.257984,6.828,106.719,1.334,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,128,128,gqa,,16,42.243935,26.028,813.564,10.1695,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,32,4,128,128,gqa,,16,33.588482,32.735,511.606,6.3951,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,4,128,128,gqa,,16,0.182016,188.773,207.392,2.5924,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,4,128,128,gqa,,16,0.6288,874.294,240.132,3.0016,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,4,128,128,gqa,,16,6.754944,1302.171,89.413,1.1177,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,4,128,128,gqa,,16,1.003616,547.775,142.093,1.7762,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,4,128,128,gqa,,16,1.92688,570.618,143.664,1.7958,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,4,128,128,gqa,,16,1.012608,1085.822,149.115,1.8639,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,4,128,128,gqa,,16,1.940768,1133.069,146.959,1.837,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,4,128,128,gqa,,16,1.883136,1.14,142.556,1.7819,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,4,128,128,gqa,,16,124.00576,2.217,277.099,3.4637,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,4,128,128,gqa,,16,1.918592,2.239,139.93,1.7491,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,4,128,128,gqa,,16,125.102013,4.394,274.687,3.4336,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,4,128,128,gqa,,16,1.923968,3.349,139.547,1.7443,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,4,128,128,gqa,,16,125.131844,6.59,274.639,3.433,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,4,128,128,gqa,,16,1.92096,4.472,139.774,1.7472,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,4,128,128,gqa,,16,124.894142,8.804,275.178,3.4397,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,128,128,gqa,,16,0.057728,297.6,326.953,4.0869,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,128,128,gqa,,16,0.21776,1262.298,346.7,4.3338,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,128,128,gqa,,16,2.577792,1706.129,117.151,1.4644,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,128,128,gqa,,16,0.720128,381.707,99.015,1.2377,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,128,128,gqa,,16,1.399456,392.835,98.904,1.2363,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,128,128,gqa,,16,0.72336,760.003,104.371,1.3046,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,128,128,gqa,,16,1.400928,784.845,101.794,1.2724,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,2,128,128,gqa,,16,0.0504,21.304,2663.213,33.2902,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,16,2,128,128,gqa,,16,0.06464,16.611,1038.257,12.9782,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,2,128,128,gqa,,16,2.2696,60.556,7570.02,94.6253,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,16,2,128,128,gqa,,16,1.18176,116.3,7269.208,90.8651,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,128,128,gqa,,16,2.18512,0.983,61.431,0.7679,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,16,2,128,128,gqa,,16,1.24912,1.719,53.731,0.6716,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,128,128,gqa,,16,21.793184,12.613,788.41,9.8551,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,16,2,128,128,gqa,,16,16.913248,16.252,507.944,6.3493,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,128,128,gqa,,16,2.185184,1.474,61.433,0.7679,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,16,2,128,128,gqa,,16,1.249856,2.577,53.703,0.6713,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,128,128,gqa,,16,21.790815,18.922,788.544,9.8568,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,16,2,128,128,gqa,,16,16.964737,24.304,506.433,6.3304,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,128,128,gqa,,16,2.18528,1.965,61.434,0.7679,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,16,2,128,128,gqa,,16,1.248032,3.441,53.785,0.6723,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,128,128,gqa,,16,21.797825,25.221,788.338,9.8542,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,16,2,128,128,gqa,,16,16.923391,32.485,507.702,6.3463,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,2,128,128,gqa,,16,0.16032,107.16,117.729,1.4716,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,2,128,128,gqa,,16,0.360448,762.601,209.455,2.6182,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,2,128,128,gqa,,16,3.159648,1391.942,95.577,1.1947,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,2,128,128,gqa,,16,0.925152,297.116,77.072,0.9634,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,2,128,128,gqa,,16,1.774784,309.759,77.988,0.9749,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,2,128,128,gqa,,16,0.924384,594.727,81.673,1.0209,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,2,128,128,gqa,,16,1.779392,617.914,80.143,1.0018,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,2,128,128,gqa,,16,1.732864,0.62,77.459,0.9682,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,2,128,128,gqa,,16,61.714687,2.227,278.393,3.4799,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,2,128,128,gqa,,16,1.772032,1.212,75.752,0.9469,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,2,128,128,gqa,,16,62.1264,4.424,276.565,3.4571,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,2,128,128,gqa,,16,1.776256,1.813,75.576,0.9447,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,2,128,128,gqa,,16,62.130783,6.636,276.562,3.457,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,2,128,128,gqa,,16,1.771104,2.425,75.8,0.9475,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,2,128,128,gqa,,16,62.157505,8.845,276.46,3.4557,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,128,128,gqa,,16,0.054368,157.996,173.58,2.1697,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,128,128,gqa,,16,0.132384,1038.184,285.146,3.5643,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,128,128,gqa,,16,1.33136,1651.712,113.414,1.4177,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,128,128,gqa,,16,0.717472,191.56,49.691,0.6211,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,128,128,gqa,,16,1.39296,197.334,49.683,0.621,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,128,128,gqa,,16,0.720352,381.588,52.403,0.655,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,128,128,gqa,,16,1.399456,392.835,50.951,0.6369,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,128,128,gqa,,16,0.049376,10.873,1359.222,16.9903,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,8,1,128,128,gqa,,16,0.063168,8.499,531.226,6.6403,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,128,128,gqa,,16,1.140992,60.228,7528.938,94.1117,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,128,128,gqa,,16,1.862496,0.577,36.036,0.4505,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,8,1,128,128,gqa,,16,1.248128,0.86,26.887,0.3361,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,128,128,gqa,,16,11.366848,12.091,755.793,9.4474,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,8,1,128,128,gqa,,16,8.52272,16.126,504.005,6.3001,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,128,128,gqa,,16,1.8616,0.865,36.056,0.4507,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,8,1,128,128,gqa,,16,1.248128,1.29,26.889,0.3361,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,128,128,gqa,,16,11.368832,18.134,755.707,9.4463,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,8,1,128,128,gqa,,16,8.495584,24.267,505.646,6.3206,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,128,128,gqa,,16,1.860736,1.154,36.075,0.4509,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,8,1,128,128,gqa,,16,1.249984,1.718,26.85,0.3356,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,128,128,gqa,,16,11.366368,24.183,755.917,9.449,20,True,,NVIDIA GB200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,8,1,128,128,gqa,,16,8.496128,32.353,505.644,6.3205,20,True,,NVIDIA GB200,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,128,128,gqa,,16,0.126592,67.855,74.548,0.9319,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,128,128,gqa,,16,0.270112,508.822,139.752,1.7469,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,128,128,gqa,,16,1.594144,1379.438,94.719,1.184,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,128,128,gqa,,16,0.86864,158.223,41.043,0.513,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,128,128,gqa,,16,1.648992,166.695,41.969,0.5246,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,128,128,gqa,,16,0.87184,315.285,43.298,0.5412,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,128,128,gqa,,16,1.647808,333.629,43.272,0.5409,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,8,1,128,128,gqa,,16,1.580672,0.34,42.458,0.5307,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,8,1,128,128,gqa,,16,21.524832,3.193,399.095,4.9887,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,128,128,gqa,,16,1.649952,0.651,40.678,0.5085,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,128,128,gqa,,16,21.807648,6.302,393.944,4.9243,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,128,128,gqa,,16,1.64496,0.979,40.804,0.5101,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,128,128,gqa,,16,21.811808,9.452,393.893,4.9237,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,128,128,gqa,,16,1.647104,1.304,40.753,0.5094,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,128,128,gqa,,16,21.815424,12.6,393.851,4.9231,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], diff --git a/benchmark/attention_inference/results/llama3.1/gb200/llama3.1_context.png b/benchmark/attention_inference/results/llama3.1/gb200/llama3.1_context.png new file mode 100644 index 000000000..1594aae43 Binary files /dev/null and b/benchmark/attention_inference/results/llama3.1/gb200/llama3.1_context.png differ diff --git a/benchmark/attention_inference/results/llama3.1/gb200/llama3.1_generation.png b/benchmark/attention_inference/results/llama3.1/gb200/llama3.1_generation.png new file mode 100644 index 000000000..a3c8d5961 Binary files /dev/null and b/benchmark/attention_inference/results/llama3.1/gb200/llama3.1_generation.png differ diff --git a/benchmark/attention_inference/results/llama3.1/gb300/llama3.1_20260818_131600.csv b/benchmark/attention_inference/results/llama3.1/gb300/llama3.1_20260818_131600.csv new file mode 100644 index 000000000..7ae74dcf5 --- /dev/null +++ b/benchmark/attention_inference/results/llama3.1/gb300/llama3.1_20260818_131600.csv @@ -0,0 +1,151 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,128,128,gqa,,16,0.077664,884.83,972.104,12.1513,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,128,128,gqa,,16,0.590272,1862.72,511.611,6.3951,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,128,128,gqa,,16,9.369152,1877.671,128.929,1.6116,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,128,128,gqa,,16,0.622432,1766.477,458.223,5.7278,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,128,128,gqa,,16,1.207136,1821.686,458.646,5.7331,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,128,128,gqa,,16,1.22368,1797.057,246.788,3.0849,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,128,128,gqa,,16,2.418784,1818.288,235.831,2.9479,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,8,128,128,gqa,,16,0.10592,40.549,5068.955,63.3619,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,64,8,128,128,gqa,,16,0.095712,44.874,2804.788,35.0598,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,8,128,128,gqa,,16,9.127456,60.231,7529.335,94.1167,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,128,128,gqa,,16,2.187392,3.927,245.469,3.0684,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,64,8,128,128,gqa,,16,0.961824,8.931,279.124,3.4891,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,8,128,128,gqa,,16,83.423615,13.18,823.842,10.298,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,64,8,128,128,gqa,,16,50.839554,21.627,675.929,8.4491,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,128,128,gqa,,16,2.189824,5.884,245.211,3.0651,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,64,8,128,128,gqa,,16,0.964416,13.36,278.391,3.4799,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,8,128,128,gqa,,16,83.422531,19.77,823.903,10.2988,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,64,8,128,128,gqa,,16,50.842335,32.439,675.933,8.4492,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,128,128,gqa,,16,2.188608,7.85,245.362,3.067,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,64,8,128,128,gqa,,16,0.963104,17.838,278.787,3.4848,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,8,128,128,gqa,,16,83.419838,26.361,823.98,10.2997,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,64,8,128,128,gqa,,16,50.845886,43.249,675.927,8.4491,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,8,128,128,gqa,,16,0.22064,311.455,342.175,4.2772,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,8,128,128,gqa,,16,1.033216,1064.164,292.281,3.6535,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,64,8,128,128,gqa,,16,10.947488,1606.961,110.341,1.3793,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,8,128,128,gqa,,16,1.035072,1062.256,275.549,3.4444,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,8,128,128,gqa,,16,1.944384,1130.961,284.742,3.5593,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,8,128,128,gqa,,16,1.694624,1297.647,178.205,2.2276,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,8,128,128,gqa,,16,3.246784,1354.586,175.689,2.1961,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,64,8,128,128,gqa,,16,1.859936,2.309,288.668,3.6083,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,64,8,128,128,gqa,,16,216.288773,2.542,317.74,3.9718,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,8,128,128,gqa,,16,1.912096,4.492,280.81,3.5101,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,64,8,128,128,gqa,,16,216.997986,5.067,316.721,3.959,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,8,128,128,gqa,,16,1.913824,6.733,280.574,3.5072,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,64,8,128,128,gqa,,16,216.990952,7.601,316.751,3.9594,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,8,128,128,gqa,,16,1.905888,9.014,281.759,3.522,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,64,8,128,128,gqa,,16,216.979782,10.135,316.786,3.9598,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,128,128,gqa,,16,0.054112,634.974,697.604,8.72,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,128,128,gqa,,16,0.31072,1769.297,485.952,6.0744,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,128,128,gqa,,16,4.223392,2082.708,143.008,1.7876,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,128,128,gqa,,16,0.604288,909.758,235.991,2.9499,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,128,128,gqa,,16,1.168096,941.285,236.987,2.9623,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,128,128,gqa,,16,0.61184,1797.057,246.788,3.0849,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,128,128,gqa,,16,1.195872,1838.845,238.498,2.9812,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,4,128,128,gqa,,16,0.069632,30.84,3855.294,48.1912,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,32,4,128,128,gqa,,16,0.069184,31.04,1940.13,24.2516,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,4,128,128,gqa,,16,4.643264,59.199,7400.363,92.5045,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,32,4,128,128,gqa,,16,2.361984,116.376,7273.935,90.9242,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,128,128,gqa,,16,2.1904,1.961,122.566,1.5321,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,32,4,128,128,gqa,,16,0.9616,4.466,139.595,1.7449,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,128,128,gqa,,16,41.936031,13.109,819.437,10.243,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,32,4,128,128,gqa,,16,25.126623,21.879,683.815,8.5477,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,128,128,gqa,,16,2.190464,2.941,122.57,1.5321,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,32,4,128,128,gqa,,16,0.961984,6.697,139.547,1.7443,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,128,128,gqa,,16,41.910271,19.676,819.991,10.2499,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,32,4,128,128,gqa,,16,25.160513,32.775,682.936,8.5367,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,128,128,gqa,,16,2.189536,3.923,122.629,1.5329,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,32,4,128,128,gqa,,16,0.96432,8.908,139.218,1.7402,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,128,128,gqa,,16,41.934048,26.22,819.576,10.2447,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,32,4,128,128,gqa,,16,25.169888,43.684,682.723,8.534,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,4,128,128,gqa,,16,0.184672,186.058,204.41,2.5551,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,4,128,128,gqa,,16,0.550048,999.469,274.512,3.4314,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,4,128,128,gqa,,16,5.415552,1624.228,111.527,1.3941,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,4,128,128,gqa,,16,0.869312,632.403,164.045,2.0506,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,4,128,128,gqa,,16,1.6248,676.706,170.374,2.1297,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,4,128,128,gqa,,16,0.877216,1253.41,172.13,2.1516,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,4,128,128,gqa,,16,1.648896,1333.634,172.972,2.1621,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,4,128,128,gqa,,16,1.566752,1.371,171.343,2.1418,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,4,128,128,gqa,,16,107.05027,2.568,320.988,4.0123,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,4,128,128,gqa,,16,1.617856,2.655,165.941,2.0743,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,4,128,128,gqa,,16,107.254944,5.126,320.395,4.0049,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,4,128,128,gqa,,16,1.617536,3.983,165.984,2.0748,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,4,128,128,gqa,,16,107.606239,7.663,319.368,3.9921,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,4,128,128,gqa,,16,1.618176,5.308,165.928,2.0741,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,4,128,128,gqa,,16,107.711105,10.208,319.077,3.9885,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,128,128,gqa,,16,0.051968,330.586,363.192,4.5399,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,128,128,gqa,,16,0.169248,1624.113,446.076,5.576,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,128,128,gqa,,16,2.124064,2070.581,142.176,1.7772,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,128,128,gqa,,16,0.603744,455.289,118.102,1.4763,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,128,128,gqa,,16,1.167264,470.978,118.578,1.4822,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,128,128,gqa,,16,0.603168,911.447,125.168,1.5646,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,128,128,gqa,,16,1.167296,941.93,122.168,1.5271,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,2,128,128,gqa,,16,0.05312,20.214,2526.843,31.5855,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,16,2,128,128,gqa,,16,0.063104,17.015,1063.529,13.2941,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,2,128,128,gqa,,16,2.336768,58.816,7352.428,91.9053,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,16,2,128,128,gqa,,16,1.233632,111.41,6963.551,87.0444,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,128,128,gqa,,16,2.184128,0.983,61.459,0.7682,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,16,2,128,128,gqa,,16,0.95424,2.25,70.336,0.8792,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,128,128,gqa,,16,21.855232,12.577,786.172,9.8271,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,16,2,128,128,gqa,,16,12.845792,21.398,668.778,8.3597,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,128,128,gqa,,16,2.18528,1.474,61.43,0.7679,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,16,2,128,128,gqa,,16,0.961792,3.349,69.788,0.8723,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,128,128,gqa,,16,21.860704,18.861,786.023,9.8253,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,16,2,128,128,gqa,,16,12.84672,32.095,668.771,8.3596,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,128,128,gqa,,16,2.182464,1.968,61.513,0.7689,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,16,2,128,128,gqa,,16,0.955072,4.497,70.283,0.8785,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,128,128,gqa,,16,21.860256,25.149,786.087,9.8261,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,16,2,128,128,gqa,,16,12.848704,42.787,668.708,8.3589,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,2,128,128,gqa,,16,0.154144,111.453,122.446,1.5306,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,2,128,128,gqa,,16,0.325088,845.549,232.237,2.903,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,2,128,128,gqa,,16,2.664896,1650.363,113.321,1.4165,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,2,128,128,gqa,,16,0.793568,346.382,89.851,1.1231,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,2,128,128,gqa,,16,1.4744,372.867,93.877,1.1735,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,2,128,128,gqa,,16,0.798272,688.682,94.576,1.1822,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,2,128,128,gqa,,16,1.472416,746.74,96.852,1.2106,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,2,128,128,gqa,,16,1.418304,0.757,94.638,1.183,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,2,128,128,gqa,,16,53.024418,2.592,324.019,4.0502,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,2,128,128,gqa,,16,1.473504,1.457,91.099,1.1387,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,2,128,128,gqa,,16,53.762943,5.113,319.588,3.9948,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,2,128,128,gqa,,16,1.470816,2.19,91.271,1.1409,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,2,128,128,gqa,,16,53.738689,7.673,319.751,3.9969,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,2,128,128,gqa,,16,1.470752,2.92,91.28,1.141,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,2,128,128,gqa,,16,53.768318,10.225,319.595,3.9949,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,128,128,gqa,,16,0.049696,172.85,189.898,2.3737,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,128,128,gqa,,16,0.104832,1311.04,360.088,4.5011,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,128,128,gqa,,16,1.088288,2020.626,138.745,1.7343,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,128,128,gqa,,16,0.601536,228.48,59.268,0.7408,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,128,128,gqa,,16,1.163936,236.162,59.459,0.7432,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,128,128,gqa,,16,0.601856,456.717,62.721,0.784,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,128,128,gqa,,16,1.165728,471.599,61.166,0.7646,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,128,128,gqa,,16,0.046208,11.619,1452.41,18.1551,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,8,1,128,128,gqa,,16,0.0624,8.604,537.764,6.7221,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,128,128,gqa,,16,1.225472,56.076,7009.919,87.624,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,128,128,gqa,,16,1.827584,0.588,36.724,0.4591,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,8,1,128,128,gqa,,16,0.95456,1.125,35.156,0.4395,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,128,128,gqa,,16,11.2064,12.264,766.614,9.5827,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,8,1,128,128,gqa,,16,6.447648,21.316,666.211,8.3276,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,128,128,gqa,,16,1.825568,0.882,36.767,0.4596,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,8,1,128,128,gqa,,16,0.95536,1.686,35.129,0.4391,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,128,128,gqa,,16,11.207008,18.395,766.619,9.5827,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,8,1,128,128,gqa,,16,6.448832,31.968,666.129,8.3266,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,128,128,gqa,,16,1.8256,1.176,36.769,0.4596,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,8,1,128,128,gqa,,16,0.955968,2.246,35.109,0.4389,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,128,128,gqa,,16,11.206304,24.529,766.714,9.5839,20,True,,NVIDIA GB300,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,8,1,128,128,gqa,,16,6.448768,42.625,666.176,8.3272,20,True,,NVIDIA GB300,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,128,128,gqa,,16,0.125728,68.322,75.06,0.9383,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,128,128,gqa,,16,0.236384,581.422,159.692,1.9962,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,128,128,gqa,,16,1.349664,1629.312,111.876,1.3984,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,128,128,gqa,,16,0.736448,186.624,48.41,0.6051,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,128,128,gqa,,16,1.327008,207.141,52.152,0.6519,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,128,128,gqa,,16,0.744832,369.047,50.681,0.6335,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,128,128,gqa,,16,1.349024,407.521,52.855,0.6607,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,8,1,128,128,gqa,,16,1.270112,0.423,52.84,0.6605,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,8,1,128,128,gqa,,16,17.482656,3.931,491.37,6.1421,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,128,128,gqa,,16,1.316992,0.815,50.962,0.637,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,128,128,gqa,,16,17.612288,7.804,487.783,6.0973,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,128,128,gqa,,16,1.322272,1.218,50.762,0.6345,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,128,128,gqa,,16,17.58032,11.727,488.7,6.1088,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,128,128,gqa,,16,1.322624,1.624,50.752,0.6344,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,128,128,gqa,,16,17.576736,15.639,488.83,6.1104,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], diff --git a/benchmark/attention_inference/results/llama3.1/gb300/llama3.1_context.png b/benchmark/attention_inference/results/llama3.1/gb300/llama3.1_context.png new file mode 100644 index 000000000..42a5c7b90 Binary files /dev/null and b/benchmark/attention_inference/results/llama3.1/gb300/llama3.1_context.png differ diff --git a/benchmark/attention_inference/results/llama3.1/gb300/llama3.1_generation.png b/benchmark/attention_inference/results/llama3.1/gb300/llama3.1_generation.png new file mode 100644 index 000000000..e16c409ac Binary files /dev/null and b/benchmark/attention_inference/results/llama3.1/gb300/llama3.1_generation.png differ diff --git a/benchmark/attention_inference/results/llama3.1/h200/llama3.1_20260818_142507.csv b/benchmark/attention_inference/results/llama3.1/h200/llama3.1_20260818_142507.csv new file mode 100644 index 000000000..5c511421c --- /dev/null +++ b/benchmark/attention_inference/results/llama3.1/h200/llama3.1_20260818_142507.csv @@ -0,0 +1,61 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,128,128,gqa,,16,0.16608,413.773,454.585,9.4705,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,128,128,gqa,,16,1.724768,637.484,175.09,3.6477,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,128,128,gqa,,16,28.892481,608.885,41.809,0.871,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,128,128,gqa,,16,1.783648,616.44,159.904,3.3313,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,128,128,gqa,,16,3.640896,603.979,152.064,3.168,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,128,128,gqa,,16,3.52144,624.467,85.757,1.7866,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,128,128,gqa,,16,7.762688,566.562,73.483,1.5309,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,8,128,128,gqa,,16,0.153952,27.898,3487.474,72.6557,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,1,131072,64,8,128,128,gqa,,16,15.240672,36.072,4509.228,93.9423,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,128,128,gqa,,16,1.542688,5.568,348.053,7.2511,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,2,131072,64,8,128,128,gqa,,16,103.439552,10.63,664.425,13.8422,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,128,128,gqa,,16,1.526528,8.441,351.759,7.3283,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,3,131072,64,8,128,128,gqa,,16,103.702019,15.904,662.784,13.808,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,128,128,gqa,,16,1.527232,11.249,351.618,7.3254,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,128,4,131072,64,8,128,128,gqa,,16,103.984482,21.148,661.024,13.7713,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,128,128,gqa,,16,0.104352,329.268,361.744,7.5363,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,128,128,gqa,,16,0.907008,606.12,166.476,3.4682,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,128,128,gqa,,16,14.58096,603.259,41.422,0.863,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,128,128,gqa,,16,0.900704,610.362,158.328,3.2985,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,128,128,gqa,,16,1.767264,622.155,156.64,3.2633,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,128,128,gqa,,16,1.760192,624.654,85.783,1.7872,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,128,128,gqa,,16,3.5184,625.007,81.063,1.6888,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,4,128,128,gqa,,16,0.093184,23.046,2880.879,60.0183,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,4,128,128,gqa,,16,7.614496,36.099,4512.687,94.0143,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,128,128,gqa,,16,1.54016,2.789,174.312,3.6315,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,128,128,gqa,,16,52.070625,10.558,659.949,13.7489,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,128,128,gqa,,16,1.524512,4.226,176.112,3.669,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,128,128,gqa,,16,52.180607,15.803,658.598,13.7208,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,128,128,gqa,,16,1.539936,5.578,174.359,3.6325,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,128,128,gqa,,16,52.198017,21.064,658.418,13.717,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,128,128,gqa,,16,0.07616,225.576,247.825,5.163,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,128,128,gqa,,16,0.500704,548.983,150.783,3.1413,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,128,128,gqa,,16,7.35264,598.159,41.072,0.8557,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,128,128,gqa,,16,0.90112,305.04,79.127,1.6485,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,128,128,gqa,,16,1.765792,311.337,78.385,1.633,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,128,128,gqa,,16,0.915264,600.653,82.487,1.7185,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,128,128,gqa,,16,1.785664,615.744,79.862,1.6638,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,2,128,128,gqa,,16,0.06352,16.904,2113.128,44.0235,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,2,128,128,gqa,,16,3.778304,36.376,4547.256,94.7345,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,128,128,gqa,,16,1.54112,1.393,87.102,1.8146,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,128,128,gqa,,16,25.687649,10.701,668.88,13.935,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,128,128,gqa,,16,1.541408,2.09,87.091,1.8144,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,128,128,gqa,,16,25.783968,15.991,666.422,13.8838,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,128,128,gqa,,16,1.54016,2.789,87.167,1.816,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,128,128,gqa,,16,25.781216,21.324,666.534,13.8861,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,128,128,gqa,,16,0.055776,154.008,169.198,3.525,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,128,128,gqa,,16,0.29856,460.339,126.436,2.6341,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,128,128,gqa,,16,3.480672,631.781,43.381,0.9038,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,128,128,gqa,,16,0.897824,153.08,39.709,0.8273,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,128,128,gqa,,16,1.762976,155.917,39.255,0.8178,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,128,128,gqa,,16,0.90112,305.04,41.891,0.8727,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,128,128,gqa,,16,1.768384,310.88,40.321,0.84,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,128,128,gqa,,16,0.053952,9.951,1243.938,25.9154,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,128,128,gqa,,16,1.901536,36.139,4517.642,94.1175,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,128,128,gqa,,16,1.537184,0.699,43.662,0.9096,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,128,128,gqa,,16,12.841184,10.703,669.018,13.9379,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,128,128,gqa,,16,1.523136,1.057,44.068,0.9181,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,128,128,gqa,,16,12.918528,15.958,665.053,13.8553,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,128,128,gqa,,16,1.534816,1.399,43.735,0.9111,20,True,,NVIDIA H200,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,128,128,gqa,,16,12.855424,21.382,668.358,13.9241,20,True,,NVIDIA H200,cudnn 9.26.0, diff --git a/benchmark/attention_inference/results/llama3.1/h200/llama3.1_context.png b/benchmark/attention_inference/results/llama3.1/h200/llama3.1_context.png new file mode 100644 index 000000000..c8f4b4168 Binary files /dev/null and b/benchmark/attention_inference/results/llama3.1/h200/llama3.1_context.png differ diff --git a/benchmark/attention_inference/results/llama3.1/h200/llama3.1_generation.png b/benchmark/attention_inference/results/llama3.1/h200/llama3.1_generation.png new file mode 100644 index 000000000..d00dc7eaf Binary files /dev/null and b/benchmark/attention_inference/results/llama3.1/h200/llama3.1_generation.png differ diff --git a/benchmark/attention_inference/results/llama3.1/rtxpro6000/llama3.1_20260818_144157.csv b/benchmark/attention_inference/results/llama3.1/rtxpro6000/llama3.1_20260818_144157.csv new file mode 100644 index 000000000..9b3ad6031 --- /dev/null +++ b/benchmark/attention_inference/results/llama3.1/rtxpro6000/llama3.1_20260818_144157.csv @@ -0,0 +1,131 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,2048,2048,64,8,128,128,gqa,,16,0.271264,253.331,278.317,15.5311,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,8192,8192,64,8,128,128,gqa,,16,3.134464,350.781,96.345,5.3764,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,32768,32768,64,8,128,128,gqa,,16,48.161983,365.271,25.081,1.3996,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,512,65536,64,8,128,128,gqa,,16,4.296704,255.897,66.379,3.7042,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,512,131072,64,8,128,128,gqa,,16,8.568,256.655,64.618,3.6059,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,1024,65536,64,8,128,128,gqa,,16,6.418336,342.616,47.051,2.6256,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1,context,cudnn,bfloat16,bfloat16,1,1024,131072,64,8,128,128,gqa,,16,12.81824,343.108,44.501,2.4833,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,1,131072,64,8,128,128,gqa,,16,0.368864,11.644,1455.56,81.2254,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,2,131072,64,8,128,128,gqa,,16,2.387424,3.598,224.902,12.5503,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,64,8,128,128,gqa,,16,2.703936,3.177,99.288,5.5406,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,3,131072,64,8,128,128,gqa,,16,2.38832,5.395,224.831,12.5464,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,64,8,128,128,gqa,,16,2.706912,4.76,99.185,5.5349,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,bfloat16,1,4,131072,64,8,128,128,gqa,,16,2.38752,7.196,224.92,12.5514,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,64,8,128,128,gqa,,16,2.705568,6.35,99.24,5.538,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,64,8,128,128,gqa,,16,0.32144,213.786,234.873,13.1067,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,64,8,128,128,gqa,,16,3.142944,349.835,96.085,5.3619,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,64,8,128,128,gqa,,16,45.15712,389.577,26.75,1.4928,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,64,8,128,128,gqa,,16,3.482848,315.693,81.891,4.5698,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,64,8,128,128,gqa,,16,6.994496,314.393,79.155,4.4171,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,64,8,128,128,gqa,,16,6.513184,337.626,46.366,2.5874,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,64,8,128,128,gqa,,16,13.60944,323.161,41.914,2.3389,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,64,8,128,128,gqa,,16,2.809248,1.529,191.12,10.6652,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,64,8,128,128,gqa,,16,2.807968,3.059,191.219,10.6707,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,64,8,128,128,gqa,,16,2.807616,4.589,191.255,10.6727,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,64,8,128,128,gqa,,16,2.808032,6.118,191.238,10.6718,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,4,128,128,gqa,,16,0.174016,197.452,216.927,12.1053,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,4,128,128,gqa,,16,1.662784,330.624,90.809,5.0674,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,4,128,128,gqa,,16,24.054913,365.667,25.108,1.4011,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,32,4,128,128,gqa,,16,2.162112,254.268,65.957,3.6806,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,32,4,128,128,gqa,,16,4.294848,256.007,64.455,3.5968,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,4,128,128,gqa,,16,4.283392,256.692,35.251,1.9671,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,4,128,128,gqa,,16,8.551648,257.146,33.352,1.8611,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,4,128,128,gqa,,16,0.195296,10.996,1374.59,76.707,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,4,128,128,gqa,,16,22.386368,12.279,1534.945,85.6554,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,4,128,128,gqa,,16,2.384288,1.801,112.599,6.2834,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,32,4,128,128,gqa,,16,2.697888,1.592,49.755,2.7765,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,4,128,128,gqa,,16,49.125793,11.191,699.509,39.0351,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,4,128,128,gqa,,16,2.382432,2.704,112.694,6.2887,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,32,4,128,128,gqa,,16,2.696512,2.389,49.784,2.7781,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,4,128,128,gqa,,16,49.200066,16.761,698.496,38.9785,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,4,128,128,gqa,,16,2.382016,3.606,112.72,6.2902,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,32,4,128,128,gqa,,16,2.696064,3.186,49.795,2.7787,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,4,128,128,gqa,,16,49.310303,22.298,696.977,38.8938,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,4,128,128,gqa,,16,0.185184,185.544,203.844,11.3752,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,4,128,128,gqa,,16,1.568928,350.402,96.241,5.3706,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,4,128,128,gqa,,16,22.476768,391.342,26.871,1.4995,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,4,128,128,gqa,,16,2.260448,243.207,63.088,3.5205,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,4,128,128,gqa,,16,4.46128,246.457,62.05,3.4626,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,4,128,128,gqa,,16,3.278272,335.394,46.059,2.5703,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,4,128,128,gqa,,16,6.550336,335.712,43.542,2.4298,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,4,128,128,gqa,,16,2.412224,0.89,111.288,6.2103,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,4,128,128,gqa,,16,226.565735,1.213,151.664,8.4634,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,4,128,128,gqa,,16,2.407008,1.784,111.536,6.2241,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,4,128,128,gqa,,16,132.844452,4.138,258.678,14.4352,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,4,128,128,gqa,,16,2.406144,2.678,111.583,6.2267,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,4,128,128,gqa,,16,132.834854,6.208,258.712,14.4371,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,4,128,128,gqa,,16,2.4056,3.571,111.615,6.2285,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,4,128,128,gqa,,16,132.847015,8.277,258.705,14.4366,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,2,128,128,gqa,,16,0.111616,153.919,169.101,9.4364,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,2,128,128,gqa,,16,0.917408,299.624,82.294,4.5923,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,2,128,128,gqa,,16,12.028096,365.648,25.107,1.4011,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,16,2,128,128,gqa,,16,2.152,127.731,33.133,1.849,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,16,2,128,128,gqa,,16,4.286208,128.262,32.292,1.802,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,2,128,128,gqa,,16,2.156512,254.928,35.009,1.9536,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,2,128,128,gqa,,16,4.292544,256.145,33.222,1.8539,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,2,128,128,gqa,,16,0.048224,22.266,2783.384,155.3228,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,2,128,128,gqa,,16,11.181312,12.292,1536.574,85.7463,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,2,128,128,gqa,,16,2.3568,0.911,56.956,3.1784,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,16,2,128,128,gqa,,16,2.562912,0.838,26.188,1.4614,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,2,128,128,gqa,,16,24.627808,11.161,697.665,38.9322,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,16,2,128,128,gqa,,16,29.450209,9.334,291.712,16.2786,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,2,128,128,gqa,,16,2.359744,1.365,56.889,3.1746,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,16,2,128,128,gqa,,16,2.706304,1.19,24.802,1.384,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,2,128,128,gqa,,16,24.679359,16.707,696.25,38.8533,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,16,2,128,128,gqa,,16,29.63344,13.914,289.926,16.1789,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,2,128,128,gqa,,16,2.357184,1.822,56.954,3.1782,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,16,2,128,128,gqa,,16,2.70112,1.59,24.851,1.3868,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,2,128,128,gqa,,16,24.731745,22.229,694.818,38.7733,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,16,2,128,128,gqa,,16,29.664991,18.532,289.635,16.1627,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,2,128,128,gqa,,16,0.123488,139.122,152.844,8.5292,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,2,128,128,gqa,,16,0.800928,343.199,94.262,5.2602,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,2,128,128,gqa,,16,11.10224,396.14,27.201,1.5179,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,2,128,128,gqa,,16,1.114272,246.688,63.991,3.5709,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,2,128,128,gqa,,16,2.232896,246.208,61.988,3.4591,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,2,128,128,gqa,,16,2.148672,255.858,35.137,1.9608,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,2,128,128,gqa,,16,4.241248,259.242,33.624,1.8763,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,2,128,128,gqa,,16,2.233856,0.481,60.087,3.3531,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,2,128,128,gqa,,16,109.101181,1.26,157.477,8.7878,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,2,128,128,gqa,,16,2.223904,0.966,60.36,3.3683,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,2,128,128,gqa,,16,66.232323,4.15,259.42,14.4765,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,2,128,128,gqa,,16,2.22384,1.448,60.365,3.3686,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,2,128,128,gqa,,16,66.192833,6.229,259.59,14.4861,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,2,128,128,gqa,,16,2.224224,1.931,60.358,3.3682,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,2,128,128,gqa,,16,66.200127,8.304,259.578,14.4854,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,128,128,gqa,,16,0.083552,102.809,112.95,6.303,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,128,128,gqa,,16,0.547008,251.256,69.009,3.851,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,128,128,gqa,,16,6.29632,349.255,23.981,1.3383,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,128,128,gqa,,16,2.15136,63.885,16.572,0.9248,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,128,128,gqa,,16,4.28528,64.145,16.15,0.9012,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,128,128,gqa,,16,2.152704,127.69,17.535,0.9785,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,128,128,gqa,,16,4.287072,128.236,16.632,0.9281,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,128,128,gqa,,16,0.039168,13.707,1713.464,95.6174,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,128,128,gqa,,16,5.599296,12.273,1534.203,85.614,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,128,128,gqa,,16,2.347936,0.457,28.586,1.5952,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,8,1,128,128,gqa,,16,2.697728,0.398,12.44,0.6942,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,128,128,gqa,,16,12.990688,10.58,661.319,36.9039,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,8,1,128,128,gqa,,16,15.90176,8.643,270.127,15.074,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,128,128,gqa,,16,2.34784,0.686,28.588,1.5953,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,8,1,128,128,gqa,,16,2.695968,0.597,12.448,0.6947,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,128,128,gqa,,16,12.972544,15.892,662.284,36.9578,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,8,1,128,128,gqa,,16,15.900768,12.965,270.16,15.0759,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,128,128,gqa,,16,2.348,0.915,28.588,1.5953,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,8,1,128,128,gqa,,16,2.695936,0.797,12.449,0.6947,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,128,128,gqa,,16,13.024384,21.105,659.688,36.813,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +llama3.1,llama3.1-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,8,1,128,128,gqa,,16,15.889408,17.299,270.37,15.0876,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn fp8 9.26.0, +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,128,128,gqa,,16,0.083328,103.086,113.253,6.3199,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,128,128,gqa,,16,0.44304,310.218,85.204,4.7547,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,128,128,gqa,,16,5.34704,411.26,28.239,1.5758,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,128,128,gqa,,16,1.041984,131.901,34.215,1.9093,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,128,128,gqa,,16,2.039328,134.788,33.936,1.8937,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,128,128,gqa,,16,1.099008,250.115,34.348,1.9167,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,128,128,gqa,,16,2.051968,267.916,34.749,1.9391,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,8,1,128,128,gqa,,16,2.045504,0.262,32.81,1.8309,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,8,1,128,128,gqa,,16,42.740768,1.608,200.99,11.2159,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,128,128,gqa,,16,2.040864,0.526,32.887,1.8352,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,128,128,gqa,,16,22.516768,6.104,381.537,21.2911,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,128,128,gqa,,16,2.040224,0.789,32.899,1.8359,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,128,128,gqa,,16,22.507839,9.159,381.712,21.3009,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,128,128,gqa,,16,2.0408,1.052,32.892,1.8355,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=128; cga=1)], +llama3.1,llama3.1-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,128,128,gqa,,16,22.518433,12.207,381.555,21.2922,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], diff --git a/benchmark/attention_inference/results/llama3.1/rtxpro6000/llama3.1_context.png b/benchmark/attention_inference/results/llama3.1/rtxpro6000/llama3.1_context.png new file mode 100644 index 000000000..9917715fe Binary files /dev/null and b/benchmark/attention_inference/results/llama3.1/rtxpro6000/llama3.1_context.png differ diff --git a/benchmark/attention_inference/results/llama3.1/rtxpro6000/llama3.1_generation.png b/benchmark/attention_inference/results/llama3.1/rtxpro6000/llama3.1_generation.png new file mode 100644 index 000000000..f1749152f Binary files /dev/null and b/benchmark/attention_inference/results/llama3.1/rtxpro6000/llama3.1_generation.png differ diff --git a/benchmark/attention_inference/results/qwen35/b300/qwen35_20260817_133148.csv b/benchmark/attention_inference/results/qwen35/b300/qwen35_20260817_133148.csv deleted file mode 100644 index fcd3848d6..000000000 --- a/benchmark/attention_inference/results/qwen35/b300/qwen35_20260817_133148.csv +++ /dev/null @@ -1,889 +0,0 @@ -config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version -qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,2,256,256,gqa,,16,0.067232,1022.125,1060.554,13.2569,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,2,256,256,gqa,,16,0.619712,1774.23,460.234,5.7529,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,2,256,256,gqa,,16,11.028192,1595.201,103.449,1.2931,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,512,65536,32,2,256,256,gqa,,16,0.619968,1773.497,243.553,3.0444,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,512,131072,32,2,256,256,gqa,,16,1.1952,1839.879,238.632,2.9829,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,2,256,256,gqa,,16,1.225312,1794.664,136.922,1.7115,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,2,256,256,gqa,,16,2.411424,1823.838,125.233,1.5654,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,2,256,256,gqa,,16,0.093216,46.075,2880.066,36.0008,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,2,256,256,gqa,,16,8.676832,63.359,3960.424,49.5053,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,2,256,256,gqa,,16,6.525344,1.316,41.147,0.5143,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,2,256,256,gqa,,16,193.834686,5.672,177.306,2.2163,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,2,256,256,gqa,,16,6.5248,1.975,41.156,0.5144,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,2,256,256,gqa,,16,193.764069,8.512,177.393,2.2174,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,2,256,256,gqa,,16,6.523776,2.633,41.167,0.5146,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,2,256,256,gqa,,16,193.464676,11.367,177.689,2.2211,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,2,256,256,gqa,,16,0.208352,329.824,342.225,4.2778,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,2,256,256,gqa,,16,1.052992,1044.179,270.859,3.3857,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,2,256,256,gqa,,16,12.249888,1436.11,93.132,1.1641,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,2,256,256,gqa,,16,0.798592,1376.813,189.076,2.3635,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,2,256,256,gqa,,16,1.529376,1437.856,186.49,2.3311,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,2,256,256,gqa,,16,1.451744,1514.746,115.566,1.4446,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,2,256,256,gqa,,16,2.751136,1598.629,109.769,1.3721,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,2,256,256,gqa,,16,1.487776,2.887,180.449,2.2556,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,2,256,256,gqa,,16,114.485535,4.802,300.16,3.752,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,2,256,256,gqa,,16,1.465408,5.862,183.226,2.2903,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,2,256,256,gqa,,16,114.229568,9.625,300.869,3.7609,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,2,256,256,gqa,,16,1.501216,8.583,178.877,2.236,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,2,256,256,gqa,,16,113.881119,14.482,301.826,3.7728,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,2,256,256,gqa,,16,1.499584,11.456,179.094,2.2387,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,2,256,256,gqa,,16,115.756157,18.997,296.974,3.7122,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,32,2,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,256,256,gqa,,16,0.056032,613.216,636.272,7.9534,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,256,256,gqa,,16,0.322976,1702.157,441.539,5.5192,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,1,256,256,gqa,,16,5.562624,1581.285,102.546,1.2818,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,256,256,gqa,,16,0.58896,933.435,128.188,1.6023,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,256,256,gqa,,16,1.139328,965.053,125.167,1.5646,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,256,256,gqa,,16,0.616768,1782.699,136.009,1.7001,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,256,256,gqa,,16,1.205472,1824.201,125.258,1.5657,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,256,256,gqa,,16,0.09648,22.258,1391.315,17.3914,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,256,256,gqa,,16,4.255296,64.597,4037.784,50.4723,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,256,256,gqa,,16,6.52016,0.659,20.59,0.2574,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,256,256,gqa,,16,97.195038,5.656,176.8,2.21,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,256,256,gqa,,16,6.520544,0.988,20.591,0.2574,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,256,256,gqa,,16,97.2248,8.482,176.767,2.2096,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,256,256,gqa,,16,6.520896,1.317,20.593,0.2574,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,256,256,gqa,,16,97.132515,11.32,176.957,2.212,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,1,256,256,gqa,,16,0.151904,226.194,234.698,2.9337,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,1,256,256,gqa,,16,0.57408,957.629,248.408,3.1051,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,1,256,256,gqa,,16,6.018784,1461.44,94.774,1.1847,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,1,256,256,gqa,,16,0.659712,833.327,114.44,1.4305,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,1,256,256,gqa,,16,1.211072,907.883,117.752,1.4719,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,1,256,256,gqa,,16,0.686592,1601.405,122.177,1.5272,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,1,256,256,gqa,,16,1.275328,1724.28,118.397,1.48,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,1,256,256,gqa,,16,1.148832,1.869,116.844,1.4605,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,1,256,256,gqa,,16,38.082176,7.218,451.181,5.6398,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,1,256,256,gqa,,16,1.205856,3.562,111.332,1.3917,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,1,256,256,gqa,,16,38.465664,14.292,446.738,5.5842,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,1,256,256,gqa,,16,1.20496,5.347,111.428,1.3929,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,1,256,256,gqa,,16,38.565536,21.383,445.635,5.5704,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,1,256,256,gqa,,16,1.205408,7.126,111.401,1.3925,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,1,256,256,gqa,,16,38.805534,28.334,442.933,5.5367,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,16,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,256,256,gqa,,16,0.05312,323.416,355.316,4.4414,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,256,256,gqa,,16,0.17968,1529.819,420.177,5.2522,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,256,256,gqa,,16,2.288288,1921.981,131.972,1.6496,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,256,256,gqa,,16,0.587712,467.709,121.323,1.5165,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,256,256,gqa,,16,1.138432,482.906,121.581,1.5198,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,256,256,gqa,,16,0.588672,933.892,128.25,1.6031,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,256,256,gqa,,16,1.138656,965.622,125.241,1.5655,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,256,256,gqa,,16,0.09504,11.298,1412.31,17.6539,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,256,256,gqa,,16,4.25296,32.316,4039.755,50.4969,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,256,256,gqa,,16,6.52144,0.329,20.584,0.2573,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,256,256,gqa,,16,51.192223,5.37,335.636,4.1955,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,256,256,gqa,,16,6.522272,0.494,20.582,0.2573,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,256,256,gqa,,16,51.001122,8.084,336.914,4.2114,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,256,256,gqa,,16,6.51984,0.659,20.591,0.2574,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,256,256,gqa,,16,50.817986,10.818,338.149,4.2269,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,256,256,gqa,,16,0.123776,138.798,152.488,1.9061,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,256,256,gqa,,16,0.344096,798.841,219.408,2.7426,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,256,256,gqa,,16,2.850752,1542.767,105.933,1.3242,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,256,256,gqa,,16,0.65552,419.328,108.773,1.3597,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,256,256,gqa,,16,1.209408,454.566,114.446,1.4306,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,256,256,gqa,,16,0.660064,832.883,114.379,1.4297,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,256,256,gqa,,16,1.211232,907.763,117.737,1.4717,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,8,1,256,256,gqa,,16,1.183424,0.907,113.422,1.4178,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,8,1,256,256,gqa,,16,19.269472,7.132,891.613,11.1452,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,256,256,gqa,,16,1.20368,1.784,111.52,1.394,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,256,256,gqa,,16,19.369152,14.192,887.079,11.0885,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,256,256,gqa,,16,1.206048,2.671,111.308,1.3913,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,256,256,gqa,,16,19.287584,21.377,890.885,11.1361,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,256,256,gqa,,16,1.205248,3.564,111.388,1.3924,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,256,256,gqa,,16,19.456127,28.256,883.221,11.0403,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,8,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,4,1,256,256,gqa,,16,0.05056,169.896,207.392,2.5924,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,4,1,256,256,gqa,,16,0.109856,1251.083,381.8,4.7725,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,4,1,256,256,gqa,,16,1.155808,1902.585,145.156,1.8144,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,4,1,256,256,gqa,,16,0.587584,233.905,117.781,1.4723,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,4,1,256,256,gqa,,16,1.139232,241.284,119.655,1.4957,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,4,1,256,256,gqa,,16,0.587584,467.81,121.35,1.5169,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,4,1,256,256,gqa,,16,1.138752,482.77,121.547,1.5193,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,4,1,256,256,gqa,,16,0.08304,6.465,1616.351,20.2044,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,1,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,4,1,256,256,gqa,,16,4.253472,16.156,4039.146,50.4893,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,1,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,4,1,256,256,gqa,,16,6.543584,0.164,20.513,0.2564,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,2,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,4,1,256,256,gqa,,16,30.132511,4.561,570.179,7.1272,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,2,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,4,1,256,256,gqa,,16,6.543264,0.246,20.514,0.2564,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,3,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,4,1,256,256,gqa,,16,30.114817,6.846,570.531,7.1316,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,3,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,4,1,256,256,gqa,,16,6.543648,0.328,20.514,0.2564,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,fp8_e4m3,1,4,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,4,1,256,256,gqa,,16,30.094816,9.134,570.928,7.1366,20,True,,NVIDIA B300 SXM6 AC,cudnn 9.25.0, -qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,fp8_e4m3,128,4,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," main() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 718, in main - fn, detail = SETUPS[args.backend](args, dtype) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,4,1,256,256,gqa,,16,0.121344,70.79,86.414,1.0802,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,4,1,256,256,gqa,,16,0.254816,539.365,164.601,2.0575,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,4,1,256,256,gqa,,16,1.521152,1445.63,110.293,1.3787,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,4,1,256,256,gqa,,16,0.654432,210.013,105.75,1.3219,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,4,1,256,256,gqa,,16,1.206624,227.807,112.972,1.4122,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,4,1,256,256,gqa,,16,0.610912,449.947,116.716,1.4589,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,4,1,256,256,gqa,,16,1.208384,454.951,114.543,1.4318,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,4,1,256,256,gqa,,16,1.190912,0.451,112.705,1.4088,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,1,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,4,1,256,256,gqa,,16,10.223104,6.722,1680.546,21.0068,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,1,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,4,1,256,256,gqa,,16,1.209984,0.887,110.932,1.3866,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,2,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,4,1,256,256,gqa,,16,10.202624,13.471,1683.97,21.0496,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,2,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,4,1,256,256,gqa,,16,1.211744,1.329,110.774,1.3847,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,3,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,4,1,256,256,gqa,,16,10.240512,20.132,1677.791,20.9724,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,3,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,4,1,256,256,gqa,,16,1.212128,1.772,110.743,1.3843,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,1,4,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,4,1,256,256,gqa,,16,10.280256,26.738,1671.356,20.8919,20,True,,NVIDIA B300 SXM6 AC,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], -qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,fp8_e4m3,128,4,131072,4,1,256,256,gqa,,16,inf,0.0,0.0,,20,False," ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 215, in setup_cudnn_oss - return setup_cudnn(args, dtype, oss=True) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 119, in setup_cudnn - return setup_cudnn_fp8(args, cudnn, oss=oss) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File ""/home/vagarwalla/attn_inf/cudnn-frontend/benchmark/attention_inference/benchmark_single_attention.py"", line 261, in setup_cudnn_fp8 - graph.validate() - File ""/home/vagarwalla/attn_inf/cudnn-frontend/python/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+)",,, diff --git a/benchmark/attention_inference/results/qwen35/b300/qwen35_context.png b/benchmark/attention_inference/results/qwen35/b300/qwen35_context.png deleted file mode 100644 index d408aeaa2..000000000 Binary files a/benchmark/attention_inference/results/qwen35/b300/qwen35_context.png and /dev/null differ diff --git a/benchmark/attention_inference/results/qwen35/b300/qwen35_generation.png b/benchmark/attention_inference/results/qwen35/b300/qwen35_generation.png deleted file mode 100644 index ca9630bbc..000000000 Binary files a/benchmark/attention_inference/results/qwen35/b300/qwen35_generation.png and /dev/null differ diff --git a/benchmark/attention_inference/results/qwen35/gb200/qwen35_20260818_131428.csv b/benchmark/attention_inference/results/qwen35/gb200/qwen35_20260818_131428.csv new file mode 100644 index 000000000..ac0758a17 --- /dev/null +++ b/benchmark/attention_inference/results/qwen35/gb200/qwen35_20260818_131428.csv @@ -0,0 +1,121 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,2,256,256,gqa,,16,0.08208,837.226,868.703,10.8588,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,2,256,256,gqa,,16,0.666784,1648.977,427.744,5.3468,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,2,256,256,gqa,,16,11.146496,1578.271,102.351,1.2794,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,512,65536,32,2,256,256,gqa,,16,0.666048,1650.799,226.703,2.8338,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,512,131072,32,2,256,256,gqa,,16,1.279136,1719.147,222.973,2.7872,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,2,256,256,gqa,,16,1.256544,1750.057,133.519,1.669,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,2,256,256,gqa,,16,2.503552,1756.723,120.625,1.5078,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,2,256,256,gqa,,16,0.102208,42.022,2626.685,32.8336,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,2,256,256,gqa,,16,8.682432,63.318,3957.869,49.4734,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,2,256,256,gqa,,16,6.91296,1.243,38.84,0.4855,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,2,256,256,gqa,,16,188.968826,5.818,181.872,2.2734,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,2,256,256,gqa,,16,6.90896,1.865,38.867,0.4858,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,2,256,256,gqa,,16,188.396729,8.754,182.446,2.2806,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,2,256,256,gqa,,16,6.910912,2.486,38.861,0.4858,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,2,256,256,gqa,,16,188.862595,11.644,182.019,2.2752,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,2,256,256,gqa,,16,0.217632,315.76,327.632,4.0954,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,2,256,256,gqa,,16,1.024672,1073.038,278.345,3.4793,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,2,256,256,gqa,,16,11.87568,1481.362,96.066,1.2008,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,2,256,256,gqa,,16,0.85232,1290.022,177.158,2.2145,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,2,256,256,gqa,,16,1.596448,1377.448,178.655,2.2332,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,2,256,256,gqa,,16,1.458112,1508.131,115.061,1.4383,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,2,256,256,gqa,,16,2.851904,1542.144,105.891,1.3236,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,2,256,256,gqa,,16,1.448224,2.966,185.378,2.3172,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,2,256,256,gqa,,16,107.157982,5.13,320.685,4.0086,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,2,256,256,gqa,,16,1.466848,5.856,183.046,2.2881,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,2,256,256,gqa,,16,107.561539,10.222,319.521,3.994,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,2,256,256,gqa,,16,1.466688,8.785,183.089,2.2886,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,2,256,256,gqa,,16,106.601791,15.471,322.437,4.0305,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,2,256,256,gqa,,16,1.469248,11.693,182.792,2.2849,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,2,256,256,gqa,,16,105.920418,20.761,324.55,4.0569,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,256,256,gqa,,16,0.059232,580.087,601.897,7.5237,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,256,256,gqa,,16,0.358592,1533.096,397.684,4.9711,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,1,256,256,gqa,,16,5.492928,1601.349,103.847,1.2981,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,256,256,gqa,,16,0.581152,945.976,129.91,1.6239,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,256,256,gqa,,16,1.120896,980.922,127.225,1.5903,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,256,256,gqa,,16,0.650816,1689.435,128.894,1.6112,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,256,256,gqa,,16,1.261408,1743.309,119.703,1.4963,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,256,256,gqa,,16,0.093824,22.888,1430.701,17.8838,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,256,256,gqa,,16,4.257056,64.57,4036.114,50.4514,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,256,256,gqa,,16,6.494464,0.661,20.672,0.2584,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,256,256,gqa,,16,96.140862,5.718,178.738,2.2342,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,256,256,gqa,,16,6.494176,0.992,20.675,0.2584,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,256,256,gqa,,16,96.160576,8.576,178.724,2.234,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,256,256,gqa,,16,6.495072,1.323,20.675,0.2584,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,256,256,gqa,,16,95.935936,11.461,179.164,2.2395,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,1,256,256,gqa,,16,0.145888,235.521,244.376,3.0547,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,1,256,256,gqa,,16,0.532992,1031.452,267.558,3.3445,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,1,256,256,gqa,,16,5.602304,1570.085,101.82,1.2727,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,1,256,256,gqa,,16,0.649184,846.841,116.296,1.4537,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,1,256,256,gqa,,16,1.199008,917.018,118.937,1.4867,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,1,256,256,gqa,,16,0.721152,1524.66,116.322,1.454,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,1,256,256,gqa,,16,1.285792,1710.248,117.433,1.4679,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,1,256,256,gqa,,16,1.141696,1.881,117.574,1.4697,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,1,256,256,gqa,,16,36.238655,7.585,474.134,5.9267,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,1,256,256,gqa,,16,1.185888,3.622,113.207,1.4151,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,1,256,256,gqa,,16,36.53075,15.049,470.4,5.88,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,1,256,256,gqa,,16,1.181664,5.452,113.625,1.4203,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,1,256,256,gqa,,16,36.322464,22.703,473.155,5.9144,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,1,256,256,gqa,,16,1.18464,7.251,113.354,1.4169,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,1,256,256,gqa,,16,36.37075,30.231,472.585,5.9073,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,256,256,gqa,,16,0.057952,296.45,325.69,4.0711,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,256,256,gqa,,16,0.201024,1367.389,375.564,4.6946,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,256,256,gqa,,16,2.322784,1893.438,130.012,1.6252,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,256,256,gqa,,16,0.57904,474.713,123.14,1.5393,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,256,256,gqa,,16,1.122976,489.553,123.255,1.5407,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,256,256,gqa,,16,0.582784,943.327,129.546,1.6193,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,256,256,gqa,,16,1.123904,978.297,126.885,1.5861,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,256,256,gqa,,16,0.092224,11.643,1455.434,18.1929,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,256,256,gqa,,16,4.256672,32.288,4036.233,50.4529,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,256,256,gqa,,16,6.497728,0.33,20.659,0.2582,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,256,256,gqa,,16,52.081055,5.278,329.908,4.1239,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,256,256,gqa,,16,6.493504,0.496,20.673,0.2584,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,256,256,gqa,,16,52.085857,7.916,329.898,4.1237,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,256,256,gqa,,16,6.493344,0.661,20.675,0.2584,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,256,256,gqa,,16,52.026176,10.567,330.296,4.1287,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,256,256,gqa,,16,0.12848,133.716,146.905,1.8363,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,256,256,gqa,,16,0.297536,923.848,253.742,3.1718,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,256,256,gqa,,16,2.617344,1680.347,115.38,1.4423,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,256,256,gqa,,16,0.644064,426.787,110.708,1.3839,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,256,256,gqa,,16,1.1904,461.824,116.274,1.4534,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,256,256,gqa,,16,0.64768,848.808,116.566,1.4571,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,256,256,gqa,,16,1.1912,923.029,119.717,1.4965,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,8,1,256,256,gqa,,16,1.137888,0.944,117.961,1.4745,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,8,1,256,256,gqa,,16,18.708769,7.346,918.335,11.4792,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,256,256,gqa,,16,1.18416,1.814,113.358,1.417,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,256,256,gqa,,16,18.746592,14.663,916.538,11.4567,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,256,256,gqa,,16,1.181728,2.726,113.598,1.42,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,256,256,gqa,,16,18.820513,21.908,912.994,11.4124,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,256,256,gqa,,16,1.183264,3.63,113.458,1.4182,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,256,256,gqa,,16,18.771233,29.287,915.447,11.4431,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,4,1,256,256,gqa,,16,0.053216,161.416,197.041,2.463,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,4,1,256,256,gqa,,16,0.12336,1114.129,340.005,4.2501,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,4,1,256,256,gqa,,16,1.183936,1857.384,141.707,1.7713,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,4,1,256,256,gqa,,16,0.580704,236.676,119.176,1.4897,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,4,1,256,256,gqa,,16,1.122368,244.909,121.453,1.5182,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,4,1,256,256,gqa,,16,0.581184,472.962,122.686,1.5336,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,4,1,256,256,gqa,,16,1.12192,490.013,123.371,1.5421,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,4,1,256,256,gqa,,16,0.092192,5.823,1455.894,18.1987,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,4,1,256,256,gqa,,16,4.257248,16.142,4035.563,50.4445,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,4,1,256,256,gqa,,16,6.313408,0.17,21.26,0.2658,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,4,1,256,256,gqa,,16,30.525248,4.502,562.843,7.0355,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,4,1,256,256,gqa,,16,6.304736,0.255,21.29,0.2661,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,4,1,256,256,gqa,,16,30.532896,6.752,562.719,7.034,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,4,1,256,256,gqa,,16,6.303424,0.341,21.295,0.2662,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,4,1,256,256,gqa,,16,30.502239,9.012,563.302,7.0413,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,4,1,256,256,gqa,,16,0.128448,66.875,81.634,1.0204,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,4,1,256,256,gqa,,16,0.233504,588.594,179.625,2.2453,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,4,1,256,256,gqa,,16,1.342528,1637.972,124.967,1.5621,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,4,1,256,256,gqa,,16,0.646816,212.485,106.995,1.3374,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,4,1,256,256,gqa,,16,1.185632,231.841,114.972,1.4372,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,4,1,256,256,gqa,,16,0.647488,424.53,110.123,1.3765,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,4,1,256,256,gqa,,16,1.186624,463.294,116.644,1.458,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,4,1,256,256,gqa,,16,1.14608,0.468,117.114,1.4639,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,4,1,256,256,gqa,,16,10.124544,6.787,1696.905,21.2113,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,4,1,256,256,gqa,,16,1.188,0.904,112.985,1.4123,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,4,1,256,256,gqa,,16,9.86192,13.936,1742.147,21.7768,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,4,1,256,256,gqa,,16,1.188352,1.355,112.955,1.4119,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,4,1,256,256,gqa,,16,10.106176,20.399,1700.093,21.2512,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,4,1,256,256,gqa,,16,1.189728,1.805,112.828,1.4103,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,4,1,256,256,gqa,,16,10.055936,27.335,1708.639,21.358,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], diff --git a/benchmark/attention_inference/results/qwen35/gb200/qwen35_context.png b/benchmark/attention_inference/results/qwen35/gb200/qwen35_context.png new file mode 100644 index 000000000..3985c57c6 Binary files /dev/null and b/benchmark/attention_inference/results/qwen35/gb200/qwen35_context.png differ diff --git a/benchmark/attention_inference/results/qwen35/gb200/qwen35_generation.png b/benchmark/attention_inference/results/qwen35/gb200/qwen35_generation.png new file mode 100644 index 000000000..334c19b72 Binary files /dev/null and b/benchmark/attention_inference/results/qwen35/gb200/qwen35_generation.png differ diff --git a/benchmark/attention_inference/results/qwen35/gb300/qwen35_20260818_132128.csv b/benchmark/attention_inference/results/qwen35/gb300/qwen35_20260818_132128.csv new file mode 100644 index 000000000..d12372038 --- /dev/null +++ b/benchmark/attention_inference/results/qwen35/gb300/qwen35_20260818_132128.csv @@ -0,0 +1,121 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,2,256,256,gqa,,16,0.07584,906.111,940.179,11.7522,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,2,256,256,gqa,,16,0.569248,1931.516,501.034,6.2629,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,2,256,256,gqa,,16,9.133024,1926.217,124.915,1.5614,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,512,65536,32,2,256,256,gqa,,16,0.586976,1873.18,257.242,3.2155,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,512,131072,32,2,256,256,gqa,,16,1.133952,1939.256,251.521,3.144,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,2,256,256,gqa,,16,1.165792,1886.291,143.913,1.7989,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,2,256,256,gqa,,16,2.290784,1919.887,131.828,1.6479,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,2,256,256,gqa,,16,0.100768,42.622,2664.221,33.3028,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,2,256,256,gqa,,16,8.582336,64.057,4004.03,50.0504,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,2,256,256,gqa,,16,6.727328,1.277,39.912,0.4989,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,2,256,256,gqa,,16,186.702393,5.889,184.08,2.301,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,2,256,256,gqa,,16,6.722016,1.917,39.948,0.4994,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,2,256,256,gqa,,16,188.030624,8.771,182.802,2.285,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,2,256,256,gqa,,16,6.72208,2.556,39.953,0.4994,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,2,256,256,gqa,,16,187.845276,11.707,183.004,2.2876,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,2,256,256,gqa,,16,0.204928,335.335,347.943,4.3493,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,2,256,256,gqa,,16,0.920576,1194.374,309.82,3.8727,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,2,256,256,gqa,,16,9.864576,1783.37,115.651,1.4456,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,2,256,256,gqa,,16,0.789344,1392.944,191.292,2.3911,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,2,256,256,gqa,,16,1.4808,1485.024,192.607,2.4076,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,2,256,256,gqa,,16,1.37008,1605.033,122.454,1.5307,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,2,256,256,gqa,,16,2.651936,1658.429,113.875,1.4234,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,2,256,256,gqa,,16,1.439936,2.983,186.445,2.3306,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,2,256,256,gqa,,16,100.468605,5.472,342.037,4.2755,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,2,256,256,gqa,,16,1.461184,5.879,183.756,2.2969,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,2,256,256,gqa,,16,100.799454,10.908,340.955,4.2619,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,2,256,256,gqa,,16,1.464704,8.797,183.337,2.2917,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,2,256,256,gqa,,16,100.972801,16.334,340.412,4.2551,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,2,256,256,gqa,,16,1.459904,11.768,183.962,2.2995,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,2,256,256,gqa,,16,100.389954,21.905,342.43,4.2804,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,256,256,gqa,,16,0.054048,635.726,659.628,8.2454,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,256,256,gqa,,16,0.301152,1825.509,473.536,5.9192,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,1,256,256,gqa,,16,4.428832,1986.098,128.798,1.61,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,256,256,gqa,,16,0.57872,949.951,130.456,1.6307,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,256,256,gqa,,16,1.109984,990.565,128.476,1.606,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,256,256,gqa,,16,0.585472,1877.992,143.279,1.791,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,256,256,gqa,,16,1.132576,1941.612,133.32,1.6665,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,256,256,gqa,,16,0.09136,23.506,1469.288,18.3661,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,256,256,gqa,,16,4.21168,65.266,4079.599,50.995,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,256,256,gqa,,16,6.477216,0.663,20.727,0.2591,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,256,256,gqa,,16,95.103294,5.781,180.688,2.2586,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,256,256,gqa,,16,6.474176,0.995,20.739,0.2592,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,256,256,gqa,,16,95.062019,8.675,180.789,2.2599,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,256,256,gqa,,16,6.475392,1.327,20.737,0.2592,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,256,256,gqa,,16,94.968163,11.578,180.99,2.2624,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,1,256,256,gqa,,16,0.144992,236.977,245.887,3.0736,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,1,256,256,gqa,,16,0.477312,1151.775,298.77,3.7346,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,1,256,256,gqa,,16,4.903936,1793.68,116.32,1.454,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,1,256,256,gqa,,16,0.64352,854.295,117.32,1.4665,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,1,256,256,gqa,,16,1.18464,928.14,120.379,1.5047,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,1,256,256,gqa,,16,0.6584,1669.975,127.409,1.5926,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,1,256,256,gqa,,16,1.205888,1823.572,125.215,1.5652,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,1,256,256,gqa,,16,1.13936,1.885,117.815,1.4727,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,1,256,256,gqa,,16,31.770241,8.652,540.82,6.7602,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,1,256,256,gqa,,16,1.180544,3.638,113.719,1.4215,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,1,256,256,gqa,,16,31.808767,17.283,540.23,6.7529,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,1,256,256,gqa,,16,1.181632,5.452,113.628,1.4204,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,1,256,256,gqa,,16,31.80784,25.925,540.312,6.7539,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,1,256,256,gqa,,16,1.183424,7.259,113.47,1.4184,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,1,256,256,gqa,,16,31.829344,34.544,540.013,6.7502,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,256,256,gqa,,16,0.050912,337.442,370.725,4.6341,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,256,256,gqa,,16,0.1688,1628.424,447.26,5.5907,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,256,256,gqa,,16,2.064992,2129.813,146.243,1.828,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,256,256,gqa,,16,0.575296,477.803,123.942,1.5493,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,256,256,gqa,,16,1.115808,492.697,124.046,1.5506,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,256,256,gqa,,16,0.579264,949.059,130.333,1.6292,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,256,256,gqa,,16,1.11664,984.661,127.71,1.5964,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,256,256,gqa,,16,0.089696,11.971,1496.454,18.7057,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,256,256,gqa,,16,4.209184,32.652,4081.769,51.0221,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,256,256,gqa,,16,6.476672,0.332,20.726,0.2591,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,256,256,gqa,,16,51.699905,5.317,332.34,4.1543,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,256,256,gqa,,16,6.476928,0.497,20.726,0.2591,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,256,256,gqa,,16,51.630718,7.986,332.806,4.1601,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,256,256,gqa,,16,6.473824,0.663,20.737,0.2592,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,256,256,gqa,,16,51.627262,10.649,332.849,4.1606,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,256,256,gqa,,16,0.132928,129.242,141.989,1.7749,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,256,256,gqa,,16,0.275136,999.062,274.401,3.43,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,256,256,gqa,,16,2.409184,1825.534,125.349,1.5669,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,256,256,gqa,,16,0.643232,427.339,110.851,1.3856,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,256,256,gqa,,16,1.18224,465.012,117.076,1.4635,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,256,256,gqa,,16,0.643552,854.252,117.314,1.4664,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,256,256,gqa,,16,1.184192,928.491,120.425,1.5053,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,8,1,256,256,gqa,,16,1.136032,0.945,118.153,1.4769,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,8,1,256,256,gqa,,16,16.317184,8.423,1052.934,13.1617,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,256,256,gqa,,16,1.18336,1.815,113.435,1.4179,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,256,256,gqa,,16,16.351328,16.811,1050.799,13.135,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,256,256,gqa,,16,1.179392,2.731,113.823,1.4228,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,256,256,gqa,,16,16.35664,25.208,1050.522,13.1315,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,256,256,gqa,,16,1.1816,3.635,113.618,1.4202,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,256,256,gqa,,16,16.366432,33.59,1049.958,13.1245,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,4,1,256,256,gqa,,16,0.052928,162.295,198.114,2.4764,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,4,1,256,256,gqa,,16,0.1048,1311.44,400.22,5.0027,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,4,1,256,256,gqa,,16,1.059808,2074.926,158.304,1.9788,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,4,1,256,256,gqa,,16,0.57712,238.146,119.916,1.499,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,4,1,256,256,gqa,,16,1.116544,246.186,122.086,1.5261,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,4,1,256,256,gqa,,16,0.575968,477.245,123.797,1.5475,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,4,1,256,256,gqa,,16,1.115008,493.051,124.135,1.5517,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,4,1,256,256,gqa,,16,0.090048,5.962,1490.559,18.632,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,4,1,256,256,gqa,,16,4.209952,16.323,4080.9,51.0113,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,4,1,256,256,gqa,,16,6.258048,0.172,21.449,0.2681,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,4,1,256,256,gqa,,16,30.455488,4.513,564.132,7.0517,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,4,1,256,256,gqa,,16,6.253216,0.258,21.466,0.2683,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,4,1,256,256,gqa,,16,30.456768,6.769,564.126,7.0516,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,4,1,256,256,gqa,,16,6.253952,0.343,21.464,0.2683,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,4,1,256,256,gqa,,16,30.506592,9.01,563.221,7.0403,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,4,1,256,256,gqa,,16,0.12944,66.362,81.009,1.0126,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,4,1,256,256,gqa,,16,0.212768,645.957,197.13,2.4641,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,4,1,256,256,gqa,,16,1.228448,1790.082,136.572,1.7072,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,4,1,256,256,gqa,,16,0.638752,215.168,108.346,1.3543,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,4,1,256,256,gqa,,16,1.181696,232.613,115.355,1.4419,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,4,1,256,256,gqa,,16,0.645344,425.94,110.489,1.3811,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,4,1,256,256,gqa,,16,1.186176,463.469,116.688,1.4586,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,4,1,256,256,gqa,,16,1.1392,0.471,117.821,1.4728,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,4,1,256,256,gqa,,16,8.434944,8.147,2036.812,25.4601,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,4,1,256,256,gqa,,16,1.185056,0.906,113.265,1.4158,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,4,1,256,256,gqa,,16,8.489984,16.188,2023.669,25.2959,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,4,1,256,256,gqa,,16,1.184224,1.36,113.349,1.4169,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,4,1,256,256,gqa,,16,8.500128,24.254,2021.316,25.2664,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,4,1,256,256,gqa,,16,1.181376,1.818,113.625,1.4203,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,4,1,256,256,gqa,,16,8.460192,32.491,2030.919,25.3865,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d256[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], diff --git a/benchmark/attention_inference/results/qwen35/gb300/qwen35_context.png b/benchmark/attention_inference/results/qwen35/gb300/qwen35_context.png new file mode 100644 index 000000000..0d80f1297 Binary files /dev/null and b/benchmark/attention_inference/results/qwen35/gb300/qwen35_context.png differ diff --git a/benchmark/attention_inference/results/qwen35/gb300/qwen35_generation.png b/benchmark/attention_inference/results/qwen35/gb300/qwen35_generation.png new file mode 100644 index 000000000..1ce337742 Binary files /dev/null and b/benchmark/attention_inference/results/qwen35/gb300/qwen35_generation.png differ diff --git a/benchmark/attention_inference/results/qwen35/h200/qwen35_20260818_143154.csv b/benchmark/attention_inference/results/qwen35/h200/qwen35_20260818_143154.csv new file mode 100644 index 000000000..601dc02ff --- /dev/null +++ b/benchmark/attention_inference/results/qwen35/h200/qwen35_20260818_143154.csv @@ -0,0 +1,61 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,2,256,256,gqa,,16,0.16768,409.825,425.234,8.859,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,2,256,256,gqa,,16,1.658016,663.149,172.02,3.5838,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,2,256,256,gqa,,16,26.680321,659.369,42.76,0.8908,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,512,65536,32,2,256,256,gqa,,16,1.496384,734.779,100.907,2.1022,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,512,131072,32,2,256,256,gqa,,16,2.947328,746.107,96.77,2.016,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,2,256,256,gqa,,16,2.945344,746.61,56.962,1.1867,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,2,256,256,gqa,,16,6.539008,672.586,46.183,0.9621,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,2,256,256,gqa,,16,0.114336,37.564,2348.064,48.918,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,2,256,256,gqa,,16,9.375232,58.639,3665.396,76.3624,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,2,256,256,gqa,,16,2.86688,2.996,93.656,1.9512,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,2,256,256,gqa,,16,97.472,11.28,352.595,7.3457,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,2,256,256,gqa,,16,2.865696,4.496,93.706,1.9522,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,2,256,256,gqa,,16,97.656189,16.889,351.973,7.3328,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,2,256,256,gqa,,16,2.838112,6.053,94.629,1.9714,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,2,256,256,gqa,,16,97.448929,22.566,352.764,7.3493,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,256,256,gqa,,16,0.113408,302.975,314.366,6.5493,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,256,256,gqa,,16,0.874176,628.885,163.132,3.3986,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,1,256,256,gqa,,16,13.419552,655.468,42.507,0.8856,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,256,256,gqa,,16,1.43552,382.966,52.592,1.0957,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,256,256,gqa,,16,2.830976,388.386,50.374,1.0494,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,256,256,gqa,,16,1.490496,737.682,56.281,1.1725,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,256,256,gqa,,16,2.942336,747.373,51.318,1.0691,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,256,256,gqa,,16,0.097888,21.938,1371.303,28.5688,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,256,256,gqa,,16,4.697984,58.51,3657.306,76.1939,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,256,256,gqa,,16,2.864896,1.499,46.861,0.9763,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,256,256,gqa,,16,48.394817,11.36,355.081,7.3975,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,256,256,gqa,,16,2.863872,2.25,46.883,0.9767,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,256,256,gqa,,16,48.21904,17.102,356.419,7.4254,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,256,256,gqa,,16,2.836736,3.028,47.337,0.9862,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,256,256,gqa,,16,48.591839,22.627,353.727,7.3693,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,256,256,gqa,,16,0.075968,226.146,248.452,5.1761,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,256,256,gqa,,16,0.50688,542.294,148.945,3.103,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,256,256,gqa,,16,6.823168,644.575,44.259,0.9221,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,256,256,gqa,,16,1.429152,192.336,49.892,1.0394,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,256,256,gqa,,16,2.828352,194.373,48.937,1.0195,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,256,256,gqa,,16,1.42944,384.595,52.816,1.1003,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,256,256,gqa,,16,2.83024,388.487,50.387,1.0497,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,256,256,gqa,,16,0.09808,10.948,1368.535,28.5111,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,256,256,gqa,,16,4.688864,29.312,3664.196,76.3374,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,256,256,gqa,,16,2.860544,0.751,46.926,0.9776,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,256,256,gqa,,16,24.501535,11.219,701.261,14.6096,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,256,256,gqa,,16,2.86096,1.126,46.922,0.9775,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,256,256,gqa,,16,24.493504,16.834,701.534,14.6153,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,256,256,gqa,,16,2.861888,1.501,46.91,0.9773,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,256,256,gqa,,16,24.445984,22.489,702.94,14.6446,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,4,1,256,256,gqa,,16,0.074784,114.863,140.214,2.9211,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,4,1,256,256,gqa,,16,0.327552,419.594,128.05,2.6677,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,4,1,256,256,gqa,,16,3.265248,673.463,51.381,1.0704,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,4,1,256,256,gqa,,16,1.429024,96.177,48.429,1.0089,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,4,1,256,256,gqa,,16,2.829728,97.139,48.172,1.0036,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,4,1,256,256,gqa,,16,1.42912,192.341,49.893,1.0394,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,4,1,256,256,gqa,,16,2.825376,194.578,48.989,1.0206,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,4,1,256,256,gqa,,16,0.097024,5.533,1383.388,28.8206,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,4,1,256,256,gqa,,16,4.696992,14.631,3657.744,76.203,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,4,1,256,256,gqa,,16,2.861024,0.375,46.915,0.9774,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,4,1,256,256,gqa,,16,12.507168,10.989,1373.686,28.6185,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,4,1,256,256,gqa,,16,2.858592,0.563,46.957,0.9783,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,4,1,256,256,gqa,,16,12.636832,16.314,1359.632,28.3257,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,4,1,256,256,gqa,,16,2.858432,0.751,46.961,0.9783,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,4,1,256,256,gqa,,16,12.629184,21.765,1360.497,28.3437,20,True,,NVIDIA H200,cudnn 9.26.0, diff --git a/benchmark/attention_inference/results/qwen35/h200/qwen35_context.png b/benchmark/attention_inference/results/qwen35/h200/qwen35_context.png new file mode 100644 index 000000000..3d4b236cf Binary files /dev/null and b/benchmark/attention_inference/results/qwen35/h200/qwen35_context.png differ diff --git a/benchmark/attention_inference/results/qwen35/h200/qwen35_generation.png b/benchmark/attention_inference/results/qwen35/h200/qwen35_generation.png new file mode 100644 index 000000000..0524e499d Binary files /dev/null and b/benchmark/attention_inference/results/qwen35/h200/qwen35_generation.png differ diff --git a/benchmark/attention_inference/results/qwen35/rtxpro6000/qwen35_20260818_145215.csv b/benchmark/attention_inference/results/qwen35/rtxpro6000/qwen35_20260818_145215.csv new file mode 100644 index 000000000..964135fc9 --- /dev/null +++ b/benchmark/attention_inference/results/qwen35/rtxpro6000/qwen35_20260818_145215.csv @@ -0,0 +1,121 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,2048,2048,32,2,256,256,gqa,,16,0.57344,119.837,124.343,6.9388,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,8192,8192,32,2,256,256,gqa,,16,7.248064,151.697,39.35,2.1959,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,32768,32768,32,2,256,256,gqa,,16,111.513054,157.759,10.231,0.5709,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,512,65536,32,2,256,256,gqa,,16,8.34096,131.821,18.103,1.0102,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,512,131072,32,2,256,256,gqa,,16,16.7848,131.013,16.992,0.9482,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,1024,65536,32,2,256,256,gqa,,16,14.777728,148.807,11.353,0.6335,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn,bfloat16,bfloat16,1,1024,131072,32,2,256,256,gqa,,16,29.878912,147.196,10.107,0.564,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,1,131072,32,2,256,256,gqa,,16,0.213568,20.111,1257.062,70.1486,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,1,131072,32,2,256,256,gqa,,16,22.321119,24.629,1539.526,85.911,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,2,131072,32,2,256,256,gqa,,16,7.010112,1.225,38.302,2.1374,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,2,131072,32,2,256,256,gqa,,16,165.801437,6.631,207.285,11.5672,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,3,131072,32,2,256,256,gqa,,16,7.009472,1.838,38.31,2.1378,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,3,131072,32,2,256,256,gqa,,16,167.107681,9.869,205.69,11.4782,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,1,4,131072,32,2,256,256,gqa,,16,7.010272,2.451,38.31,2.1379,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,generation,cudnn,bfloat16,bfloat16,128,4,131072,32,2,256,256,gqa,,16,167.928192,13.095,204.71,11.4235,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,32,2,256,256,gqa,,16,0.378432,181.59,188.417,10.5144,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,32,2,256,256,gqa,,16,3.400224,323.364,83.881,4.6808,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,32,2,256,256,gqa,,16,50.009216,351.779,22.813,1.273,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,32,2,256,256,gqa,,16,5.851936,187.889,25.803,1.4399,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,32,2,256,256,gqa,,16,11.693184,188.06,24.391,1.3611,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,32,2,256,256,gqa,,16,8.662464,253.857,19.368,1.0808,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,32,2,256,256,gqa,,16,17.319551,253.935,17.436,0.973,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,32,2,256,256,gqa,,16,6.815424,0.63,39.391,2.1982,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,32,2,256,256,gqa,,16,402.956085,1.364,85.28,4.7589,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,32,2,256,256,gqa,,16,6.056064,1.418,44.336,2.4741,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,32,2,256,256,gqa,,16,241.215424,4.558,142.479,7.9508,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,32,2,256,256,gqa,,16,6.056896,2.127,44.335,2.4741,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,32,2,256,256,gqa,,16,241.674179,6.824,142.226,7.9367,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,32,2,256,256,gqa,,16,6.05712,2.836,44.339,2.4743,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,32,2,256,256,gqa,,16,241.829849,9.093,142.152,7.9326,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,2048,2048,16,1,256,256,gqa,,16,0.315968,108.744,112.833,6.2965,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,8192,8192,16,1,256,256,gqa,,16,3.782784,145.331,37.699,2.1037,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,32768,32768,16,1,256,256,gqa,,16,55.97187,157.152,10.191,0.5687,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,512,65536,16,1,256,256,gqa,,16,4.353312,126.284,17.343,0.9678,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,512,131072,16,1,256,256,gqa,,16,8.74416,125.742,16.309,0.9101,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,1024,65536,16,1,256,256,gqa,,16,8.734112,125.887,9.604,0.536,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn,bfloat16,bfloat16,1,1024,131072,16,1,256,256,gqa,,16,16.757313,131.228,9.011,0.5028,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,1,131072,16,1,256,256,gqa,,16,0.081856,26.235,1639.881,91.5112,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,1,131072,16,1,256,256,gqa,,16,12.077408,22.76,1422.653,79.3891,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,2,131072,16,1,256,256,gqa,,16,6.018304,0.714,22.307,1.2448,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,2,131072,16,1,256,256,gqa,,16,83.603806,6.576,205.542,11.47,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,3,131072,16,1,256,256,gqa,,16,6.019168,1.07,22.307,1.2448,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,3,131072,16,1,256,256,gqa,,16,83.83194,9.837,205.007,11.4401,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,1,4,131072,16,1,256,256,gqa,,16,6.022496,1.426,22.297,1.2442,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,generation,cudnn,bfloat16,bfloat16,128,4,131072,16,1,256,256,gqa,,16,84.138557,13.068,204.285,11.3998,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,16,1,256,256,gqa,,16,0.209984,163.63,169.782,9.4745,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,16,1,256,256,gqa,,16,1.685888,326.093,84.588,4.7203,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,16,1,256,256,gqa,,16,24.845505,354.032,22.959,1.2812,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,16,1,256,256,gqa,,16,2.874304,191.266,26.266,1.4658,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,16,1,256,256,gqa,,16,5.694272,193.091,25.044,1.3975,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,16,1,256,256,gqa,,16,5.643872,194.815,14.863,0.8294,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,16,1,256,256,gqa,,16,11.28512,194.86,13.38,0.7467,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,16,1,256,256,gqa,,16,6.346976,0.338,21.149,1.1802,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,16,1,256,256,gqa,,16,174.928452,1.571,98.223,5.4812,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,16,1,256,256,gqa,,16,5.69856,0.754,23.559,1.3147,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,16,1,256,256,gqa,,16,96.262497,5.711,178.513,9.9616,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,16,1,256,256,gqa,,16,5.696896,1.131,23.568,1.3152,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,16,1,256,256,gqa,,16,96.428383,8.552,178.227,9.9457,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,16,1,256,256,gqa,,16,5.698048,1.508,23.567,1.3151,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp2,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,16,1,256,256,gqa,,16,96.522499,11.391,178.075,9.9372,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,2048,2048,8,1,256,256,gqa,,16,0.195744,87.767,96.424,5.3808,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,8192,8192,8,1,256,256,gqa,,16,2.032896,135.215,37.138,2.0724,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,32768,32768,8,1,256,256,gqa,,16,28.438721,154.65,10.619,0.5926,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,512,65536,8,1,256,256,gqa,,16,3.263712,84.222,21.847,1.2192,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,512,131072,8,1,256,256,gqa,,16,6.679712,82.302,20.721,1.1563,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,1024,65536,8,1,256,256,gqa,,16,4.331072,126.933,17.432,0.9727,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn,bfloat16,bfloat16,1,1024,131072,8,1,256,256,gqa,,16,8.720512,126.083,16.353,0.9126,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,1,131072,8,1,256,256,gqa,,16,0.081952,13.102,1637.86,91.3985,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,1,131072,8,1,256,256,gqa,,16,12.115488,11.344,1418.095,79.1348,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,2,131072,8,1,256,256,gqa,,16,6.018112,0.357,22.305,1.2447,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,2,131072,8,1,256,256,gqa,,16,44.665951,6.154,384.677,21.4664,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,3,131072,8,1,256,256,gqa,,16,6.01856,0.535,22.305,1.2447,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,3,131072,8,1,256,256,gqa,,16,44.690079,9.226,384.493,21.4561,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,1,4,131072,8,1,256,256,gqa,,16,6.037792,0.711,22.235,1.2408,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,generation,cudnn,bfloat16,bfloat16,128,4,131072,8,1,256,256,gqa,,16,44.710114,12.296,384.344,21.4478,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,8,1,256,256,gqa,,16,0.141856,121.108,133.053,7.4248,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,8,1,256,256,gqa,,16,1.100352,249.809,68.612,3.8288,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,8,1,256,256,gqa,,16,12.331008,356.666,24.49,1.3666,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,8,1,256,256,gqa,,16,2.869056,95.808,24.852,1.3869,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,8,1,256,256,gqa,,16,5.69328,96.562,24.311,1.3567,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,8,1,256,256,gqa,,16,2.871168,191.475,26.295,1.4674,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,8,1,256,256,gqa,,16,5.692096,193.165,25.053,1.3981,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,8,1,256,256,gqa,,16,6.370752,0.169,21.069,1.1757,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,8,1,256,256,gqa,,16,86.380417,1.591,198.898,11.0992,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,8,1,256,256,gqa,,16,5.69632,0.377,23.565,1.315,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,8,1,256,256,gqa,,16,51.141762,5.375,335.967,18.7482,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,8,1,256,256,gqa,,16,5.69696,0.565,23.564,1.3149,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,8,1,256,256,gqa,,16,51.170815,8.058,335.797,18.7387,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,8,1,256,256,gqa,,16,5.69632,0.754,23.568,1.3152,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp4,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,8,1,256,256,gqa,,16,51.136192,10.751,336.045,18.7525,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,2048,2048,4,1,256,256,gqa,,16,0.146208,58.751,71.718,4.0021,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,8192,8192,4,1,256,256,gqa,,16,1.181344,116.341,35.505,1.9813,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,32768,32768,4,1,256,256,gqa,,16,14.663648,149.964,11.441,0.6385,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,512,65536,4,1,256,256,gqa,,16,3.196704,42.994,21.649,1.2081,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,512,131072,4,1,256,256,gqa,,16,6.639552,41.4,20.531,1.1457,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,1024,65536,4,1,256,256,gqa,,16,3.261152,84.289,21.864,1.2201,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn,bfloat16,bfloat16,1,1024,131072,4,1,256,256,gqa,,16,6.673056,82.384,20.742,1.1575,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,1,131072,4,1,256,256,gqa,,16,0.081568,6.582,1645.521,91.8259,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,1,131072,4,1,256,256,gqa,,16,12.071936,5.692,1423.168,79.4179,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,2,131072,4,1,256,256,gqa,,16,6.025632,0.178,22.276,1.2431,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,2,131072,4,1,256,256,gqa,,16,23.676767,5.805,725.645,40.4936,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,3,131072,4,1,256,256,gqa,,16,6.041568,0.267,22.218,1.2398,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,3,131072,4,1,256,256,gqa,,16,23.718975,8.692,724.375,40.4227,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,1,4,131072,4,1,256,256,gqa,,16,6.038304,0.356,22.23,1.2405,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,generation,cudnn,bfloat16,bfloat16,128,4,131072,4,1,256,256,gqa,,16,23.692768,11.602,725.199,40.4687,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,2048,2048,4,1,256,256,gqa,,16,0.13648,62.939,76.83,4.2874,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,8192,8192,4,1,256,256,gqa,,16,0.608128,226.003,68.971,3.8488,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,32768,32768,4,1,256,256,gqa,,16,5.985184,367.411,28.031,1.5642,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,65536,4,1,256,256,gqa,,16,2.866592,47.945,24.142,1.3472,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,512,131072,4,1,256,256,gqa,,16,5.692384,48.289,23.947,1.3363,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,65536,4,1,256,256,gqa,,16,2.868256,95.835,24.859,1.3872,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,context,cudnn_oss,bfloat16,bfloat16,1,1024,131072,4,1,256,256,gqa,,16,5.691008,96.601,24.321,1.3572,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,1,131072,4,1,256,256,gqa,,16,6.351296,0.085,21.133,1.1793,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,1,131072,4,1,256,256,gqa,,16,39.744961,1.729,432.266,24.122,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,2,131072,4,1,256,256,gqa,,16,5.69504,0.189,23.569,1.3152,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,2,131072,4,1,256,256,gqa,,16,16.979776,8.094,1011.846,56.4646,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,3,131072,4,1,256,256,gqa,,16,5.69616,0.283,23.565,1.315,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,3,131072,4,1,256,256,gqa,,16,16.981249,12.14,1011.789,56.4614,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,1,4,131072,4,1,256,256,gqa,,16,5.696,0.377,23.566,1.3151,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], +qwen35,qwen3.5-tp8,generation,cudnn_oss,bfloat16,bfloat16,128,4,131072,4,1,256,256,gqa,,16,16.981407,16.187,1011.811,56.4626,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=64; tile_n=64; cga=1)], diff --git a/benchmark/attention_inference/results/qwen35/rtxpro6000/qwen35_context.png b/benchmark/attention_inference/results/qwen35/rtxpro6000/qwen35_context.png new file mode 100644 index 000000000..67863f18b Binary files /dev/null and b/benchmark/attention_inference/results/qwen35/rtxpro6000/qwen35_context.png differ diff --git a/benchmark/attention_inference/results/qwen35/rtxpro6000/qwen35_generation.png b/benchmark/attention_inference/results/qwen35/rtxpro6000/qwen35_generation.png new file mode 100644 index 000000000..cba17487a Binary files /dev/null and b/benchmark/attention_inference/results/qwen35/rtxpro6000/qwen35_generation.png differ diff --git a/benchmark/attention_inference/results/qwen3vl_vit/gb200/qwen3vl_vit_20260818_160317.csv b/benchmark/attention_inference/results/qwen3vl_vit/gb200/qwen3vl_vit_20260818_160317.csv new file mode 100644 index 000000000..f2bb41a5f --- /dev/null +++ b/benchmark/attention_inference/results/qwen3vl_vit/gb200/qwen3vl_vit_20260818_160317.csv @@ -0,0 +1,13 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,8836,8836,16,16,80,80,gqa,,64,0.428704,932.446,211.056,2.6382,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,15376,15376,16,16,80,80,gqa,,64,1.157632,1045.65,136.011,1.7001,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,24336,24336,16,16,80,80,gqa,,64,2.7648,1096.742,90.133,1.1267,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,35344,35344,16,16,80,80,gqa,,64,5.913824,1081.516,61.199,0.765,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,47376,47376,16,16,80,80,gqa,,64,10.549568,1089.311,45.986,0.5748,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,62500,62500,16,16,80,80,gqa,,64,18.479616,1082.274,34.633,0.4329,20,True,,NVIDIA GB200,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,8836,8836,16,16,80,80,gqa,,64,0.629408,635.11,143.755,1.7969,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,15376,15376,16,16,80,80,gqa,,64,1.547264,782.334,101.76,1.272,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,24336,24336,16,16,80,80,gqa,,64,3.53024,858.943,70.59,0.8824,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,35344,35344,16,16,80,80,gqa,,64,6.762272,945.821,53.521,0.669,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,47376,47376,16,16,80,80,gqa,,64,11.88512,966.904,40.818,0.5102,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,62500,62500,16,16,80,80,gqa,,64,20.345888,983.0,31.456,0.3932,20,True,,NVIDIA GB200,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], diff --git a/benchmark/attention_inference/results/qwen3vl_vit/gb200/qwen3vl_vit_context.png b/benchmark/attention_inference/results/qwen3vl_vit/gb200/qwen3vl_vit_context.png new file mode 100644 index 000000000..d950090d3 Binary files /dev/null and b/benchmark/attention_inference/results/qwen3vl_vit/gb200/qwen3vl_vit_context.png differ diff --git a/benchmark/attention_inference/results/qwen3vl_vit/gb300/qwen3vl_vit_20260818_160308.csv b/benchmark/attention_inference/results/qwen3vl_vit/gb300/qwen3vl_vit_20260818_160308.csv new file mode 100644 index 000000000..b7d10b0e0 --- /dev/null +++ b/benchmark/attention_inference/results/qwen3vl_vit/gb300/qwen3vl_vit_20260818_160308.csv @@ -0,0 +1,13 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,8836,8836,16,16,80,80,gqa,,64,0.347456,1150.487,260.409,3.2551,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,15376,15376,16,16,80,80,gqa,,64,0.972128,1245.183,161.965,2.0246,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,24336,24336,16,16,80,80,gqa,,64,2.360064,1284.827,105.591,1.3199,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,35344,35344,16,16,80,80,gqa,,64,5.007904,1277.16,72.27,0.9034,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,47376,47376,16,16,80,80,gqa,,64,8.846304,1299.047,54.84,0.6855,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,62500,62500,16,16,80,80,gqa,,64,15.315744,1305.846,41.787,0.5223,20,True,,NVIDIA GB300,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,8836,8836,16,16,80,80,gqa,,64,0.535968,745.835,168.817,2.1102,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,15376,15376,16,16,80,80,gqa,,64,1.293184,936.044,121.754,1.5219,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,24336,24336,16,16,80,80,gqa,,64,2.906112,1043.412,85.751,1.0719,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,35344,35344,16,16,80,80,gqa,,64,5.764288,1109.573,62.787,0.7848,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,47376,47376,16,16,80,80,gqa,,64,10.13984,1133.328,47.844,0.598,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,62500,62500,16,16,80,80,gqa,,64,17.393728,1149.84,36.795,0.4599,20,True,,NVIDIA GB300,cudnn_oss plan=sdpa_fwd_prefill_sm100_d128[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=2)], diff --git a/benchmark/attention_inference/results/qwen3vl_vit/gb300/qwen3vl_vit_context.png b/benchmark/attention_inference/results/qwen3vl_vit/gb300/qwen3vl_vit_context.png new file mode 100644 index 000000000..57046dfe9 Binary files /dev/null and b/benchmark/attention_inference/results/qwen3vl_vit/gb300/qwen3vl_vit_context.png differ diff --git a/benchmark/attention_inference/results/qwen3vl_vit/h200/qwen3vl_vit_20260818_160230.csv b/benchmark/attention_inference/results/qwen3vl_vit/h200/qwen3vl_vit_20260818_160230.csv new file mode 100644 index 000000000..8686a3bbe --- /dev/null +++ b/benchmark/attention_inference/results/qwen3vl_vit/h200/qwen3vl_vit_20260818_160230.csv @@ -0,0 +1,7 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,8836,8836,16,16,80,80,gqa,,64,1.081792,369.52,83.64,1.7425,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,15376,15376,16,16,80,80,gqa,,64,3.185856,379.954,49.422,1.0296,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,24336,24336,16,16,80,80,gqa,,64,8.152448,371.946,30.568,0.6368,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,35344,35344,16,16,80,80,gqa,,64,17.008673,376.037,21.279,0.4433,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,47376,47376,16,16,80,80,gqa,,64,30.198944,380.535,16.064,0.3347,20,True,,NVIDIA H200,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,62500,62500,16,16,80,80,gqa,,64,52.976704,377.524,12.081,0.2517,20,True,,NVIDIA H200,cudnn 9.26.0, diff --git a/benchmark/attention_inference/results/qwen3vl_vit/h200/qwen3vl_vit_context.png b/benchmark/attention_inference/results/qwen3vl_vit/h200/qwen3vl_vit_context.png new file mode 100644 index 000000000..4475f9ff0 Binary files /dev/null and b/benchmark/attention_inference/results/qwen3vl_vit/h200/qwen3vl_vit_context.png differ diff --git a/benchmark/attention_inference/results/qwen3vl_vit/rtxpro6000/qwen3vl_vit_20260818_160141.csv b/benchmark/attention_inference/results/qwen3vl_vit/rtxpro6000/qwen3vl_vit_20260818_160141.csv new file mode 100644 index 000000000..db2eb6766 --- /dev/null +++ b/benchmark/attention_inference/results/qwen3vl_vit/rtxpro6000/qwen3vl_vit_20260818_160141.csv @@ -0,0 +1,13 @@ +config_name,model_name,phase,backend,data_type,kv_cache_dtype,batch_size,q_tokens,kv_len,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,kind,sliding_window_size,page_size,time_ms,tflops,gbps,sol_pct,num_iterations,success,error_message,gpu_name,backend_detail,cudnn_backend_version +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,8836,8836,16,16,80,80,gqa,,64,1.64624,242.822,54.962,3.0671,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,15376,15376,16,16,80,80,gqa,,64,5.01424,241.408,31.401,1.7523,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,24336,24336,16,16,80,80,gqa,,64,12.196128,248.626,20.433,1.1402,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,35344,35344,16,16,80,80,gqa,,64,25.696863,248.898,14.084,0.786,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,47376,47376,16,16,80,80,gqa,,64,46.026592,249.677,10.54,0.5882,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn,bfloat16,bfloat16,1,62500,62500,16,16,80,80,gqa,,64,79.768639,250.725,8.023,0.4477,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn 9.26.0, +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,8836,8836,16,16,80,80,gqa,,64,1.177216,339.567,76.86,4.2891,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,15376,15376,16,16,80,80,gqa,,64,3.375904,358.564,46.639,2.6026,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,24336,24336,16,16,80,80,gqa,,64,8.264736,366.893,30.152,1.6826,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,35344,35344,16,16,80,80,gqa,,64,17.154305,372.845,21.098,1.1773,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,47376,47376,16,16,80,80,gqa,,64,30.450048,377.397,15.932,0.8891,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], +qwen3vl_vit,qwen3vl_vit,context,cudnn_oss,bfloat16,bfloat16,1,62500,62500,16,16,80,80,gqa,,64,52.421505,381.523,12.209,0.6813,20,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,cudnn_oss plan=sdpa_fwd_prefill_sm120[SdpaFwdKnobs(sched_policy=0; tile_m=128; tile_n=128; cga=1)], diff --git a/benchmark/attention_inference/results/qwen3vl_vit/rtxpro6000/qwen3vl_vit_context.png b/benchmark/attention_inference/results/qwen3vl_vit/rtxpro6000/qwen3vl_vit_context.png new file mode 100644 index 000000000..cbfec9b74 Binary files /dev/null and b/benchmark/attention_inference/results/qwen3vl_vit/rtxpro6000/qwen3vl_vit_context.png differ diff --git a/benchmark/sdpa_benchmark_training/ACKNOWLEDGEMENTS.md b/benchmark/attention_training/ACKNOWLEDGEMENTS.md similarity index 100% rename from benchmark/sdpa_benchmark_training/ACKNOWLEDGEMENTS.md rename to benchmark/attention_training/ACKNOWLEDGEMENTS.md diff --git a/benchmark/sdpa_benchmark_training/Dockerfile b/benchmark/attention_training/Dockerfile similarity index 100% rename from benchmark/sdpa_benchmark_training/Dockerfile rename to benchmark/attention_training/Dockerfile diff --git a/benchmark/sdpa_benchmark_training/README.md b/benchmark/attention_training/README.md similarity index 87% rename from benchmark/sdpa_benchmark_training/README.md rename to benchmark/attention_training/README.md index 4aeff4650..6d3d5ce2e 100755 --- a/benchmark/sdpa_benchmark_training/README.md +++ b/benchmark/attention_training/README.md @@ -16,7 +16,6 @@ This directory contains benchmarking tools for Scaled Dot Product Attention (SDP - `ltx2.py` - LTX-2 video DiT self-attention benchmarks (bidirectional, no mask) - `gpt_oss.py` - GPT-OSS sliding-window-attention GQA benchmarks (causal, SWA=128) - `qwen35.py` - Qwen 3.5 GQA benchmarks (head_dim=256, causal, bf16 bidirectional — Blackwell fp8/fa4 limits) - - `qwen3vl_vit.py` - Qwen3-VL vision-encoder (ViT) self-attention benchmarks (bidirectional, no mask, fwd-only, head_dim 72-in-80) - `auto_regressive_dit.py` - Autoregressive video DiT (short Q, long cached KV, bf16/mxfp8, no_mask) - `kimi_k3.py` - Kimi K3 MLA benchmarks (96 heads, unabsorbed 192/128, NoPE) - `deepseek_v4.py` - DeepSeek-V4 shared-K=V MQA benchmarks (head_dim=512, SWA=128) @@ -39,46 +38,43 @@ docker run -it --gpus all --rm cudnn_attention_benchmark ```bash # Run Llama 3.1 benchmark suite -python -m benchmark.sdpa_benchmark_training.runner --config llama +python -m benchmark.attention_training.runner --config llama # Run DeepSeek V3 benchmark suite -python -m benchmark.sdpa_benchmark_training.runner --config dsv3 +python -m benchmark.attention_training.runner --config dsv3 # Run Kimi-K2.6 benchmark suite -python -m benchmark.sdpa_benchmark_training.runner --config kimiK26 +python -m benchmark.attention_training.runner --config kimiK26 # Run GPT-OSS benchmark suite (sliding window attention, W=128) -python -m benchmark.sdpa_benchmark_training.runner --config gpt_oss +python -m benchmark.attention_training.runner --config gpt_oss # Run Wan 2.2 A14B benchmark suite -python -m benchmark.sdpa_benchmark_training.runner --config wan22 +python -m benchmark.attention_training.runner --config wan22 # Run LTX-2 benchmark suite -python -m benchmark.sdpa_benchmark_training.runner --config ltx2 +python -m benchmark.attention_training.runner --config ltx2 # Run Qwen 3.5 benchmark suite (cuDNN bf16 at head_dim=256) -python -m benchmark.sdpa_benchmark_training.runner --config qwen35 - -# Run Qwen3-VL vision-encoder (ViT) benchmark suite (inference fwd-only) -python -m benchmark.sdpa_benchmark_training.runner --config qwen3vl_vit +python -m benchmark.attention_training.runner --config qwen35 # Run Autoregressive video DiT benchmark suite (short Q, long cached KV) -python -m benchmark.sdpa_benchmark_training.runner --config auto_regressive_dit +python -m benchmark.attention_training.runner --config auto_regressive_dit # Run Kimi K3 benchmark suite (MLA, 96 heads) -python -m benchmark.sdpa_benchmark_training.runner --config kimi_k3 +python -m benchmark.attention_training.runner --config kimi_k3 # Run DeepSeek-V4 benchmark suite (shared-K=V MQA, head_dim=512) -python -m benchmark.sdpa_benchmark_training.runner --config deepseek_v4 +python -m benchmark.attention_training.runner --config deepseek_v4 # Dry run (show what would be executed) -python -m benchmark.sdpa_benchmark_training.runner --config llama --dry-run +python -m benchmark.attention_training.runner --config llama --dry-run # Filter by backend -python -m benchmark.sdpa_benchmark_training.runner --config llama --backend cudnn +python -m benchmark.attention_training.runner --config llama --backend cudnn # Filter by data type -python -m benchmark.sdpa_benchmark_training.runner --config llama --dtype bfloat16 +python -m benchmark.attention_training.runner --config llama --dtype bfloat16 ``` ## Configuration-Based Benchmarking @@ -115,7 +111,7 @@ python -m benchmark.sdpa_benchmark_training.runner --config llama --dtype bfloat 3. Run: ```bash - python -m benchmark.sdpa_benchmark_training.runner --config my_config + python -m benchmark.attention_training.runner --config my_config ``` ### Configuration Options @@ -237,7 +233,7 @@ The benchmark script above is useful for standalone SDPA timing. To compare thos ```bash # benchmark (CUPTI-style timing) -python benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py \ +python benchmark/attention_training/benchmark_single_sdpa.py \ --batch_size 1 --q_seqlen 8192 --kv_seqlen 8192 \ --num_q_heads 64 --num_kv_heads 8 --head_dim 128 \ --sdpa_backend cudnn --data_type bfloat16 \ @@ -283,7 +279,7 @@ pytest -vv -s test/python/test_mhas_v2.py::test_repro --perf --timing_method eve ## Programmatic Usage ```python -from benchmark.sdpa_benchmark_training import ( +from benchmark.attention_training import ( BenchmarkRunner, BenchmarkConfig, ModelPreset, @@ -337,7 +333,7 @@ results/// __det_overhead.png # bwd bf16: det vs non-det comparison ``` -Runs were captured on GB200 and GB300 with cuDNN 9.23.0 and FAv4 4.0.0b15. +Runs were captured on H200, GB200, GB300 and RTX PRO 6000 Server Edition (GB300 results shown below). ### GB300 - Llama 3.1 Causal (top_left) ![Llama 3.1 Causal on GB300](results/llama3.1/gb300/llama3.1_top_left.png) @@ -377,13 +373,9 @@ Runs were captured on GB200 and GB300 with cuDNN 9.23.0 and FAv4 4.0.0b15. ![Autoregressive DiT on GB300](results/auto_regressive_dit/gb300/auto_regressive_dit_no_mask.png) - `batch=1; num_q_heads=9; num_kv_heads=9; head_dim=128; s_q ∈ {985..8192}; s_kv=62208` - Forward-only (autoregressive inference). cuDNN 9.30.0 with prefill split-K on bf16/fp8/mxfp8; FAv4 BF16 swept over `num_splits ∈ {1, 2, 4, 8, 16, 32}` with the best annotated on each bar (`ks=`). FAv4 FP8/MXFP8 are absent — the CuTe-DSL FAv4 build rejects those input types. -- Reproduce with `python -m benchmark.sdpa_benchmark_training.bench_ar_dit_peak --out `. - -### GB200 - Autoregressive video DiT -![Autoregressive DiT on GB200](results/auto_regressive_dit/gb200/auto_regressive_dit_no_mask.png) -- Same configuration as the GB300 chart above, captured on GB200. +- Reproduce with `python -m benchmark.attention_training.bench_ar_dit_peak --out `. -GB200 results are available under the same layout at `results//gb200/`. +Results on other archs are available under the same layout at `results//`. Inference-phase benchmarks (context vs generation decode, MTP widths, cudnn vs the frontend's open-source engines) live in diff --git a/benchmark/sdpa_benchmark_training/__init__.py b/benchmark/attention_training/__init__.py similarity index 82% rename from benchmark/sdpa_benchmark_training/__init__.py rename to benchmark/attention_training/__init__.py index b4aec6109..0f2855ba3 100644 --- a/benchmark/sdpa_benchmark_training/__init__.py +++ b/benchmark/attention_training/__init__.py @@ -9,13 +9,13 @@ Usage: # Run benchmarks from command line - python -m benchmark.sdpa_benchmark_training.runner --config mlperf + python -m benchmark.attention_training.runner --config mlperf # Dry run to see what would be executed - python -m benchmark.sdpa_benchmark_training.runner --config mlperf --dry-run + python -m benchmark.attention_training.runner --config mlperf --dry-run # Import and use programmatically - from benchmark.sdpa_benchmark_training import ( + from benchmark.attention_training import ( BenchmarkRunner, BenchmarkConfig, BenchmarkResult, diff --git a/benchmark/sdpa_benchmark_training/bench_ar_dit_peak.py b/benchmark/attention_training/bench_ar_dit_peak.py similarity index 96% rename from benchmark/sdpa_benchmark_training/bench_ar_dit_peak.py rename to benchmark/attention_training/bench_ar_dit_peak.py index 0c21af518..01b33af25 100644 --- a/benchmark/sdpa_benchmark_training/bench_ar_dit_peak.py +++ b/benchmark/attention_training/bench_ar_dit_peak.py @@ -8,7 +8,7 @@ against a cuDNN build that has split-K so the comparison is split-KV on both sides). -CSV schema matches the rest of ``benchmark.sdpa_benchmark_training`` +CSV schema matches the rest of ``benchmark.attention_training`` plus an extra ``num_splits`` column so the per-seqlen winners are visible. """ @@ -27,7 +27,7 @@ import torch import cudnn -from benchmark.sdpa_benchmark_training.benchmark_single_sdpa import run_benchmark +from benchmark.attention_training.benchmark_single_sdpa import run_benchmark H = 9 D = 128 diff --git a/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py b/benchmark/attention_training/benchmark_single_sdpa.py similarity index 99% rename from benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py rename to benchmark/attention_training/benchmark_single_sdpa.py index fa814de45..7d0607890 100755 --- a/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py +++ b/benchmark/attention_training/benchmark_single_sdpa.py @@ -49,6 +49,10 @@ class UnsupportedConfigError(RuntimeError): # sparsity figure), which at 188 SMs x 2.62 GHz is 1024 BF16 FLOPs/clk/SM. # Keys match the strings accepted by the --data_type CLI flag. _FLOPS_PER_CLOCK_PER_SM = { + # Hopper (sm90): H100 SXM lists 989.5 dense BF16 TFLOPS (FP32 accumulate; + # 1979 is the sparsity figure) = 132 SMs x 1.83 GHz x 4096 FLOPs/clk/SM; + # FP8 dense is 2x. (No mxfp8 entry: Hopper has no MXFP8 datapath.) + 9: {"bfloat16": 4096, "float16": 4096, "fp8": 8192}, 10: {"bfloat16": 8192, "float16": 8192, "fp8": 16384, "mxfp8": 16384}, 12: {"bfloat16": 1024, "float16": 1024, "fp8": 2048, "mxfp8": 2048}, } diff --git a/benchmark/sdpa_benchmark_training/charts.py b/benchmark/attention_training/charts.py similarity index 100% rename from benchmark/sdpa_benchmark_training/charts.py rename to benchmark/attention_training/charts.py diff --git a/benchmark/sdpa_benchmark_training/config_types.py b/benchmark/attention_training/config_types.py similarity index 100% rename from benchmark/sdpa_benchmark_training/config_types.py rename to benchmark/attention_training/config_types.py diff --git a/benchmark/sdpa_benchmark_training/configs/__init__.py b/benchmark/attention_training/configs/__init__.py similarity index 100% rename from benchmark/sdpa_benchmark_training/configs/__init__.py rename to benchmark/attention_training/configs/__init__.py diff --git a/benchmark/sdpa_benchmark_training/configs/auto_regressive_dit.py b/benchmark/attention_training/configs/auto_regressive_dit.py similarity index 91% rename from benchmark/sdpa_benchmark_training/configs/auto_regressive_dit.py rename to benchmark/attention_training/configs/auto_regressive_dit.py index 6cc22476c..c58a0efb9 100644 --- a/benchmark/sdpa_benchmark_training/configs/auto_regressive_dit.py +++ b/benchmark/attention_training/configs/auto_regressive_dit.py @@ -26,8 +26,8 @@ operator-level mask is ``no_mask``. Usage: - python -m benchmark.sdpa_benchmark_training.runner --config auto_regressive_dit - python -m benchmark.sdpa_benchmark_training.runner --config auto_regressive_dit --dry-run + python -m benchmark.attention_training.runner --config auto_regressive_dit + python -m benchmark.attention_training.runner --config auto_regressive_dit --dry-run """ from ..config_types import ModelPreset, BenchmarkConfig diff --git a/benchmark/sdpa_benchmark_training/configs/deepseek_v4.py b/benchmark/attention_training/configs/deepseek_v4.py similarity index 91% rename from benchmark/sdpa_benchmark_training/configs/deepseek_v4.py rename to benchmark/attention_training/configs/deepseek_v4.py index 207c10ad7..b4e46fdba 100644 --- a/benchmark/sdpa_benchmark_training/configs/deepseek_v4.py +++ b/benchmark/attention_training/configs/deepseek_v4.py @@ -20,8 +20,8 @@ unsupported (backend, pass) combos are recorded in the CSV rather than hidden. Usage: - python -m benchmark.sdpa_benchmark_training.runner --config deepseek_v4 - python -m benchmark.sdpa_benchmark_training.runner --config deepseek_v4 --dry-run + python -m benchmark.attention_training.runner --config deepseek_v4 + python -m benchmark.attention_training.runner --config deepseek_v4 --dry-run """ from ..config_types import ModelPreset, BenchmarkConfig diff --git a/benchmark/sdpa_benchmark_training/configs/dsv3.py b/benchmark/attention_training/configs/dsv3.py similarity index 88% rename from benchmark/sdpa_benchmark_training/configs/dsv3.py rename to benchmark/attention_training/configs/dsv3.py index 28b878980..a94c17732 100644 --- a/benchmark/sdpa_benchmark_training/configs/dsv3.py +++ b/benchmark/attention_training/configs/dsv3.py @@ -9,8 +9,8 @@ Includes forward and backward pass benchmarking with deterministic mode options. Usage: - python -m benchmark.sdpa_benchmark_training.runner --config dsv3 - python -m benchmark.sdpa_benchmark_training.runner --config dsv3 --dry-run + python -m benchmark.attention_training.runner --config dsv3 + python -m benchmark.attention_training.runner --config dsv3 --dry-run """ from ..config_types import ModelPreset, BenchmarkConfig diff --git a/benchmark/sdpa_benchmark_training/configs/gpt_oss.py b/benchmark/attention_training/configs/gpt_oss.py similarity index 87% rename from benchmark/sdpa_benchmark_training/configs/gpt_oss.py rename to benchmark/attention_training/configs/gpt_oss.py index ff4943d49..60d52fdc0 100644 --- a/benchmark/sdpa_benchmark_training/configs/gpt_oss.py +++ b/benchmark/attention_training/configs/gpt_oss.py @@ -8,8 +8,8 @@ Uses a 128-token sliding window for local attention. Usage: - python -m benchmark.sdpa_benchmark_training.runner --config gpt_oss - python -m benchmark.sdpa_benchmark_training.runner --config gpt_oss --dry-run + python -m benchmark.attention_training.runner --config gpt_oss + python -m benchmark.attention_training.runner --config gpt_oss --dry-run """ from ..config_types import ModelPreset, BenchmarkConfig diff --git a/benchmark/sdpa_benchmark_training/configs/kimiK26.py b/benchmark/attention_training/configs/kimiK26.py similarity index 90% rename from benchmark/sdpa_benchmark_training/configs/kimiK26.py rename to benchmark/attention_training/configs/kimiK26.py index 6eca789e1..11ad74af9 100644 --- a/benchmark/sdpa_benchmark_training/configs/kimiK26.py +++ b/benchmark/attention_training/configs/kimiK26.py @@ -17,8 +17,8 @@ fwd AND bwd (requires cuDNN 9.19+), so full training benchmarking works. Usage: - python -m benchmark.sdpa_benchmark_training.runner --config kimiK26 - python -m benchmark.sdpa_benchmark_training.runner --config kimiK26 --dry-run + python -m benchmark.attention_training.runner --config kimiK26 + python -m benchmark.attention_training.runner --config kimiK26 --dry-run """ from ..config_types import ModelPreset, BenchmarkConfig diff --git a/benchmark/sdpa_benchmark_training/configs/kimi_k3.py b/benchmark/attention_training/configs/kimi_k3.py similarity index 90% rename from benchmark/sdpa_benchmark_training/configs/kimi_k3.py rename to benchmark/attention_training/configs/kimi_k3.py index 667ba367c..627ecd3a7 100644 --- a/benchmark/sdpa_benchmark_training/configs/kimi_k3.py +++ b/benchmark/attention_training/configs/kimi_k3.py @@ -15,8 +15,8 @@ benchmark/attention_inference's kimi_k3 config. Usage: - python -m benchmark.sdpa_benchmark_training.runner --config kimi_k3 - python -m benchmark.sdpa_benchmark_training.runner --config kimi_k3 --dry-run + python -m benchmark.attention_training.runner --config kimi_k3 + python -m benchmark.attention_training.runner --config kimi_k3 --dry-run """ from ..config_types import ModelPreset, BenchmarkConfig diff --git a/benchmark/sdpa_benchmark_training/configs/llama.py b/benchmark/attention_training/configs/llama.py similarity index 87% rename from benchmark/sdpa_benchmark_training/configs/llama.py rename to benchmark/attention_training/configs/llama.py index 11562a1e3..d1df996c2 100644 --- a/benchmark/sdpa_benchmark_training/configs/llama.py +++ b/benchmark/attention_training/configs/llama.py @@ -8,8 +8,8 @@ Includes forward and backward pass benchmarking with deterministic mode options. Usage: - python -m benchmark.sdpa_benchmark_training.runner --config llama - python -m benchmark.sdpa_benchmark_training.runner --config llama --dry-run + python -m benchmark.attention_training.runner --config llama + python -m benchmark.attention_training.runner --config llama --dry-run """ from ..config_types import ModelPreset, BenchmarkConfig diff --git a/benchmark/sdpa_benchmark_training/configs/ltx2.py b/benchmark/attention_training/configs/ltx2.py similarity index 92% rename from benchmark/sdpa_benchmark_training/configs/ltx2.py rename to benchmark/attention_training/configs/ltx2.py index 8399815c3..ad127d6e0 100644 --- a/benchmark/sdpa_benchmark_training/configs/ltx2.py +++ b/benchmark/attention_training/configs/ltx2.py @@ -27,8 +27,8 @@ (37632, 161 frames, 1024 x 1792) Usage: - python -m benchmark.sdpa_benchmark_training.runner --config ltx2 - python -m benchmark.sdpa_benchmark_training.runner --config ltx2 --dry-run + python -m benchmark.attention_training.runner --config ltx2 + python -m benchmark.attention_training.runner --config ltx2 --dry-run """ from ..config_types import ModelPreset, BenchmarkConfig diff --git a/benchmark/sdpa_benchmark_training/configs/qwen35.py b/benchmark/attention_training/configs/qwen35.py similarity index 89% rename from benchmark/sdpa_benchmark_training/configs/qwen35.py rename to benchmark/attention_training/configs/qwen35.py index ef45467f1..c431b4bee 100755 --- a/benchmark/sdpa_benchmark_training/configs/qwen35.py +++ b/benchmark/attention_training/configs/qwen35.py @@ -9,8 +9,8 @@ Forward-only pass with bfloat16 (backward blocked at head_dim=256 on Blackwell). Usage: - python -m benchmark.sdpa_benchmark_training.runner --config qwen35 - python -m benchmark.sdpa_benchmark_training.runner --config qwen35 --dry-run + python -m benchmark.attention_training.runner --config qwen35 + python -m benchmark.attention_training.runner --config qwen35 --dry-run """ from ..config_types import ModelPreset, BenchmarkConfig diff --git a/benchmark/sdpa_benchmark_training/configs/wan22.py b/benchmark/attention_training/configs/wan22.py similarity index 93% rename from benchmark/sdpa_benchmark_training/configs/wan22.py rename to benchmark/attention_training/configs/wan22.py index b39e04e4c..b9bd15ae9 100644 --- a/benchmark/sdpa_benchmark_training/configs/wan22.py +++ b/benchmark/attention_training/configs/wan22.py @@ -28,8 +28,8 @@ (75600, 81 frames, 720p 1280x720) Usage: - python -m benchmark.sdpa_benchmark_training.runner --config wan22 - python -m benchmark.sdpa_benchmark_training.runner --config wan22 --dry-run + python -m benchmark.attention_training.runner --config wan22 + python -m benchmark.attention_training.runner --config wan22 --dry-run """ from ..config_types import ModelPreset, BenchmarkConfig diff --git a/benchmark/attention_training/results/auto_regressive_dit/gb200/auto_regressive_dit_20260818_124333.csv b/benchmark/attention_training/results/auto_regressive_dit/gb200/auto_regressive_dit_20260818_124333.csv new file mode 100644 index 000000000..51d7e318a --- /dev/null +++ b/benchmark/attention_training/results/auto_regressive_dit/gb200/auto_regressive_dit_20260818_124333.csv @@ -0,0 +1,31 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,0.214,1321.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,985,62208,9,9,128,128,fwd,False,0.163,1728.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,985,62208,9,9,128,128,fwd,False,0.166,1706.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,0.742,381.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,985,62208,9,9,128,128,fwd,False,0.730,387.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,0.221,1279.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.213,1376.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.162,1808.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.166,1765.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.740,397.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.730,402.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,4201.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.220,1334.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.403,1458.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.300,1954.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.303,1934.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.741,792.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.732,802.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.401,1462.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2493.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.756,1553.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.568,2068.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,3529.000 +auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.588,1996.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,3454.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.802,1463.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.734,1599.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,4184.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,1.030,1140.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.408,1668.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.123,2091.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.162,2021.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.551,1514.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.455,1614.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.990,1180.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 diff --git a/benchmark/attention_training/results/auto_regressive_dit/gb200/auto_regressive_dit_no_mask.png b/benchmark/attention_training/results/auto_regressive_dit/gb200/auto_regressive_dit_no_mask.png new file mode 100644 index 000000000..aabaf0aae Binary files /dev/null and b/benchmark/attention_training/results/auto_regressive_dit/gb200/auto_regressive_dit_no_mask.png differ diff --git a/benchmark/attention_training/results/auto_regressive_dit/gb300/auto_regressive_dit_20260818_124935.csv b/benchmark/attention_training/results/auto_regressive_dit/gb300/auto_regressive_dit_20260818_124935.csv new file mode 100644 index 000000000..ca6865740 --- /dev/null +++ b/benchmark/attention_training/results/auto_regressive_dit/gb300/auto_regressive_dit_20260818_124935.csv @@ -0,0 +1,31 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,0.156,1814.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,985,62208,9,9,128,128,fwd,False,0.110,2562.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,985,62208,9,9,128,128,fwd,False,0.117,2417.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,0.602,469.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,985,62208,9,9,128,128,fwd,False,0.494,572.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,0.194,1455.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.156,1884.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.110,2672.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.117,2506.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.595,493.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.494,595.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.193,1520.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.295,1991.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.207,2829.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.220,2667.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.602,975.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.496,1185.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.363,1619.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.564,2081.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2316.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.394,2981.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.425,2764.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.631,1860.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2521.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.501,2345.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.559,2099.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.114,2108.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,0.780,3009.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,0.843,2785.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.247,1884.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,0.990,2371.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.108,2120.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 diff --git a/benchmark/attention_training/results/auto_regressive_dit/gb300/auto_regressive_dit_no_mask.png b/benchmark/attention_training/results/auto_regressive_dit/gb300/auto_regressive_dit_no_mask.png new file mode 100644 index 000000000..d0f8e02c1 Binary files /dev/null and b/benchmark/attention_training/results/auto_regressive_dit/gb300/auto_regressive_dit_no_mask.png differ diff --git a/benchmark/attention_training/results/auto_regressive_dit/h200/auto_regressive_dit_20260818_133357.csv b/benchmark/attention_training/results/auto_regressive_dit/h200/auto_regressive_dit_20260818_133357.csv new file mode 100644 index 000000000..20d9ff009 --- /dev/null +++ b/benchmark/attention_training/results/auto_regressive_dit/h200/auto_regressive_dit_20260818_133357.csv @@ -0,0 +1,16 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,0.723,390.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,985,62208,9,9,128,128,fwd,False,0.513,551.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,0.645,438.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.720,408.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.511,575.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.642,457.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,1.466,400.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,1.023,574.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,1.321,444.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,2.215,530.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,1.528,768.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,2.007,585.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,3.750,626.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,2.545,923.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,3.409,689.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 diff --git a/benchmark/attention_training/results/auto_regressive_dit/h200/auto_regressive_dit_no_mask.png b/benchmark/attention_training/results/auto_regressive_dit/h200/auto_regressive_dit_no_mask.png new file mode 100644 index 000000000..a082a27fe Binary files /dev/null and b/benchmark/attention_training/results/auto_regressive_dit/h200/auto_regressive_dit_no_mask.png differ diff --git a/benchmark/attention_training/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_20260818_134814.csv b/benchmark/attention_training/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_20260818_134814.csv new file mode 100644 index 000000000..d67e2f664 --- /dev/null +++ b/benchmark/attention_training/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_20260818_134814.csv @@ -0,0 +1,21 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,1.997,141.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,985,62208,9,9,128,128,fwd,False,1.267,223.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,933.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,0.993,284.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,1.990,142.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,1.998,147.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,1.268,232.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,933.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.998,294.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,1.987,148.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,1.844,318.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,1.277,460.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,933.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,2.142,274.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,1.998,294.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,3.683,319.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,2.541,462.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,933.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,3.538,332.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,3.991,294.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,7.332,320.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,5.031,467.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,933.000 +auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,7.041,334.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,7.942,296.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 diff --git a/benchmark/attention_training/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_no_mask.png b/benchmark/attention_training/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_no_mask.png new file mode 100644 index 000000000..e29a25f61 Binary files /dev/null and b/benchmark/attention_training/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_no_mask.png differ diff --git a/benchmark/attention_training/results/deepseek_v4/gb200/deepseek_v4_20260818_123739.csv b/benchmark/attention_training/results/deepseek_v4/gb200/deepseek_v4_20260818_123739.csv new file mode 100644 index 000000000..8d753fb24 --- /dev/null +++ b/benchmark/attention_training/results/deepseek_v4/gb200/deepseek_v4_20260818_123739.csv @@ -0,0 +1,9 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,16384,16384,64,1,512,512,fwd,False,3.489,157.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,8192,8192,64,1,512,512,fwd,False,1.692,161.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,4096,4096,64,1,512,512,fwd,False,0.873,155.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,2048,2048,64,1,512,512,fwd,False,0.439,152.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,16384,16384,128,1,512,512,fwd,False,7.010,156.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,8192,8192,128,1,512,512,fwd,False,3.392,161.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,4096,4096,128,1,512,512,fwd,False,1.688,160.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,2048,2048,128,1,512,512,fwd,False,0.860,155.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 diff --git a/benchmark/attention_training/results/deepseek_v4/gb200/deepseek_v4_top_left.png b/benchmark/attention_training/results/deepseek_v4/gb200/deepseek_v4_top_left.png new file mode 100644 index 000000000..7223c2488 Binary files /dev/null and b/benchmark/attention_training/results/deepseek_v4/gb200/deepseek_v4_top_left.png differ diff --git a/benchmark/attention_training/results/deepseek_v4/gb300/deepseek_v4_20260818_124332.csv b/benchmark/attention_training/results/deepseek_v4/gb300/deepseek_v4_20260818_124332.csv new file mode 100644 index 000000000..43028c589 --- /dev/null +++ b/benchmark/attention_training/results/deepseek_v4/gb300/deepseek_v4_20260818_124332.csv @@ -0,0 +1,9 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,16384,16384,64,1,512,512,fwd,False,3.283,167.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,8192,8192,64,1,512,512,fwd,False,1.588,172.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,4096,4096,64,1,512,512,fwd,False,0.785,172.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,2048,2048,64,1,512,512,fwd,False,0.391,170.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,16384,16384,128,1,512,512,fwd,False,6.579,166.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,8192,8192,128,1,512,512,fwd,False,3.177,172.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,4096,4096,128,1,512,512,fwd,False,1.559,174.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,2048,2048,128,1,512,512,fwd,False,0.773,172.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 diff --git a/benchmark/attention_training/results/deepseek_v4/gb300/deepseek_v4_top_left.png b/benchmark/attention_training/results/deepseek_v4/gb300/deepseek_v4_top_left.png new file mode 100644 index 000000000..78f509219 Binary files /dev/null and b/benchmark/attention_training/results/deepseek_v4/gb300/deepseek_v4_top_left.png differ diff --git a/benchmark/attention_training/results/deepseek_v4/h200/deepseek_v4_20260818_124629.csv b/benchmark/attention_training/results/deepseek_v4/h200/deepseek_v4_20260818_124629.csv new file mode 100644 index 000000000..f72cf752e --- /dev/null +++ b/benchmark/attention_training/results/deepseek_v4/h200/deepseek_v4_20260818_124629.csv @@ -0,0 +1,9 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,16384,16384,64,1,512,512,fwd,False,9.944,55.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,8192,8192,64,1,512,512,fwd,False,4.903,56.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,4096,4096,64,1,512,512,fwd,False,2.393,57.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,2048,2048,64,1,512,512,fwd,False,1.188,56.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,16384,16384,128,1,512,512,fwd,False,19.773,55.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,8192,8192,128,1,512,512,fwd,False,9.785,56.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,4096,4096,128,1,512,512,fwd,False,4.761,57.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,2048,2048,128,1,512,512,fwd,False,2.354,57.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 diff --git a/benchmark/attention_training/results/deepseek_v4/h200/deepseek_v4_top_left.png b/benchmark/attention_training/results/deepseek_v4/h200/deepseek_v4_top_left.png new file mode 100644 index 000000000..b7296482b Binary files /dev/null and b/benchmark/attention_training/results/deepseek_v4/h200/deepseek_v4_top_left.png differ diff --git a/benchmark/attention_training/results/deepseek_v4/rtxpro6000/deepseek_v4_20260818_125724.csv b/benchmark/attention_training/results/deepseek_v4/rtxpro6000/deepseek_v4_20260818_125724.csv new file mode 100644 index 000000000..050c9ee09 --- /dev/null +++ b/benchmark/attention_training/results/deepseek_v4/rtxpro6000/deepseek_v4_20260818_125724.csv @@ -0,0 +1,9 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,16384,16384,64,1,512,512,fwd,False,9.456,58.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,8192,8192,64,1,512,512,fwd,False,4.716,58.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,4096,4096,64,1,512,512,fwd,False,2.351,58.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,2048,2048,64,1,512,512,fwd,False,1.237,54.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,16384,16384,128,1,512,512,fwd,False,18.831,58.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,460.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,8192,8192,128,1,512,512,fwd,False,9.399,58.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,4096,4096,128,1,512,512,fwd,False,4.666,58.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,2048,2048,128,1,512,512,fwd,False,2.317,57.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 diff --git a/benchmark/attention_training/results/deepseek_v4/rtxpro6000/deepseek_v4_top_left.png b/benchmark/attention_training/results/deepseek_v4/rtxpro6000/deepseek_v4_top_left.png new file mode 100644 index 000000000..5858c4fa8 Binary files /dev/null and b/benchmark/attention_training/results/deepseek_v4/rtxpro6000/deepseek_v4_top_left.png differ diff --git a/benchmark/attention_training/results/dsv3/gb200/dsv3_20260818_125849.csv b/benchmark/attention_training/results/dsv3/gb200/dsv3_20260818_125849.csv new file mode 100644 index 000000000..22a45012a --- /dev/null +++ b/benchmark/attention_training/results/dsv3/gb200/dsv3_20260818_125849.csv @@ -0,0 +1,131 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,50.703,1735.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,214.227,1068.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,True,213.585,1071.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,111.295,1581.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,428.714,1067.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,True,422.837,1082.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,fwd,False,36.034,2441.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,bwd,False,124.151,1842.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,bwd,True,119.146,1920.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,75.982,2315.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,243.446,1879.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,241.692,1892.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,fwd,False,39.145,2247.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,False,144.983,1577.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,True,144.370,1584.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,72.799,2417.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,289.195,1582.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,298.924,1530.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,62.961,1397.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,125.537,1401.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,55.722,1579.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,215.314,1062.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,True,219.163,1044.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,111.654,1576.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,398.471,1148.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,True,406.441,1125.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,12.472,1763.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,50.304,1137.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,True,53.400,1071.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,24.346,1806.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,104.106,1098.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,True,97.591,1172.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,fwd,False,8.645,2544.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,bwd,False,30.363,1883.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,bwd,True,28.918,1977.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,17.267,2547.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,54.760,2088.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,60.491,1890.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,fwd,False,9.199,2391.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,False,33.826,1690.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,True,35.171,1626.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,17.458,2519.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,67.117,1704.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,66.044,1731.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,14.062,1564.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,29.832,1474.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,12.588,1747.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,52.802,1083.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,True,54.765,1044.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,24.175,1819.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,102.553,1115.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,True,106.553,1073.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,3.443,1597.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,12.594,1135.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,11.629,1229.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,6.006,1831.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,24.755,1155.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,26.375,1084.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,fwd,False,2.289,2402.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,bwd,False,7.249,1972.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,bwd,True,7.312,1955.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,4.223,2604.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,13.948,2049.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,13.475,2121.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,fwd,False,2.415,2277.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,False,9.282,1540.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,True,8.701,1643.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,4.431,2481.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,16.812,1700.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,16.167,1768.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,3.755,1464.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,6.937,1585.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,3.483,1579.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,13.798,1036.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,13.238,1080.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,6.125,1795.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,24.843,1151.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,25.613,1116.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,1.018,1351.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,3.347,1068.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,3.267,1094.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.640,1676.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,5.980,1195.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,5.967,1198.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,fwd,False,0.656,2096.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,bwd,False,2.084,1715.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,bwd,True,2.087,1713.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.089,2523.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,3.468,2061.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,3.480,2054.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,fwd,False,0.672,2046.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,False,2.555,1399.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,True,2.533,1411.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.155,2380.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,4.134,1729.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,4.188,1707.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,1.043,1317.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.764,1558.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,1.006,1366.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,3.709,964.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,3.840,931.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.607,1710.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,6.119,1168.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,6.263,1141.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,0.317,1084.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,1.047,854.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,0.997,897.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.467,1473.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,1.677,1066.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,1.613,1107.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,fwd,False,0.205,1677.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,bwd,False,0.667,1340.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,bwd,True,0.672,1331.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.313,2199.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,0.985,1813.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,0.964,1854.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,fwd,False,0.219,1566.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,False,0.843,1060.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,True,0.837,1068.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.332,2068.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,1.176,1520.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,1.172,1525.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,0.336,1022.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.491,1400.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,0.319,1077.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,1.192,750.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,1.237,722.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.464,1480.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,1.757,1017.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,1.795,995.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 diff --git a/benchmark/attention_training/results/dsv3/gb200/dsv3_no_mask.png b/benchmark/attention_training/results/dsv3/gb200/dsv3_no_mask.png new file mode 100644 index 000000000..27719b760 Binary files /dev/null and b/benchmark/attention_training/results/dsv3/gb200/dsv3_no_mask.png differ diff --git a/benchmark/attention_training/results/dsv3/gb200/dsv3_no_mask_det_overhead.png b/benchmark/attention_training/results/dsv3/gb200/dsv3_no_mask_det_overhead.png new file mode 100644 index 000000000..f866052c9 Binary files /dev/null and b/benchmark/attention_training/results/dsv3/gb200/dsv3_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/dsv3/gb200/dsv3_top_left.png b/benchmark/attention_training/results/dsv3/gb200/dsv3_top_left.png new file mode 100644 index 000000000..c1244be62 Binary files /dev/null and b/benchmark/attention_training/results/dsv3/gb200/dsv3_top_left.png differ diff --git a/benchmark/attention_training/results/dsv3/gb200/dsv3_top_left_det_overhead.png b/benchmark/attention_training/results/dsv3/gb200/dsv3_top_left_det_overhead.png new file mode 100644 index 000000000..427ded130 Binary files /dev/null and b/benchmark/attention_training/results/dsv3/gb200/dsv3_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/dsv3/gb300/dsv3_20260818_130454.csv b/benchmark/attention_training/results/dsv3/gb300/dsv3_20260818_130454.csv new file mode 100644 index 000000000..d723ae09a --- /dev/null +++ b/benchmark/attention_training/results/dsv3/gb300/dsv3_20260818_130454.csv @@ -0,0 +1,131 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,44.119,1994.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,173.829,1316.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,True,174.704,1309.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,90.481,1944.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,365.232,1252.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,True,351.969,1300.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,fwd,False,25.525,3446.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,bwd,False,92.824,2464.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,bwd,True,93.200,2454.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,53.305,3300.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,185.264,2469.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,184.040,2485.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,fwd,False,28.766,3058.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,False,114.657,1995.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,True,115.237,1985.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,58.126,3027.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,226.915,2016.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,228.260,2004.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,48.659,1808.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,101.184,1739.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,47.678,1845.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,184.638,1239.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,True,191.496,1194.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,86.303,2038.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,357.430,1280.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,True,370.257,1235.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,10.785,2039.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,44.694,1279.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,True,42.942,1332.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,20.851,2109.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,87.017,1314.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,True,84.627,1351.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,fwd,False,6.538,3364.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,bwd,False,23.489,2434.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,bwd,True,22.705,2518.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,12.675,3470.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,43.480,2630.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,45.032,2539.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,fwd,False,7.309,3009.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,False,28.880,1980.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,True,29.196,1958.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,14.053,3130.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,54.570,2095.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,54.289,2106.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,12.371,1778.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,23.689,1857.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,11.487,1915.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,47.113,1214.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,True,48.081,1189.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,20.766,2118.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,88.892,1286.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,True,91.822,1245.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,2.881,1908.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,11.045,1294.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,10.626,1345.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,5.195,2117.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,20.785,1375.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,20.523,1393.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,fwd,False,1.723,3190.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,bwd,False,6.107,2341.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,bwd,True,6.116,2337.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,3.181,3456.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,11.039,2590.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,11.059,2585.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,fwd,False,1.936,2839.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,False,7.777,1838.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,True,7.779,1838.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,3.544,3102.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,13.839,2066.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,13.983,2044.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,3.209,1713.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,5.960,1845.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,3.058,1798.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,12.310,1161.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,12.894,1109.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,5.310,2071.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,22.008,1299.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,23.045,1240.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,0.818,1681.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,3.113,1148.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,2.983,1198.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.357,2026.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,5.628,1270.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,5.599,1277.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,fwd,False,0.485,2834.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,bwd,False,1.784,2004.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,bwd,True,1.788,1999.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,0.816,3367.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,2.931,2438.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,2.932,2437.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,fwd,False,0.544,2529.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,False,2.308,1548.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,True,2.304,1551.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,0.908,3027.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,3.673,1946.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,3.671,1947.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,0.885,1553.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.523,1805.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,0.868,1584.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,3.566,1002.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,3.737,957.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.380,1991.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,5.938,1204.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,6.148,1162.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,0.256,1341.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,0.978,914.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,0.930,961.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.364,1888.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,1.608,1111.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,1.555,1149.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,fwd,False,0.151,2279.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,bwd,False,0.589,1518.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,bwd,True,0.588,1519.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.220,3121.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,0.844,2116.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,0.844,2116.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,fwd,False,0.168,2051.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,False,0.775,1153.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,True,0.777,1151.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.248,2775.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,1.069,1671.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,1.070,1670.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,0.272,1266.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.396,1734.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,0.275,1252.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,1.149,778.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,1.204,742.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.387,1778.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,1.715,1042.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,1.764,1013.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 diff --git a/benchmark/attention_training/results/dsv3/gb300/dsv3_no_mask.png b/benchmark/attention_training/results/dsv3/gb300/dsv3_no_mask.png new file mode 100644 index 000000000..5c6c0b7e4 Binary files /dev/null and b/benchmark/attention_training/results/dsv3/gb300/dsv3_no_mask.png differ diff --git a/benchmark/attention_training/results/dsv3/gb300/dsv3_no_mask_det_overhead.png b/benchmark/attention_training/results/dsv3/gb300/dsv3_no_mask_det_overhead.png new file mode 100644 index 000000000..3d2ade640 Binary files /dev/null and b/benchmark/attention_training/results/dsv3/gb300/dsv3_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/dsv3/gb300/dsv3_top_left.png b/benchmark/attention_training/results/dsv3/gb300/dsv3_top_left.png new file mode 100644 index 000000000..566251c11 Binary files /dev/null and b/benchmark/attention_training/results/dsv3/gb300/dsv3_top_left.png differ diff --git a/benchmark/attention_training/results/dsv3/gb300/dsv3_top_left_det_overhead.png b/benchmark/attention_training/results/dsv3/gb300/dsv3_top_left_det_overhead.png new file mode 100644 index 000000000..01888ea2b Binary files /dev/null and b/benchmark/attention_training/results/dsv3/gb300/dsv3_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/dsv3/h200/dsv3_20260818_124320.csv b/benchmark/attention_training/results/dsv3/h200/dsv3_20260818_124320.csv new file mode 100644 index 000000000..1f39a28e4 --- /dev/null +++ b/benchmark/attention_training/results/dsv3/h200/dsv3_20260818_124320.csv @@ -0,0 +1,67 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,147.794,595.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,456.609,501.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,286.173,615.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,1425.937,321.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,fwd,False,141.374,622.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,282.565,623.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,130.691,673.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,465.198,492.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,True,528.566,433.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,275.044,640.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,931.029,491.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,True,970.564,471.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,36.796,598.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,115.503,495.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,70.862,621.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,353.356,324.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,fwd,False,35.119,626.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,69.693,631.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,31.722,693.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,120.111,476.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,True,138.037,414.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,67.628,650.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,230.835,495.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,True,246.450,464.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,8.613,638.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,28.819,496.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,35.691,401.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,17.642,623.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,89.386,320.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,91.821,311.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,fwd,False,9.025,609.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,17.306,635.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,7.680,716.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,29.481,485.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,35.152,407.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,16.582,663.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,57.134,500.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,62.915,454.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,2.261,608.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,7.306,489.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,9.145,391.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,4.084,673.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,22.323,320.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,22.260,321.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,fwd,False,2.352,585.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,4.435,620.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,2.063,666.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,7.306,489.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,9.630,371.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,3.913,703.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,14.222,503.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,16.485,434.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,0.626,549.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,2.160,414.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,2.529,353.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,1.068,643.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,5.936,301.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,5.673,315.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,fwd,False,0.649,529.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,1.150,597.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,0.602,571.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,2.160,414.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,2.870,311.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,1.036,663.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,3.738,478.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,4.710,379.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 diff --git a/benchmark/attention_training/results/dsv3/h200/dsv3_no_mask.png b/benchmark/attention_training/results/dsv3/h200/dsv3_no_mask.png new file mode 100644 index 000000000..d54bc0541 Binary files /dev/null and b/benchmark/attention_training/results/dsv3/h200/dsv3_no_mask.png differ diff --git a/benchmark/attention_training/results/dsv3/h200/dsv3_no_mask_det_overhead.png b/benchmark/attention_training/results/dsv3/h200/dsv3_no_mask_det_overhead.png new file mode 100644 index 000000000..40e3a77cd Binary files /dev/null and b/benchmark/attention_training/results/dsv3/h200/dsv3_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/dsv3/h200/dsv3_top_left.png b/benchmark/attention_training/results/dsv3/h200/dsv3_top_left.png new file mode 100644 index 000000000..6c182e88f Binary files /dev/null and b/benchmark/attention_training/results/dsv3/h200/dsv3_top_left.png differ diff --git a/benchmark/attention_training/results/dsv3/h200/dsv3_top_left_det_overhead.png b/benchmark/attention_training/results/dsv3/h200/dsv3_top_left_det_overhead.png new file mode 100644 index 000000000..0bbbe9912 Binary files /dev/null and b/benchmark/attention_training/results/dsv3/h200/dsv3_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/dsv3/rtxpro6000/dsv3_20260818_124554.csv b/benchmark/attention_training/results/dsv3/rtxpro6000/dsv3_20260818_124554.csv new file mode 100644 index 000000000..8b9e68ce3 --- /dev/null +++ b/benchmark/attention_training/results/dsv3/rtxpro6000/dsv3_20260818_124554.csv @@ -0,0 +1,41 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,384.165,229.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,770.095,228.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,218.841,402.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,434.211,405.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,249.481,353.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,1427.766,160.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,468.624,375.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,2886.154,158.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,98.372,224.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,195.169,225.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,55.222,398.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,108.806,404.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,62.986,349.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,463.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,362.238,158.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,116.580,377.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,724.367,158.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,24.126,228.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,47.610,231.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,13.430,409.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,26.907,409.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,15.702,350.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,455.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,93.064,154.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,28.716,383.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,183.320,156.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,6.236,220.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,12.051,228.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,3.650,377.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,6.613,416.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,4.148,331.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,24.516,146.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,463.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,7.223,381.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,46.971,152.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,1.679,205.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,3.104,221.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,1.082,318.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,1.777,387.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,1.184,290.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,6.803,131.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,453.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,1.934,355.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,12.328,145.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,458.000 diff --git a/benchmark/attention_training/results/dsv3/rtxpro6000/dsv3_no_mask.png b/benchmark/attention_training/results/dsv3/rtxpro6000/dsv3_no_mask.png new file mode 100644 index 000000000..2a59540b3 Binary files /dev/null and b/benchmark/attention_training/results/dsv3/rtxpro6000/dsv3_no_mask.png differ diff --git a/benchmark/attention_training/results/dsv3/rtxpro6000/dsv3_top_left.png b/benchmark/attention_training/results/dsv3/rtxpro6000/dsv3_top_left.png new file mode 100644 index 000000000..1fac25ee9 Binary files /dev/null and b/benchmark/attention_training/results/dsv3/rtxpro6000/dsv3_top_left.png differ diff --git a/benchmark/attention_training/results/gpt_oss/gb200/gpt_oss_20260818_123255.csv b/benchmark/attention_training/results/gpt_oss/gb200/gpt_oss_20260818_123255.csv new file mode 100644 index 000000000..ceec9a59b --- /dev/null +++ b/benchmark/attention_training/results/gpt_oss/gb200/gpt_oss_20260818_123255.csv @@ -0,0 +1,66 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,1.642,167.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,3.647,188.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2223.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,4.722,145.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,fwd,False,1.498,183.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,bwd,False,91.604,7.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,bwd,True,91.467,7.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,fwd,False,1.669,164.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,False,107.211,6.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,True,107.209,6.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,4.423,62.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,1.818,151.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,4.677,147.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,28.601,24.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,0.824,166.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,1.811,189.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,2.373,144.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,fwd,False,0.751,182.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,bwd,False,23.564,15.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,bwd,True,23.565,15.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,fwd,False,0.835,164.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,False,27.681,12.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,True,27.660,12.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,2.213,62.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,0.912,150.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,2.328,147.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,7.905,43.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.416,164.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,0.900,189.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,1.194,143.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,fwd,False,0.378,180.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,bwd,False,6.286,27.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,bwd,True,6.282,27.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,fwd,False,0.424,161.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,False,7.356,23.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,True,7.357,23.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,1.109,61.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.459,149.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,1.158,147.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,2.327,73.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.211,160.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,0.459,184.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,0.608,139.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,fwd,False,0.192,177.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,bwd,False,1.771,48.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,bwd,True,1.769,48.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,fwd,False,0.212,159.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,False,2.067,41.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,True,2.065,41.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.556,61.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.232,146.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,0.579,146.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,0.761,111.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.109,153.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.233,179.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,0.312,133.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,fwd,False,0.099,168.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,bwd,False,0.547,76.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,bwd,True,0.547,76.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,fwd,False,0.111,150.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,False,0.641,65.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,True,0.642,65.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.281,59.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.119,139.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.293,142.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,0.324,128.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 diff --git a/benchmark/attention_training/results/gpt_oss/gb200/gpt_oss_top_left.png b/benchmark/attention_training/results/gpt_oss/gb200/gpt_oss_top_left.png new file mode 100644 index 000000000..baf071449 Binary files /dev/null and b/benchmark/attention_training/results/gpt_oss/gb200/gpt_oss_top_left.png differ diff --git a/benchmark/attention_training/results/gpt_oss/gb200/gpt_oss_top_left_det_overhead.png b/benchmark/attention_training/results/gpt_oss/gb200/gpt_oss_top_left_det_overhead.png new file mode 100644 index 000000000..db8597e13 Binary files /dev/null and b/benchmark/attention_training/results/gpt_oss/gb200/gpt_oss_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/gpt_oss/gb300/gpt_oss_20260818_123851.csv b/benchmark/attention_training/results/gpt_oss/gb300/gpt_oss_20260818_123851.csv new file mode 100644 index 000000000..b2635601d --- /dev/null +++ b/benchmark/attention_training/results/gpt_oss/gb300/gpt_oss_20260818_123851.csv @@ -0,0 +1,66 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,1.518,181.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,3.453,199.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,4.336,158.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,fwd,False,1.318,208.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,bwd,False,74.874,9.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,bwd,True,74.897,9.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,fwd,False,1.347,204.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,False,94.766,7.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,True,94.774,7.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,4.385,63.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,1.681,163.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,4.358,157.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,27.964,25.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,0.762,180.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,1.720,199.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,2.177,157.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,fwd,False,0.661,207.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,bwd,False,19.392,18.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,bwd,True,19.392,18.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,fwd,False,0.678,202.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,False,24.468,14.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,True,24.448,14.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,2.194,62.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,0.844,162.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,2.180,157.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,7.712,44.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.384,177.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,0.863,198.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,1.101,155.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,fwd,False,0.333,205.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,bwd,False,5.203,33.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,bwd,True,5.203,33.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,fwd,False,0.343,199.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,False,6.518,26.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,True,6.516,26.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,1.098,62.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2316.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.424,161.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,1.089,157.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,2.267,75.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.196,173.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,0.441,192.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,0.563,150.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,fwd,False,0.170,199.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,bwd,False,1.489,57.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,bwd,True,1.489,57.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,fwd,False,0.175,193.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,False,1.846,46.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,True,1.846,46.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.550,62.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.215,157.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,0.549,154.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,0.740,114.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.101,165.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.228,183.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,0.290,144.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,fwd,False,0.088,189.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,bwd,False,0.470,89.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,bwd,True,0.470,89.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,fwd,False,0.092,182.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,False,0.577,72.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,True,0.577,72.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.277,60.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.111,150.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.278,150.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,0.310,134.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 diff --git a/benchmark/attention_training/results/gpt_oss/gb300/gpt_oss_top_left.png b/benchmark/attention_training/results/gpt_oss/gb300/gpt_oss_top_left.png new file mode 100644 index 000000000..377e69d37 Binary files /dev/null and b/benchmark/attention_training/results/gpt_oss/gb300/gpt_oss_top_left.png differ diff --git a/benchmark/attention_training/results/gpt_oss/gb300/gpt_oss_top_left_det_overhead.png b/benchmark/attention_training/results/gpt_oss/gb300/gpt_oss_top_left_det_overhead.png new file mode 100644 index 000000000..4d7e45c84 Binary files /dev/null and b/benchmark/attention_training/results/gpt_oss/gb300/gpt_oss_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/gpt_oss/h200/gpt_oss_20260818_122903.csv b/benchmark/attention_training/results/gpt_oss/h200/gpt_oss_20260818_122903.csv new file mode 100644 index 000000000..7240fdd44 --- /dev/null +++ b/benchmark/attention_training/results/gpt_oss/h200/gpt_oss_20260818_122903.csv @@ -0,0 +1,34 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,2.047,134.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,7.296,94.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,fwd,False,2.975,92.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,2.717,101.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,6.029,114.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,6.259,110.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,1.025,134.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,3.645,94.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,fwd,False,1.471,93.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,1.368,100.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,3.010,114.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,3.119,110.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.518,132.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,1.826,93.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,16.733,10.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,fwd,False,0.737,93.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.685,100.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,1.504,113.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,1.557,109.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.264,128.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,0.917,92.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,4.635,18.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,fwd,False,0.372,91.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.352,96.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,0.756,112.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,0.785,108.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.134,124.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.464,90.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,1.326,31.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,fwd,False,0.190,88.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.178,93.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.386,108.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,0.403,103.000,0.000,10,128,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 diff --git a/benchmark/attention_training/results/gpt_oss/h200/gpt_oss_top_left.png b/benchmark/attention_training/results/gpt_oss/h200/gpt_oss_top_left.png new file mode 100644 index 000000000..fdf0a63a2 Binary files /dev/null and b/benchmark/attention_training/results/gpt_oss/h200/gpt_oss_top_left.png differ diff --git a/benchmark/attention_training/results/gpt_oss/h200/gpt_oss_top_left_det_overhead.png b/benchmark/attention_training/results/gpt_oss/h200/gpt_oss_top_left_det_overhead.png new file mode 100644 index 000000000..f486fb260 Binary files /dev/null and b/benchmark/attention_training/results/gpt_oss/h200/gpt_oss_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/gpt_oss/rtxpro6000/gpt_oss_20260818_122958.csv b/benchmark/attention_training/results/gpt_oss/rtxpro6000/gpt_oss_20260818_122958.csv new file mode 100644 index 000000000..a9f8f7ad8 --- /dev/null +++ b/benchmark/attention_training/results/gpt_oss/rtxpro6000/gpt_oss_20260818_122958.csv @@ -0,0 +1,44 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,2.961,93.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,14.284,48.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,fwd,False,3.007,91.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,933.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,2.918,94.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,14.604,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,14.613,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,110.809,2.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,12.967,53.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,1.483,92.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,7.102,48.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,fwd,False,1.510,91.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,933.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,1.477,93.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,7.223,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,7.237,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,27.318,5.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,463.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,6.437,53.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.757,90.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,3.516,48.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,32.001,5.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,fwd,False,0.763,89.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,933.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.758,90.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,3.589,48.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,3.596,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,7.013,10.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,3.190,53.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,459.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.395,86.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,1.755,48.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,8.509,10.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,fwd,False,0.410,83.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,904.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.400,85.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,1.800,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,1.810,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,1.859,18.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,1.583,53.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.210,79.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.881,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,2.411,17.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,fwd,False,0.212,78.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,904.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.216,77.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.901,46.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,0.901,46.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.571,29.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.797,52.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 diff --git a/benchmark/attention_training/results/gpt_oss/rtxpro6000/gpt_oss_top_left.png b/benchmark/attention_training/results/gpt_oss/rtxpro6000/gpt_oss_top_left.png new file mode 100644 index 000000000..a4ed2d860 Binary files /dev/null and b/benchmark/attention_training/results/gpt_oss/rtxpro6000/gpt_oss_top_left.png differ diff --git a/benchmark/attention_training/results/gpt_oss/rtxpro6000/gpt_oss_top_left_det_overhead.png b/benchmark/attention_training/results/gpt_oss/rtxpro6000/gpt_oss_top_left_det_overhead.png new file mode 100644 index 000000000..89f37c62e Binary files /dev/null and b/benchmark/attention_training/results/gpt_oss/rtxpro6000/gpt_oss_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/kimiK26/gb200/kimiK26_20260818_124112.csv b/benchmark/attention_training/results/kimiK26/gb200/kimiK26_20260818_124112.csv new file mode 100644 index 000000000..c0149272a --- /dev/null +++ b/benchmark/attention_training/results/kimiK26/gb200/kimiK26_20260818_124112.csv @@ -0,0 +1,131 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,25.241,1743.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,107.597,1063.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,True,102.701,1113.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,1335.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,55.030,1598.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,217.583,1051.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,True,207.979,1100.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,fwd,False,17.497,2514.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,bwd,False,61.052,1873.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,bwd,True,55.221,2071.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,35.971,2445.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,119.822,1909.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,115.253,1984.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,fwd,False,18.851,2333.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,False,73.627,1553.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,True,70.530,1621.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,37.653,2336.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,137.867,1659.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,137.487,1663.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,31.366,1402.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,64.237,1369.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,27.329,1609.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,105.506,1084.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,True,108.815,1051.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,52.567,1673.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,208.327,1098.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,True,209.888,1090.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,6.353,1731.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2194.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,25.558,1119.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,22.761,1256.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,13.438,1636.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,52.532,1088.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,47.138,1213.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,fwd,False,4.298,2558.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,bwd,False,14.144,2021.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,bwd,True,13.682,2090.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,8.241,2668.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,28.176,2029.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,26.836,2130.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,fwd,False,4.526,2429.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,False,16.719,1710.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,True,17.625,1622.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,8.797,2500.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,33.663,1698.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,32.630,1752.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,7.089,1551.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,14.446,1522.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,6.465,1701.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,26.282,1088.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,26.637,1073.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,12.272,1792.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,50.138,1140.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,49.546,1154.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,1.717,1602.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,5.963,1199.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,5.800,1232.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,3.132,1755.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,11.036,1295.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,11.884,1203.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,fwd,False,1.161,2368.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,bwd,False,3.640,1963.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,bwd,True,3.645,1961.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,2.122,2591.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,6.520,2192.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,6.549,2183.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,fwd,False,1.227,2241.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,False,4.341,1647.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,True,4.342,1646.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,2.220,2477.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,8.094,1766.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,7.924,1804.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,1.896,1450.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,3.460,1589.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,1.733,1586.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,6.462,1106.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,6.689,1069.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,3.144,1748.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,12.366,1156.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,11.789,1212.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,0.508,1354.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,1.682,1062.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,1.661,1076.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.833,1649.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,2.983,1198.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,2.980,1199.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,fwd,False,0.340,2019.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,bwd,False,1.068,1673.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,bwd,True,1.064,1680.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.562,2446.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,1.753,2038.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,1.749,2043.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,fwd,False,0.359,1912.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,False,1.304,1371.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,True,1.311,1364.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.612,2246.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,2.108,1695.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,2.090,1710.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,0.543,1266.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.905,1519.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,0.516,1331.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,1.889,946.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,1.955,914.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.848,1622.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,3.078,1161.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,3.148,1135.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.168,1024.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,0.534,837.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,0.522,857.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.240,1434.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.867,1030.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.836,1068.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,fwd,False,0.111,1543.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,bwd,False,0.353,1266.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,bwd,True,0.351,1271.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.167,2053.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.522,1711.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.521,1714.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,fwd,False,0.118,1453.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,False,0.440,1015.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,True,0.441,1014.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.175,1966.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.630,1418.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.628,1422.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.178,968.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.256,1340.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.169,1015.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,0.623,718.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,0.642,696.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.243,1417.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.909,983.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.928,963.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 diff --git a/benchmark/attention_training/results/kimiK26/gb200/kimiK26_no_mask.png b/benchmark/attention_training/results/kimiK26/gb200/kimiK26_no_mask.png new file mode 100644 index 000000000..56748ed2a Binary files /dev/null and b/benchmark/attention_training/results/kimiK26/gb200/kimiK26_no_mask.png differ diff --git a/benchmark/attention_training/results/kimiK26/gb200/kimiK26_no_mask_det_overhead.png b/benchmark/attention_training/results/kimiK26/gb200/kimiK26_no_mask_det_overhead.png new file mode 100644 index 000000000..a765f02d2 Binary files /dev/null and b/benchmark/attention_training/results/kimiK26/gb200/kimiK26_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/kimiK26/gb200/kimiK26_top_left.png b/benchmark/attention_training/results/kimiK26/gb200/kimiK26_top_left.png new file mode 100644 index 000000000..89a57cf91 Binary files /dev/null and b/benchmark/attention_training/results/kimiK26/gb200/kimiK26_top_left.png differ diff --git a/benchmark/attention_training/results/kimiK26/gb200/kimiK26_top_left_det_overhead.png b/benchmark/attention_training/results/kimiK26/gb200/kimiK26_top_left_det_overhead.png new file mode 100644 index 000000000..e0c4f4908 Binary files /dev/null and b/benchmark/attention_training/results/kimiK26/gb200/kimiK26_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/kimiK26/gb300/kimiK26_20260818_124710.csv b/benchmark/attention_training/results/kimiK26/gb300/kimiK26_20260818_124710.csv new file mode 100644 index 000000000..690f2234a --- /dev/null +++ b/benchmark/attention_training/results/kimiK26/gb300/kimiK26_20260818_124710.csv @@ -0,0 +1,131 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,21.051,2089.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,86.031,1329.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,True,87.310,1310.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,43.052,2043.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,185.917,1230.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,True,175.926,1300.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,fwd,False,12.784,3440.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,4388.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,bwd,False,45.000,2541.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,bwd,True,44.323,2580.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,24.981,3521.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,91.134,2509.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,91.860,2490.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,fwd,False,14.154,3107.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,False,58.991,1938.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,True,56.831,2012.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,28.691,3066.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,112.009,2042.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,112.350,2036.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,24.410,1802.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,49.740,1768.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,22.372,1966.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,91.498,1250.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,True,96.370,1187.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,42.210,2084.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,178.694,1280.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,True,185.574,1232.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,5.335,2061.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,21.260,1345.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,20.962,1364.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,10.126,2172.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,42.801,1336.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,39.594,1444.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,fwd,False,3.243,3391.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,bwd,False,11.348,2519.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,bwd,True,11.243,2543.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,6.217,3537.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,21.703,2634.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,21.690,2636.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,fwd,False,3.635,3025.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,False,14.244,2007.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,True,14.283,2002.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,7.002,3141.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,27.453,2083.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,27.642,2068.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,6.062,1814.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,11.976,1836.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,5.677,1937.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,22.753,1257.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,24.057,1188.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,10.266,2142.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,44.164,1295.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,44.484,1285.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,1.423,1932.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,5.486,1303.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,5.275,1355.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,2.589,2123.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,10.393,1375.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,9.993,1430.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,fwd,False,0.857,3207.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,bwd,False,3.057,2338.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,bwd,True,3.061,2335.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,1.572,3498.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,5.462,2617.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,5.454,2621.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,fwd,False,0.963,2853.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,False,3.899,1833.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,True,3.897,1834.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,1.761,3122.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,6.838,2090.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,6.837,2091.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,1.590,1729.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,2.950,1864.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,1.526,1802.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,6.194,1154.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,6.514,1097.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,2.581,2130.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,11.037,1295.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,11.505,1242.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,0.412,1669.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,1.555,1149.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,1.507,1186.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.682,2016.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,2.831,1262.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,2.821,1267.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,fwd,False,0.246,2793.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,bwd,False,0.905,1974.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,bwd,True,0.904,1978.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.415,3309.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,1.474,2424.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,1.470,2430.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,fwd,False,0.275,2502.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,False,1.174,1523.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,True,1.174,1522.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.467,2946.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,1.850,1932.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,1.850,1932.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,0.441,1558.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.759,1810.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,0.438,1570.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,1.817,983.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,1.902,940.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.698,1969.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,2.995,1193.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,3.095,1155.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.133,1289.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,0.496,901.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,0.479,934.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.189,1814.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.837,1067.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.805,1110.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,fwd,False,0.080,2158.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,bwd,False,0.305,1464.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,bwd,True,0.305,1465.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.114,3005.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.440,2030.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.441,2026.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,fwd,False,0.088,1951.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,False,0.400,1116.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,True,0.401,1113.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.129,2673.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.558,1602.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.561,1592.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.139,1236.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.203,1693.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.142,1207.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,0.596,749.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,0.625,715.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.193,1777.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.885,1009.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.914,977.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 diff --git a/benchmark/attention_training/results/kimiK26/gb300/kimiK26_no_mask.png b/benchmark/attention_training/results/kimiK26/gb300/kimiK26_no_mask.png new file mode 100644 index 000000000..8b372c23f Binary files /dev/null and b/benchmark/attention_training/results/kimiK26/gb300/kimiK26_no_mask.png differ diff --git a/benchmark/attention_training/results/kimiK26/gb300/kimiK26_no_mask_det_overhead.png b/benchmark/attention_training/results/kimiK26/gb300/kimiK26_no_mask_det_overhead.png new file mode 100644 index 000000000..eb2b521ce Binary files /dev/null and b/benchmark/attention_training/results/kimiK26/gb300/kimiK26_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/kimiK26/gb300/kimiK26_top_left.png b/benchmark/attention_training/results/kimiK26/gb300/kimiK26_top_left.png new file mode 100644 index 000000000..d89005e18 Binary files /dev/null and b/benchmark/attention_training/results/kimiK26/gb300/kimiK26_top_left.png differ diff --git a/benchmark/attention_training/results/kimiK26/gb300/kimiK26_top_left_det_overhead.png b/benchmark/attention_training/results/kimiK26/gb300/kimiK26_top_left_det_overhead.png new file mode 100644 index 000000000..e8ae57ba8 Binary files /dev/null and b/benchmark/attention_training/results/kimiK26/gb300/kimiK26_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/kimiK26/h200/kimiK26_20260818_132228.csv b/benchmark/attention_training/results/kimiK26/h200/kimiK26_20260818_132228.csv new file mode 100644 index 000000000..76edecc68 --- /dev/null +++ b/benchmark/attention_training/results/kimiK26/h200/kimiK26_20260818_132228.csv @@ -0,0 +1,69 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,72.379,608.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,225.108,508.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,141.197,623.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,712.503,321.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,fwd,False,69.918,629.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,140.808,625.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,64.192,685.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,236.068,484.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,True,265.510,431.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,136.675,644.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,461.860,495.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,True,486.338,470.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,17.952,612.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,55.794,512.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,70.227,407.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,35.357,622.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,176.653,324.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,182.094,314.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,fwd,False,17.517,628.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,34.452,638.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,16.288,675.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,58.114,492.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,67.323,425.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,33.379,659.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,116.286,492.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,122.055,468.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,4.320,636.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,14.335,499.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,17.403,411.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,8.358,658.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,43.925,325.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,44.145,324.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,fwd,False,4.462,616.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,8.793,625.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,3.817,720.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,14.416,496.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,17.440,410.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,7.628,721.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,28.231,506.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,31.445,455.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,1.150,598.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,3.703,483.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,4.574,391.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,2.057,668.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,11.279,317.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,10.959,326.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,fwd,False,1.163,591.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,2.258,609.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,1.032,666.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,3.700,483.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,4.839,369.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,1.963,700.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,6.687,534.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,8.186,437.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.328,525.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,1.116,400.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,1.311,341.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.546,630.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,3.042,294.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,2.879,310.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,fwd,False,0.325,528.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.588,584.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.305,563.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,1.110,402.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,1.446,309.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.529,650.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,1.914,467.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,2.326,384.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 diff --git a/benchmark/attention_training/results/kimiK26/h200/kimiK26_no_mask.png b/benchmark/attention_training/results/kimiK26/h200/kimiK26_no_mask.png new file mode 100644 index 000000000..c4f81fa67 Binary files /dev/null and b/benchmark/attention_training/results/kimiK26/h200/kimiK26_no_mask.png differ diff --git a/benchmark/attention_training/results/kimiK26/h200/kimiK26_no_mask_det_overhead.png b/benchmark/attention_training/results/kimiK26/h200/kimiK26_no_mask_det_overhead.png new file mode 100644 index 000000000..ad21e57f0 Binary files /dev/null and b/benchmark/attention_training/results/kimiK26/h200/kimiK26_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/kimiK26/h200/kimiK26_top_left.png b/benchmark/attention_training/results/kimiK26/h200/kimiK26_top_left.png new file mode 100644 index 000000000..fb067c094 Binary files /dev/null and b/benchmark/attention_training/results/kimiK26/h200/kimiK26_top_left.png differ diff --git a/benchmark/attention_training/results/kimiK26/h200/kimiK26_top_left_det_overhead.png b/benchmark/attention_training/results/kimiK26/h200/kimiK26_top_left_det_overhead.png new file mode 100644 index 000000000..a892fc79c Binary files /dev/null and b/benchmark/attention_training/results/kimiK26/h200/kimiK26_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/kimiK26/rtxpro6000/kimiK26_20260818_133419.csv b/benchmark/attention_training/results/kimiK26/rtxpro6000/kimiK26_20260818_133419.csv new file mode 100644 index 000000000..ec5a79e7b --- /dev/null +++ b/benchmark/attention_training/results/kimiK26/rtxpro6000/kimiK26_20260818_133419.csv @@ -0,0 +1,41 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,191.584,230.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,383.331,229.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,108.950,404.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,216.258,407.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,124.522,353.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,459.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,715.385,160.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,232.797,378.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,1443.942,158.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,48.410,227.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,96.223,229.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,27.287,403.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,54.177,406.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,30.964,355.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,181.398,158.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,58.112,378.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,363.536,157.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,12.197,225.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,23.926,230.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,6.668,412.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,13.006,423.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,7.977,345.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,46.636,153.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,14.168,388.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,455.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,91.990,155.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,3.204,215.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,6.114,225.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,1.868,368.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,3.334,412.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,2.148,320.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,12.409,144.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,453.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,3.636,378.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,23.587,152.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,463.000 +kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.909,189.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,1.577,218.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.594,289.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.910,378.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.666,258.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,3.464,129.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.992,346.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,6.220,144.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 diff --git a/benchmark/attention_training/results/kimiK26/rtxpro6000/kimiK26_no_mask.png b/benchmark/attention_training/results/kimiK26/rtxpro6000/kimiK26_no_mask.png new file mode 100644 index 000000000..304291b31 Binary files /dev/null and b/benchmark/attention_training/results/kimiK26/rtxpro6000/kimiK26_no_mask.png differ diff --git a/benchmark/attention_training/results/kimiK26/rtxpro6000/kimiK26_top_left.png b/benchmark/attention_training/results/kimiK26/rtxpro6000/kimiK26_top_left.png new file mode 100644 index 000000000..c6cae4f92 Binary files /dev/null and b/benchmark/attention_training/results/kimiK26/rtxpro6000/kimiK26_top_left.png differ diff --git a/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_20260818_123442.csv b/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_20260818_123442.csv new file mode 100644 index 000000000..404f094ac --- /dev/null +++ b/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_20260818_123442.csv @@ -0,0 +1,121 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,fwd,False,39.868,1655.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,False,156.318,1097.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,True,153.604,1117.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,fwd,False,86.626,1523.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,False,316.630,1083.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,True,307.681,1115.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,32768,32768,96,96,192,128,fwd,False,25.968,2541.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,32768,32768,96,96,192,128,bwd,False,91.225,1880.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,32768,32768,96,96,192,128,bwd,True,87.845,1953.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,32768,32768,96,96,192,128,fwd,False,52.661,2506.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,32768,32768,96,96,192,128,bwd,False,180.944,1896.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,32768,32768,96,96,192,128,bwd,True,179.095,1915.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,32768,32768,96,96,192,128,fwd,False,27.685,2383.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,32768,32768,96,96,192,128,bwd,False,106.982,1603.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,32768,32768,96,96,192,128,bwd,True,106.209,1615.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,32768,32768,96,96,192,128,fwd,False,56.524,2334.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,32768,32768,96,96,192,128,bwd,False,200.797,1708.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,32768,32768,96,96,192,128,bwd,True,204.158,1680.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,fwd,False,40.451,1631.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,False,161.465,1062.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,True,161.057,1065.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,fwd,False,81.944,1610.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,False,298.865,1148.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,True,302.017,1136.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,fwd,False,9.814,1681.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,False,36.541,1174.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,True,36.305,1181.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,fwd,False,20.147,1637.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,False,81.985,1046.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,True,78.189,1097.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,16384,16384,96,96,192,128,fwd,False,6.390,2581.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,16384,16384,96,96,192,128,bwd,False,21.013,2041.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,16384,16384,96,96,192,128,bwd,True,21.102,2032.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,16384,16384,96,96,192,128,fwd,False,13.599,2426.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,16384,16384,96,96,192,128,bwd,False,42.085,2038.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,16384,16384,96,96,192,128,bwd,True,41.828,2050.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,16384,16384,96,96,192,128,fwd,False,6.764,2439.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,16384,16384,96,96,192,128,bwd,False,26.245,1634.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,16384,16384,96,96,192,128,bwd,True,23.561,1820.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,16384,16384,96,96,192,128,fwd,False,12.914,2554.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,16384,16384,96,96,192,128,bwd,False,50.927,1684.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,16384,16384,96,96,192,128,bwd,True,47.612,1801.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,fwd,False,9.897,1667.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,False,39.448,1087.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,True,40.055,1071.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,fwd,False,19.250,1714.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,False,76.728,1118.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,True,75.106,1142.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,fwd,False,2.559,1611.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,False,9.346,1147.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,True,9.360,1145.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,fwd,False,4.640,1777.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,False,18.303,1171.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,True,17.928,1196.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,8192,8192,96,96,192,128,fwd,False,1.721,2397.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,8192,8192,96,96,192,128,bwd,False,5.423,1977.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,4333.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,8192,8192,96,96,192,128,bwd,True,5.421,1978.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,8192,8192,96,96,192,128,fwd,False,3.146,2621.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,8192,8192,96,96,192,128,bwd,False,10.166,2109.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,8192,8192,96,96,192,128,bwd,True,10.420,2058.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,8192,8192,96,96,192,128,fwd,False,1.848,2232.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,8192,8192,96,96,192,128,bwd,False,6.445,1663.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,8192,8192,96,96,192,128,bwd,True,6.437,1666.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,8192,8192,96,96,192,128,fwd,False,3.282,2513.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,8192,8192,96,96,192,128,bwd,False,11.827,1813.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,8192,8192,96,96,192,128,bwd,True,11.614,1846.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,fwd,False,2.573,1603.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,False,9.673,1108.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,True,10.023,1070.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,fwd,False,4.634,1779.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,False,17.938,1195.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,True,18.916,1133.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,fwd,False,0.763,1351.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,False,2.477,1082.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,True,2.423,1106.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,fwd,False,1.227,1681.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,False,4.441,1207.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,True,4.414,1214.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,4096,4096,96,96,192,128,fwd,False,0.495,2083.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,4096,4096,96,96,192,128,bwd,False,1.582,1695.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,4096,4096,96,96,192,128,bwd,True,1.567,1711.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,4096,4096,96,96,192,128,fwd,False,0.831,2482.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,4096,4096,96,96,192,128,bwd,False,2.584,2074.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,4096,4096,96,96,192,128,bwd,True,2.577,2080.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,4096,4096,96,96,192,128,fwd,False,0.528,1953.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,4096,4096,96,96,192,128,bwd,False,1.900,1411.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,4096,4096,96,96,192,128,bwd,True,1.899,1412.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,4096,4096,96,96,192,128,fwd,False,0.894,2306.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,4096,4096,96,96,192,128,bwd,False,3.089,1735.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,4096,4096,96,96,192,128,bwd,True,3.087,1737.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,fwd,False,0.741,1391.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,False,2.796,959.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,True,2.897,925.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,fwd,False,1.248,1652.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,False,4.581,1170.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,True,4.739,1131.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,fwd,False,0.241,1070.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,False,0.784,855.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,True,0.747,898.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,fwd,False,0.355,1454.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,False,1.298,1032.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,True,1.228,1092.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,2048,2048,96,96,192,128,fwd,False,0.159,1626.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,2048,2048,96,96,192,128,bwd,False,0.514,1304.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,2048,2048,96,96,192,128,bwd,True,0.512,1309.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,2048,2048,96,96,192,128,fwd,False,0.245,2104.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,2048,2048,96,96,192,128,bwd,False,0.742,1805.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,2048,2048,96,96,192,128,bwd,True,0.754,1777.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,2048,2048,96,96,192,128,fwd,False,0.167,1543.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,2048,2048,96,96,192,128,bwd,False,0.639,1049.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,2048,2048,96,96,192,128,bwd,True,0.639,1049.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,2048,2048,96,96,192,128,fwd,False,0.259,1986.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,2048,2048,96,96,192,128,bwd,False,0.909,1474.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,2048,2048,96,96,192,128,bwd,True,0.910,1473.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,5135.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,fwd,False,0.242,1065.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,False,0.905,741.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,True,0.936,716.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,fwd,False,0.365,1413.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,False,1.347,995.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,True,1.381,970.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600,2568.000 diff --git a/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_no_mask.png b/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_no_mask.png new file mode 100644 index 000000000..3e59f9125 Binary files /dev/null and b/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_no_mask.png differ diff --git a/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_no_mask_det_overhead.png b/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_no_mask_det_overhead.png new file mode 100644 index 000000000..66553d4a1 Binary files /dev/null and b/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_top_left.png b/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_top_left.png new file mode 100644 index 000000000..c51581856 Binary files /dev/null and b/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_top_left.png differ diff --git a/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_top_left_det_overhead.png b/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_top_left_det_overhead.png new file mode 100644 index 000000000..5ffc426af Binary files /dev/null and b/benchmark/attention_training/results/kimi_k3/gb200/kimi_k3_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_20260818_124028.csv b/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_20260818_124028.csv new file mode 100644 index 000000000..85ca7e797 --- /dev/null +++ b/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_20260818_124028.csv @@ -0,0 +1,121 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,fwd,False,32.293,2043.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,False,130.780,1312.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,True,131.587,1304.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,fwd,False,64.415,2048.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,False,275.080,1247.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,True,263.949,1300.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,32768,32768,96,96,192,128,fwd,False,19.186,3439.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,32768,32768,96,96,192,128,bwd,False,68.617,2500.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,32768,32768,96,96,192,128,bwd,True,69.002,2486.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,32768,32768,96,96,192,128,fwd,False,38.719,3408.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,32768,32768,96,96,192,128,bwd,False,138.501,2477.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,32768,32768,96,96,192,128,bwd,True,138.941,2469.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,32768,32768,96,96,192,128,fwd,False,21.458,3074.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,32768,32768,96,96,192,128,bwd,False,86.370,1986.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,32768,32768,96,96,192,128,bwd,True,86.781,1977.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,32768,32768,96,96,192,128,fwd,False,42.949,3072.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,32768,32768,96,96,192,128,bwd,False,170.203,2016.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,32768,32768,96,96,192,128,bwd,True,171.801,1997.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,fwd,False,34.314,1923.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,False,136.573,1256.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,True,144.869,1184.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,fwd,False,68.311,1931.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,False,268.970,1275.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,True,277.985,1234.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,fwd,False,8.082,2041.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,False,32.474,1321.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,True,30.872,1389.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,fwd,False,15.434,2137.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,False,66.148,1297.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,True,62.646,1369.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,16384,16384,96,96,192,128,fwd,False,4.873,3384.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,16384,16384,96,96,192,128,bwd,False,17.049,2515.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,16384,16384,96,96,192,128,bwd,True,16.828,2548.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,16384,16384,96,96,192,128,fwd,False,9.357,3525.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,16384,16384,96,96,192,128,bwd,False,32.559,2634.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,16384,16384,96,96,192,128,bwd,True,32.845,2611.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,16384,16384,96,96,192,128,fwd,False,5.450,3026.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,16384,16384,96,96,192,128,bwd,False,21.253,2018.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,16384,16384,96,96,192,128,bwd,True,21.628,1983.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,16384,16384,96,96,192,128,fwd,False,10.522,3135.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,16384,16384,96,96,192,128,bwd,False,41.151,2084.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,16384,16384,96,96,192,128,bwd,True,41.069,2088.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,fwd,False,8.526,1934.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,False,34.629,1238.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,True,36.010,1191.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,fwd,False,15.947,2068.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,False,65.555,1308.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,True,70.066,1224.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,fwd,False,2.137,1930.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,False,8.199,1308.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,True,8.012,1338.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,fwd,False,3.880,2125.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,False,15.981,1342.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,True,15.555,1378.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,8192,8192,96,96,192,128,fwd,False,1.284,3211.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,8192,8192,96,96,192,128,bwd,False,4.565,2349.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,8192,8192,96,96,192,128,bwd,True,4.561,2351.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,8192,8192,96,96,192,128,fwd,False,2.374,3474.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,8192,8192,96,96,192,128,bwd,False,8.176,2623.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,8192,8192,96,96,192,128,bwd,True,8.227,2606.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,8192,8192,96,96,192,128,fwd,False,1.440,2864.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,8192,8192,96,96,192,128,bwd,False,5.805,1847.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,8192,8192,96,96,192,128,bwd,True,5.807,1846.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,8192,8192,96,96,192,128,fwd,False,2.654,3107.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,8192,8192,96,96,192,128,bwd,False,10.349,2072.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,8192,8192,96,96,192,128,bwd,True,10.238,2094.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,fwd,False,2.282,1807.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,False,9.221,1163.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,True,9.682,1107.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,fwd,False,3.925,2101.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,False,16.640,1289.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,True,17.134,1251.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,fwd,False,0.613,1681.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,False,2.325,1153.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,True,2.233,1201.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,fwd,False,1.017,2027.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,False,4.225,1269.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,True,4.189,1280.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,4096,4096,96,96,192,128,fwd,False,0.363,2840.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,4096,4096,96,96,192,128,bwd,False,1.338,2003.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,4096,4096,96,96,192,128,bwd,True,1.339,2003.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,4096,4096,96,96,192,128,fwd,False,0.618,3336.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,4096,4096,96,96,192,128,bwd,False,2.195,2442.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,4096,4096,96,96,192,128,bwd,True,2.192,2445.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,4096,4096,96,96,192,128,fwd,False,0.407,2531.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,4096,4096,96,96,192,128,bwd,False,1.733,1547.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,4096,4096,96,96,192,128,bwd,True,1.734,1546.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,4096,4096,96,96,192,128,fwd,False,0.694,2969.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,4096,4096,96,96,192,128,bwd,False,2.762,1941.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,4096,4096,96,96,192,128,bwd,True,2.761,1942.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,fwd,False,0.648,1590.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,False,2.688,997.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,True,2.813,953.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,fwd,False,1.052,1961.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,False,4.476,1198.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,True,4.646,1154.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,fwd,False,0.195,1320.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,False,0.733,914.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,True,0.703,954.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,fwd,False,0.283,1820.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,False,1.244,1077.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,True,1.169,1146.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,2048,2048,96,96,192,128,fwd,False,0.115,2240.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,2048,2048,96,96,192,128,bwd,False,0.447,1500.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,2048,2048,96,96,192,128,bwd,True,0.447,1501.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,2048,2048,96,96,192,128,fwd,False,0.173,2976.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,2048,2048,96,96,192,128,bwd,False,0.641,2089.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,2048,2048,96,96,192,128,bwd,True,0.641,2090.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,2048,2048,96,96,192,128,fwd,False,0.127,2023.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,2048,2048,96,96,192,128,bwd,False,0.587,1141.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,2048,2048,96,96,192,128,bwd,True,0.588,1140.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,2048,2048,96,96,192,128,fwd,False,0.195,2644.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,2048,2048,96,96,192,128,bwd,False,0.816,1643.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,2048,2048,96,96,192,128,bwd,True,0.816,1643.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,5155.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,fwd,False,0.208,1242.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,False,0.872,769.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,True,0.916,732.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,fwd,False,0.295,1749.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,False,1.315,1019.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,True,1.358,987.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600,2578.000 diff --git a/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_no_mask.png b/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_no_mask.png new file mode 100644 index 000000000..cf9e6e8bb Binary files /dev/null and b/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_no_mask.png differ diff --git a/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_no_mask_det_overhead.png b/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_no_mask_det_overhead.png new file mode 100644 index 000000000..f178ac351 Binary files /dev/null and b/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_top_left.png b/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_top_left.png new file mode 100644 index 000000000..9f19c1cc2 Binary files /dev/null and b/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_top_left.png differ diff --git a/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_top_left_det_overhead.png b/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_top_left_det_overhead.png new file mode 100644 index 000000000..5135beb4a Binary files /dev/null and b/benchmark/attention_training/results/kimi_k3/gb300/kimi_k3_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_20260818_130936.csv b/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_20260818_130936.csv new file mode 100644 index 000000000..56d9dc617 --- /dev/null +++ b/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_20260818_130936.csv @@ -0,0 +1,67 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,fwd,False,111.120,594.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,False,338.906,506.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,fwd,False,210.655,626.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,False,1068.200,321.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,32768,32768,96,96,192,128,fwd,False,106.484,620.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,32768,32768,96,96,192,128,fwd,False,211.563,624.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,fwd,False,96.836,681.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,False,351.070,489.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,True,396.560,433.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,fwd,False,204.334,646.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,False,693.496,495.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,True,727.878,471.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,fwd,False,26.703,618.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,False,86.602,495.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,fwd,False,53.002,622.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,False,266.713,322.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,16384,16384,96,96,192,128,fwd,False,26.218,629.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,16384,16384,96,96,192,128,fwd,False,52.074,633.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,fwd,False,23.627,698.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,False,89.028,482.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,True,102.673,418.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,fwd,False,49.954,660.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,False,174.275,492.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,True,184.718,464.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,fwd,False,6.459,638.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,False,21.627,496.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,True,26.268,408.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,fwd,False,12.755,647.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,False,66.684,322.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,True,66.944,320.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,8192,8192,96,96,192,128,fwd,False,6.901,598.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,8192,8192,96,96,192,128,fwd,False,13.008,634.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,fwd,False,5.735,719.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,False,21.691,494.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,True,26.402,406.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,fwd,False,12.026,686.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,False,42.226,508.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,True,47.339,453.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,fwd,False,1.708,604.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,False,5.502,487.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,True,6.793,395.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,fwd,False,3.077,670.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,False,16.858,318.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,True,16.262,330.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,4096,4096,96,96,192,128,fwd,False,1.757,587.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,4096,4096,96,96,192,128,fwd,False,3.329,619.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,fwd,False,1.547,667.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,False,5.508,487.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,True,7.235,371.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,fwd,False,2.951,699.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,False,10.546,508.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,True,12.382,433.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,fwd,False,0.476,542.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,False,1.636,410.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,True,1.918,349.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,fwd,False,0.810,636.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,False,4.509,297.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,True,4.292,312.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,cudnn,fp8,top_left,2,2048,2048,96,96,192,128,fwd,False,0.488,528.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,2048,2048,96,96,192,128,fwd,False,0.869,593.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,fwd,False,0.451,571.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,False,1.639,409.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,True,2.162,310.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,fwd,False,0.787,655.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,False,2.836,473.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,True,3.496,383.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 diff --git a/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_no_mask.png b/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_no_mask.png new file mode 100644 index 000000000..576830e48 Binary files /dev/null and b/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_no_mask.png differ diff --git a/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_no_mask_det_overhead.png b/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_no_mask_det_overhead.png new file mode 100644 index 000000000..608438e5f Binary files /dev/null and b/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_top_left.png b/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_top_left.png new file mode 100644 index 000000000..80fae7d89 Binary files /dev/null and b/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_top_left.png differ diff --git a/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_top_left_det_overhead.png b/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_top_left_det_overhead.png new file mode 100644 index 000000000..bd58a70ce Binary files /dev/null and b/benchmark/attention_training/results/kimi_k3/h200/kimi_k3_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/kimi_k3/rtxpro6000/kimi_k3_20260818_132057.csv b/benchmark/attention_training/results/kimi_k3/rtxpro6000/kimi_k3_20260818_132057.csv new file mode 100644 index 000000000..91d886bfa --- /dev/null +++ b/benchmark/attention_training/results/kimi_k3/rtxpro6000/kimi_k3_20260818_132057.csv @@ -0,0 +1,31 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,fwd,False,289.060,228.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,fwd,False,578.919,228.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,fwd,False,187.301,352.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,False,1073.056,160.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,fwd,False,350.471,376.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,False,2163.711,159.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,fwd,False,73.937,223.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,fwd,False,146.412,225.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,fwd,False,47.474,347.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,False,271.405,158.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,fwd,False,87.023,379.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,False,544.475,158.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,fwd,False,18.251,226.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,fwd,False,35.977,229.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,fwd,False,11.849,348.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,False,69.774,154.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,fwd,False,21.283,387.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,False,138.089,155.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,fwd,False,4.738,218.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,fwd,False,9.137,226.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,fwd,False,3.155,327.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,False,18.466,145.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,460.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,fwd,False,5.435,379.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,False,35.307,152.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,fwd,False,1.349,191.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,fwd,False,2.355,219.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,fwd,False,0.955,270.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,False,5.131,131.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,fwd,False,1.581,326.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,False,9.290,144.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 diff --git a/benchmark/attention_training/results/kimi_k3/rtxpro6000/kimi_k3_no_mask.png b/benchmark/attention_training/results/kimi_k3/rtxpro6000/kimi_k3_no_mask.png new file mode 100644 index 000000000..97119d9cd Binary files /dev/null and b/benchmark/attention_training/results/kimi_k3/rtxpro6000/kimi_k3_no_mask.png differ diff --git a/benchmark/attention_training/results/kimi_k3/rtxpro6000/kimi_k3_top_left.png b/benchmark/attention_training/results/kimi_k3/rtxpro6000/kimi_k3_top_left.png new file mode 100644 index 000000000..daa33c486 Binary files /dev/null and b/benchmark/attention_training/results/kimi_k3/rtxpro6000/kimi_k3_top_left.png differ diff --git a/benchmark/attention_training/results/llama3.1/gb200/llama3.1_20260818_124202.csv b/benchmark/attention_training/results/llama3.1/gb200/llama3.1_20260818_124202.csv new file mode 100644 index 000000000..f33232c80 --- /dev/null +++ b/benchmark/attention_training/results/llama3.1/gb200/llama3.1_20260818_124202.csv @@ -0,0 +1,141 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,22.750,1547.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,73.459,1197.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,85.443,1030.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,45.263,1555.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,147.288,1194.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,167.219,1052.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,fwd,False,16.405,2145.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,bwd,False,52.653,1671.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,bwd,True,51.811,1698.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,33.158,2122.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,102.282,1720.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,101.815,1728.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,fwd,False,17.312,2032.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,False,63.736,1380.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,True,65.268,1348.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,34.787,2023.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,128.287,1371.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,124.932,1408.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,23.567,1493.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,49.117,1433.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,32768,32768,64,8,128,128,fwd,False,21.824,1612.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,41.181,1709.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,23.995,1466.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,70.785,1243.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,74.484,1181.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,59.911,1175.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,129.288,1361.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,134.910,1304.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,5.425,1622.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,17.415,1263.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,18.947,1161.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,11.110,1583.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,34.328,1281.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,39.649,1109.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,fwd,False,4.174,2107.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,bwd,False,13.656,1610.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,bwd,True,13.463,1634.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,7.942,2215.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,25.059,1755.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,24.348,1806.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,fwd,False,4.266,2062.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,False,15.548,1414.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,True,16.000,1374.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,8.210,2143.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,31.033,1417.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,29.977,1467.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,5.615,1567.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,11.361,1548.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,16384,16384,64,8,128,128,fwd,False,5.758,1528.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,10.354,1699.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,5.329,1651.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,16.430,1338.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,15.852,1387.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,14.214,1238.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,29.003,1516.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,33.006,1333.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,1.435,1533.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,4.234,1299.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,4.830,1138.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.587,1700.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,7.874,1396.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,8.943,1229.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,fwd,False,1.101,1998.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,bwd,False,3.800,1447.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,bwd,True,3.806,1445.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.008,2190.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,6.190,1776.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,6.190,1776.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,fwd,False,1.123,1959.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,False,4.351,1264.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,True,4.355,1263.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.073,2122.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,7.440,1478.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,7.406,1485.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,1.588,1385.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.725,1614.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,8192,8192,64,8,128,128,fwd,False,1.594,1380.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.634,1670.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,1.409,1560.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,4.137,1329.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,4.365,1260.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,3.501,1256.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,7.335,1499.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,8.014,1372.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,0.406,1354.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,1.201,1144.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,1.365,1007.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.707,1556.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,2.073,1326.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,2.364,1163.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,fwd,False,0.311,1769.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,bwd,False,1.174,1171.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,bwd,True,1.171,1174.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.537,2046.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,1.744,1576.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,1.741,1579.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,fwd,False,0.321,1716.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,False,1.334,1031.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,True,1.334,1031.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.556,1977.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,2.059,1335.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,2.060,1335.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,0.450,1223.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.740,1486.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,4096,4096,64,8,128,128,fwd,False,0.495,1111.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.710,1549.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,0.414,1329.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,1.230,1117.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,1.448,950.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.921,1194.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,2.040,1348.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,2.250,1222.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.129,1069.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,0.394,872.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,0.437,786.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.198,1391.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.618,1111.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.693,992.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.100,1377.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,bwd,False,0.417,824.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,bwd,True,0.417,824.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.149,1847.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.562,1223.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.561,1224.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.103,1334.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,False,0.469,733.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,True,0.467,736.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.156,1762.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.646,1065.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.646,1064.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.151,913.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.205,1339.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.184,748.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.197,1393.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,5135.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.132,1045.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,0.418,823.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,0.547,629.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.252,1093.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.607,1132.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.721,952.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 diff --git a/benchmark/attention_training/results/llama3.1/gb200/llama3.1_no_mask.png b/benchmark/attention_training/results/llama3.1/gb200/llama3.1_no_mask.png new file mode 100644 index 000000000..462690879 Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/gb200/llama3.1_no_mask.png differ diff --git a/benchmark/attention_training/results/llama3.1/gb200/llama3.1_no_mask_det_overhead.png b/benchmark/attention_training/results/llama3.1/gb200/llama3.1_no_mask_det_overhead.png new file mode 100644 index 000000000..5ac605afd Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/gb200/llama3.1_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/llama3.1/gb200/llama3.1_top_left.png b/benchmark/attention_training/results/llama3.1/gb200/llama3.1_top_left.png new file mode 100644 index 000000000..4bfefd18b Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/gb200/llama3.1_top_left.png differ diff --git a/benchmark/attention_training/results/llama3.1/gb200/llama3.1_top_left_det_overhead.png b/benchmark/attention_training/results/llama3.1/gb200/llama3.1_top_left_det_overhead.png new file mode 100644 index 000000000..546c8b3fb Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/gb200/llama3.1_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/llama3.1/gb300/llama3.1_20260818_124759.csv b/benchmark/attention_training/results/llama3.1/gb300/llama3.1_20260818_124759.csv new file mode 100644 index 000000000..5bc6dec0e --- /dev/null +++ b/benchmark/attention_training/results/llama3.1/gb300/llama3.1_20260818_124759.csv @@ -0,0 +1,141 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,16.837,2090.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,61.234,1437.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,69.491,1266.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,34.329,2050.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,124.511,1413.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,136.676,1287.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,fwd,False,11.550,3046.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,bwd,False,42.662,2062.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,bwd,True,42.578,2066.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,23.169,3037.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,80.600,2183.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,80.622,2182.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,fwd,False,12.324,2855.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,False,55.320,1590.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,True,54.663,1609.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,24.844,2832.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,105.746,1664.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,106.604,1650.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,18.822,1869.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,36.920,1906.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,32768,32768,64,8,128,128,fwd,False,17.323,2031.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,27.847,2527.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,18.095,1945.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,56.576,1555.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,59.139,1487.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,32.603,2158.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,117.803,1493.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,125.734,1399.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,4.379,2009.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,14.667,1499.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,16.522,1331.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,8.245,2134.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,29.391,1496.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,32.073,1371.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,fwd,False,2.897,3037.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,bwd,False,10.886,2020.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,bwd,True,10.914,2015.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,5.549,3170.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,20.291,2168.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,20.309,2166.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,fwd,False,3.151,2792.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,False,14.097,1560.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,True,14.123,1557.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,5.987,2938.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,26.550,1657.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,25.942,1695.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,4.733,1858.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,8.994,1956.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,16384,16384,64,8,128,128,fwd,False,4.648,1892.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,7.033,2501.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,4.425,1988.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,13.967,1575.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,14.935,1473.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,8.257,2131.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,28.925,1521.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,29.821,1475.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,1.133,1941.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,3.846,1430.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,4.268,1288.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.089,2105.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,7.190,1529.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,8.067,1363.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,fwd,False,0.774,2840.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,bwd,False,3.116,1764.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,bwd,True,3.120,1762.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,1.406,3127.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,5.240,2098.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,5.238,2099.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,fwd,False,0.840,2619.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,False,3.943,1395.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,True,3.940,1396.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,1.517,2899.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,6.770,1624.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,6.775,1623.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,1.307,1682.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.278,1931.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,8192,8192,64,8,128,128,fwd,False,1.344,1636.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,1.816,2421.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,1.174,1873.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,3.763,1461.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,4.181,1315.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.070,2125.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,6.960,1580.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,7.440,1478.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,0.309,1781.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,1.103,1247.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,1.218,1128.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.537,2047.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,1.937,1419.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,2.206,1246.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,fwd,False,0.223,2468.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,bwd,False,0.995,1382.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,bwd,True,0.994,1383.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.375,2932.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,1.509,1822.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,1.510,1821.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,fwd,False,0.242,2277.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,False,1.214,1132.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,True,1.215,1131.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.405,2713.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,1.896,1450.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,1.894,1451.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,0.370,1486.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.590,1865.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,4096,4096,64,8,128,128,fwd,False,0.442,1243.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.503,2185.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,0.338,1627.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,1.136,1210.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,1.421,968.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.549,2004.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,1.911,1438.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,2.169,1267.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.097,1424.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,0.360,954.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,0.392,877.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.147,1868.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.570,1205.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.641,1073.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.072,1904.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,bwd,False,0.368,935.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,bwd,True,0.367,936.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.103,2681.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.499,1377.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.499,1377.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.079,1742.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,False,0.429,801.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,True,0.429,801.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.112,2463.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.598,1149.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.598,1149.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.125,1102.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.159,1727.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.175,785.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.148,1861.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,5155.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.108,1268.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,0.397,866.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,0.535,642.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.147,1871.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.580,1184.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.713,963.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 diff --git a/benchmark/attention_training/results/llama3.1/gb300/llama3.1_no_mask.png b/benchmark/attention_training/results/llama3.1/gb300/llama3.1_no_mask.png new file mode 100644 index 000000000..c7e4efbc7 Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/gb300/llama3.1_no_mask.png differ diff --git a/benchmark/attention_training/results/llama3.1/gb300/llama3.1_no_mask_det_overhead.png b/benchmark/attention_training/results/llama3.1/gb300/llama3.1_no_mask_det_overhead.png new file mode 100644 index 000000000..532394817 Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/gb300/llama3.1_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/llama3.1/gb300/llama3.1_top_left.png b/benchmark/attention_training/results/llama3.1/gb300/llama3.1_top_left.png new file mode 100644 index 000000000..8cba6013b Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/gb300/llama3.1_top_left.png differ diff --git a/benchmark/attention_training/results/llama3.1/gb300/llama3.1_top_left_det_overhead.png b/benchmark/attention_training/results/llama3.1/gb300/llama3.1_top_left_det_overhead.png new file mode 100644 index 000000000..9ff1373ea Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/gb300/llama3.1_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/llama3.1/h200/llama3.1_20260818_125922.csv b/benchmark/attention_training/results/llama3.1/h200/llama3.1_20260818_125922.csv new file mode 100644 index 000000000..e65486d94 --- /dev/null +++ b/benchmark/attention_training/results/llama3.1/h200/llama3.1_20260818_125922.csv @@ -0,0 +1,89 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,57.396,613.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,166.565,528.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,113.495,620.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,327.154,538.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,fwd,False,38.188,921.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,bwd,False,242.472,363.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,bwd,True,242.468,363.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,75.109,937.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,243.582,722.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,244.431,720.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,55.630,632.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,145.207,606.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,154.596,569.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,104.517,673.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,284.883,618.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,290.367,606.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,14.042,626.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,41.225,533.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,48.455,454.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,28.238,623.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,82.050,536.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,93.794,469.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,fwd,False,8.892,989.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,bwd,False,61.819,356.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,bwd,True,61.766,356.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,18.763,938.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,61.064,720.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,60.894,722.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,13.390,657.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,36.174,608.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,38.186,576.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,25.866,680.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,70.821,621.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,72.453,607.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,3.377,651.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,10.245,537.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,12.203,451.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,6.315,696.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,19.575,562.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,22.750,483.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,fwd,False,2.288,961.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,bwd,False,16.055,342.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,bwd,True,16.043,343.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,4.285,1026.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,15.489,710.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,15.486,710.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,3.302,666.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,8.652,635.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,9.672,568.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,5.865,750.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,16.976,648.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,18.133,606.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,0.897,613.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,2.747,500.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,3.151,436.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,1.611,683.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,4.727,581.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,5.581,493.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,fwd,False,0.634,868.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,bwd,False,4.296,320.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,bwd,True,4.291,320.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,1.122,980.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,4.169,659.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,4.171,659.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,0.898,613.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,2.390,575.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,2.786,493.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,1.518,724.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,4.213,652.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,4.543,605.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.258,533.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,0.841,409.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,0.918,375.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.422,652.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,1.306,526.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,1.488,462.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.193,712.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,bwd,False,1.211,284.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,bwd,True,1.211,284.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.306,899.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,1.164,590.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,1.163,591.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,2141.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.267,515.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,0.723,476.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,0.904,380.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.418,657.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,1.172,586.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,1.354,508.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 diff --git a/benchmark/attention_training/results/llama3.1/h200/llama3.1_no_mask.png b/benchmark/attention_training/results/llama3.1/h200/llama3.1_no_mask.png new file mode 100644 index 000000000..9aa739464 Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/h200/llama3.1_no_mask.png differ diff --git a/benchmark/attention_training/results/llama3.1/h200/llama3.1_no_mask_det_overhead.png b/benchmark/attention_training/results/llama3.1/h200/llama3.1_no_mask_det_overhead.png new file mode 100644 index 000000000..bd9ec4847 Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/h200/llama3.1_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/llama3.1/h200/llama3.1_top_left.png b/benchmark/attention_training/results/llama3.1/h200/llama3.1_top_left.png new file mode 100644 index 000000000..37416d14f Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/h200/llama3.1_top_left.png differ diff --git a/benchmark/attention_training/results/llama3.1/h200/llama3.1_top_left_det_overhead.png b/benchmark/attention_training/results/llama3.1/h200/llama3.1_top_left_det_overhead.png new file mode 100644 index 000000000..08bbb418c Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/h200/llama3.1_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_20260818_131244.csv b/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_20260818_131244.csv new file mode 100644 index 000000000..9f6a180e3 --- /dev/null +++ b/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_20260818_131244.csv @@ -0,0 +1,87 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,90.516,389.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,453.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,304.567,289.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,181.827,387.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,602.874,292.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,fwd,False,62.874,560.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,933.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,122.796,573.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,933.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,87.306,403.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,251.768,349.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,359.170,245.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,175.473,401.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,492.434,357.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,695.662,253.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,104.878,335.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,297.020,296.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,187.492,375.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,609.145,289.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,22.490,391.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,78.004,282.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,45.130,390.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,152.394,289.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,fwd,False,15.423,570.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,933.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,30.166,583.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,933.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,21.296,413.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,64.530,341.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,94.242,233.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,43.040,409.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,123.515,356.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,176.659,249.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,25.867,340.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,73.497,299.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,46.418,379.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,151.165,291.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,5.560,396.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,20.185,272.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,21.364,257.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,10.755,409.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,39.377,279.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,42.628,258.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,fwd,False,4.010,549.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,933.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,7.391,595.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,933.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,5.377,409.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,16.695,329.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,462.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,25.057,219.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,10.362,424.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,31.273,352.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,45.549,241.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,6.650,331.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,19.206,286.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,459.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,11.358,387.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,38.173,288.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,1.512,364.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,5.623,244.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,453.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,5.784,238.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,458.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,2.753,399.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,10.587,260.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,453.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,10.778,255.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,453.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,fwd,False,1.098,501.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,904.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,1.903,578.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,904.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,1.454,378.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,4.888,281.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,7.488,184.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,2.671,412.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,8.244,333.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,12.219,225.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,1.772,310.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,5.220,263.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,453.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,2.904,379.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,9.924,277.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.447,308.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,1.720,200.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,1.693,203.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.732,376.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,2.942,234.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,2.977,231.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.345,398.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,904.000 +llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.510,539.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,904.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.448,307.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,1.603,215.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,2.368,145.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.721,381.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,2.407,286.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,3.602,191.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.534,257.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,1.532,224.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.789,348.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,2.669,258.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 diff --git a/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_no_mask.png b/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_no_mask.png new file mode 100644 index 000000000..da36fe7c6 Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_no_mask.png differ diff --git a/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_no_mask_det_overhead.png b/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_no_mask_det_overhead.png new file mode 100644 index 000000000..1a0cfa9cf Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_top_left.png b/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_top_left.png new file mode 100644 index 000000000..4d3f8c565 Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_top_left.png differ diff --git a/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_top_left_det_overhead.png b/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_top_left_det_overhead.png new file mode 100644 index 000000000..0f8842353 Binary files /dev/null and b/benchmark/attention_training/results/llama3.1/rtxpro6000/llama3.1_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/ltx2/gb200/ltx2_20260818_122458.csv b/benchmark/attention_training/results/ltx2/gb200/ltx2_20260818_122458.csv new file mode 100644 index 000000000..ce52ce1ad --- /dev/null +++ b/benchmark/attention_training/results/ltx2/gb200/ltx2_20260818_122458.csv @@ -0,0 +1,36 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,0.445,1388.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,1.210,1277.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,1.355,1141.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,0.470,1316.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,0.592,1044.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,1.163,1330.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,1.254,1233.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,1.820,1611.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,5.270,1391.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,6.489,1129.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,1.962,1494.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,2.531,1158.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,4.954,1479.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,5.284,1387.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,3.075,1642.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,9.580,1318.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,11.499,1098.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,3.239,1559.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,4.259,1186.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,8.274,1526.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,8.784,1437.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,9.470,1582.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,29.222,1282.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,36.059,1039.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,9.416,1591.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,12.723,1178.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,25.084,1493.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,26.255,1427.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,14.178,1637.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,44.687,1298.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,51.614,1124.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,15.607,1487.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2456.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,19.712,1177.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,37.971,1528.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,41.651,1393.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 diff --git a/benchmark/attention_training/results/ltx2/gb200/ltx2_no_mask.png b/benchmark/attention_training/results/ltx2/gb200/ltx2_no_mask.png new file mode 100644 index 000000000..408dd95e6 Binary files /dev/null and b/benchmark/attention_training/results/ltx2/gb200/ltx2_no_mask.png differ diff --git a/benchmark/attention_training/results/ltx2/gb200/ltx2_no_mask_det_overhead.png b/benchmark/attention_training/results/ltx2/gb200/ltx2_no_mask_det_overhead.png new file mode 100644 index 000000000..74e591f1f Binary files /dev/null and b/benchmark/attention_training/results/ltx2/gb200/ltx2_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/ltx2/gb300/ltx2_20260818_123039.csv b/benchmark/attention_training/results/ltx2/gb300/ltx2_20260818_123039.csv new file mode 100644 index 000000000..d0f380313 --- /dev/null +++ b/benchmark/attention_training/results/ltx2/gb300/ltx2_20260818_123039.csv @@ -0,0 +1,36 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,0.336,1843.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,1.086,1424.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,1.209,1279.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,0.369,1678.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,0.339,1826.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,1.039,1488.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,1.233,1254.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,1.443,2031.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,4.769,1537.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,5.765,1271.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,1.605,1826.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,1.438,2038.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,4.446,1648.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,4.912,1492.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,2.380,2122.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,8.107,1557.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,9.727,1298.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,2.639,1913.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,2.354,2145.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,7.501,1683.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,8.430,1498.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,7.061,2122.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,24.475,1530.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,29.684,1262.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,7.831,1913.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,6.980,2147.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,23.313,1607.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,24.972,1500.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,11.082,2094.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,38.074,1524.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,40.880,1419.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,11.856,1957.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,10.874,2134.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,34.868,1664.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,41.036,1414.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 diff --git a/benchmark/attention_training/results/ltx2/gb300/ltx2_no_mask.png b/benchmark/attention_training/results/ltx2/gb300/ltx2_no_mask.png new file mode 100644 index 000000000..cf26332d8 Binary files /dev/null and b/benchmark/attention_training/results/ltx2/gb300/ltx2_no_mask.png differ diff --git a/benchmark/attention_training/results/ltx2/gb300/ltx2_no_mask_det_overhead.png b/benchmark/attention_training/results/ltx2/gb300/ltx2_no_mask_det_overhead.png new file mode 100644 index 000000000..50532c603 Binary files /dev/null and b/benchmark/attention_training/results/ltx2/gb300/ltx2_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/ltx2/h200/ltx2_20260818_122311.csv b/benchmark/attention_training/results/ltx2/h200/ltx2_20260818_122311.csv new file mode 100644 index 000000000..ab6aebaf4 --- /dev/null +++ b/benchmark/attention_training/results/ltx2/h200/ltx2_20260818_122311.csv @@ -0,0 +1,30 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,0.913,678.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,2.625,589.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,3.078,502.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,0.855,723.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,2.325,665.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,2.514,615.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,4.642,631.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,12.842,571.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,14.952,490.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,3.942,744.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,11.055,663.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,11.235,652.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,8.132,621.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,22.445,562.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,26.212,482.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,6.769,746.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,19.592,644.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,20.146,627.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,25.682,583.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,67.103,558.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,78.556,477.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,21.973,682.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,58.867,636.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,60.171,622.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,37.160,624.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,106.600,544.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,33.847,686.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,92.258,629.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,95.002,611.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 diff --git a/benchmark/attention_training/results/ltx2/h200/ltx2_no_mask.png b/benchmark/attention_training/results/ltx2/h200/ltx2_no_mask.png new file mode 100644 index 000000000..dd98fa282 Binary files /dev/null and b/benchmark/attention_training/results/ltx2/h200/ltx2_no_mask.png differ diff --git a/benchmark/attention_training/results/ltx2/h200/ltx2_no_mask_det_overhead.png b/benchmark/attention_training/results/ltx2/h200/ltx2_no_mask_det_overhead.png new file mode 100644 index 000000000..29d9b56cb Binary files /dev/null and b/benchmark/attention_training/results/ltx2/h200/ltx2_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/ltx2/rtxpro6000/ltx2_20260818_122308.csv b/benchmark/attention_training/results/ltx2/rtxpro6000/ltx2_20260818_122308.csv new file mode 100644 index 000000000..25e30e87e --- /dev/null +++ b/benchmark/attention_training/results/ltx2/rtxpro6000/ltx2_20260818_122308.csv @@ -0,0 +1,39 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,1.791,345.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,450.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,5.932,261.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,450.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,6.596,234.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,450.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,1.732,357.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,4.396,352.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,6.719,230.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,1.922,322.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,7.230,214.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,7.161,409.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,25.857,283.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,453.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,29.331,250.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,6.904,425.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,19.973,367.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,30.334,242.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,7.746,378.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,32.863,223.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,12.573,402.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,51.329,246.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,50.598,250.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,12.022,420.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,34.951,361.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,51.537,245.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,13.516,374.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,56.259,224.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,38.258,392.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,150.729,249.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,37.112,404.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,103.500,362.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,151.271,248.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,39.864,376.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,164.785,227.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,59.811,388.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,197.424,294.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,57.688,402.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,159.914,363.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,230.769,251.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,62.676,370.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,254.244,228.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 diff --git a/benchmark/attention_training/results/ltx2/rtxpro6000/ltx2_no_mask.png b/benchmark/attention_training/results/ltx2/rtxpro6000/ltx2_no_mask.png new file mode 100644 index 000000000..97ab4066a Binary files /dev/null and b/benchmark/attention_training/results/ltx2/rtxpro6000/ltx2_no_mask.png differ diff --git a/benchmark/attention_training/results/ltx2/rtxpro6000/ltx2_no_mask_det_overhead.png b/benchmark/attention_training/results/ltx2/rtxpro6000/ltx2_no_mask_det_overhead.png new file mode 100644 index 000000000..170d40969 Binary files /dev/null and b/benchmark/attention_training/results/ltx2/rtxpro6000/ltx2_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/qwen35/gb200/qwen35_20260818_122507.csv b/benchmark/attention_training/results/qwen35/gb200/qwen35_20260818_122507.csv new file mode 100644 index 000000000..0df7348d8 --- /dev/null +++ b/benchmark/attention_training/results/qwen35/gb200/qwen35_20260818_122507.csv @@ -0,0 +1,31 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,10.261,1714.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,43.324,1015.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,True,43.234,1017.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,10.083,1745.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,10.849,1622.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,48.441,908.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,2.428,1812.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,10.706,1027.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,True,9.778,1125.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,2.391,1840.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,3.197,1376.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,12.522,878.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,0.635,1730.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,2.703,1017.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,True,2.696,1020.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,0.670,1641.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,1.128,975.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,3.220,854.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.188,1463.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,0.788,873.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,True,0.792,868.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.204,1348.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.463,594.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,1.405,489.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.058,1191.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,0.270,637.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,1756.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,True,0.271,635.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.071,964.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2549.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.221,311.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,0.667,258.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 diff --git a/benchmark/attention_training/results/qwen35/gb200/qwen35_top_left.png b/benchmark/attention_training/results/qwen35/gb200/qwen35_top_left.png new file mode 100644 index 000000000..c7a570242 Binary files /dev/null and b/benchmark/attention_training/results/qwen35/gb200/qwen35_top_left.png differ diff --git a/benchmark/attention_training/results/qwen35/gb200/qwen35_top_left_det_overhead.png b/benchmark/attention_training/results/qwen35/gb200/qwen35_top_left_det_overhead.png new file mode 100644 index 000000000..8a59653cc Binary files /dev/null and b/benchmark/attention_training/results/qwen35/gb200/qwen35_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/qwen35/gb300/qwen35_20260818_123050.csv b/benchmark/attention_training/results/qwen35/gb300/qwen35_20260818_123050.csv new file mode 100644 index 000000000..34150d6ba --- /dev/null +++ b/benchmark/attention_training/results/qwen35/gb300/qwen35_20260818_123050.csv @@ -0,0 +1,31 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,8.317,2115.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,37.344,1178.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,True,37.290,1179.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,8.140,2161.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,10.365,1697.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,43.523,1011.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,2.073,2122.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,9.019,1219.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,True,9.059,1214.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,2.093,2102.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,3.101,1418.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,11.122,989.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,0.537,2046.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,2.425,1134.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,True,2.427,1133.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,0.562,1958.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,1.119,983.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,3.106,885.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.149,1840.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,0.716,960.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,True,0.718,957.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.167,1651.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.460,598.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,1.357,507.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.049,1395.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,0.248,693.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,True,0.247,695.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.059,1175.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2250.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.220,313.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,0.643,267.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 diff --git a/benchmark/attention_training/results/qwen35/gb300/qwen35_top_left.png b/benchmark/attention_training/results/qwen35/gb300/qwen35_top_left.png new file mode 100644 index 000000000..bdfe580f0 Binary files /dev/null and b/benchmark/attention_training/results/qwen35/gb300/qwen35_top_left.png differ diff --git a/benchmark/attention_training/results/qwen35/gb300/qwen35_top_left_det_overhead.png b/benchmark/attention_training/results/qwen35/gb300/qwen35_top_left_det_overhead.png new file mode 100644 index 000000000..95a7e8836 Binary files /dev/null and b/benchmark/attention_training/results/qwen35/gb300/qwen35_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/qwen35/h200/qwen35_20260818_132623.csv b/benchmark/attention_training/results/qwen35/h200/qwen35_20260818_132623.csv new file mode 100644 index 000000000..2a3854ddd --- /dev/null +++ b/benchmark/attention_training/results/qwen35/h200/qwen35_20260818_132623.csv @@ -0,0 +1,31 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,26.173,672.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,124.951,352.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,True,137.501,320.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,23.923,735.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,140.529,313.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,True,141.746,310.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,6.094,722.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,31.015,355.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,True,34.371,320.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,5.528,796.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,34.170,322.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,True,34.284,321.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,1.607,684.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,7.002,393.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,True,7.879,349.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,1.419,775.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,8.839,311.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,True,9.045,304.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.449,612.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,1.939,355.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,True,2.212,311.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.381,722.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,2.409,285.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,True,2.497,275.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.142,485.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,0.611,281.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,True,0.668,257.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.111,620.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,0.704,244.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,True,0.775,222.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 diff --git a/benchmark/attention_training/results/qwen35/h200/qwen35_top_left.png b/benchmark/attention_training/results/qwen35/h200/qwen35_top_left.png new file mode 100644 index 000000000..493ffa581 Binary files /dev/null and b/benchmark/attention_training/results/qwen35/h200/qwen35_top_left.png differ diff --git a/benchmark/attention_training/results/qwen35/h200/qwen35_top_left_det_overhead.png b/benchmark/attention_training/results/qwen35/h200/qwen35_top_left_det_overhead.png new file mode 100644 index 000000000..de79919f4 Binary files /dev/null and b/benchmark/attention_training/results/qwen35/h200/qwen35_top_left_det_overhead.png differ diff --git a/benchmark/attention_training/results/qwen35/rtxpro6000/qwen35_20260818_133907.csv b/benchmark/attention_training/results/qwen35/rtxpro6000/qwen35_20260818_133907.csv new file mode 100644 index 000000000..37f286e33 --- /dev/null +++ b/benchmark/attention_training/results/qwen35/rtxpro6000/qwen35_20260818_133907.csv @@ -0,0 +1,11 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,107.173,164.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,47.376,371.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,453.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,27.198,162.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,456.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,11.416,385.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,6.994,157.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,2.955,372.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,1.863,148.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.806,341.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.532,129.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.308,223.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 diff --git a/benchmark/attention_training/results/qwen35/rtxpro6000/qwen35_top_left.png b/benchmark/attention_training/results/qwen35/rtxpro6000/qwen35_top_left.png new file mode 100644 index 000000000..c70bf6db6 Binary files /dev/null and b/benchmark/attention_training/results/qwen35/rtxpro6000/qwen35_top_left.png differ diff --git a/benchmark/attention_training/results/wan22/gb200/wan22_20260818_122522.csv b/benchmark/attention_training/results/wan22/gb200/wan22_20260818_122522.csv new file mode 100644 index 000000000..0d83b827b --- /dev/null +++ b/benchmark/attention_training/results/wan22/gb200/wan22_20260818_122522.csv @@ -0,0 +1,36 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,0.793,1570.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,2.334,1335.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,2.871,1085.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,0.899,1386.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,1.132,1101.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,2.201,1416.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,2.367,1316.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,3.629,1662.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,10.756,1402.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,14.596,1033.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,3.823,1577.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,4.918,1226.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,9.786,1541.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,10.523,1433.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,14.417,1525.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,43.048,1276.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,54.241,1013.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,14.486,1517.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,17.895,1228.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,37.074,1482.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,40.940,1342.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,32.344,1481.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,100.324,1194.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,124.065,965.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,32.054,1494.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,40.589,1180.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,86.312,1387.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,94.079,1273.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,83.758,1397.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,243.172,1203.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,308.334,949.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,83.894,1395.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,108.625,1078.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,218.518,1339.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,244.671,1196.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92600.000,2568.000 diff --git a/benchmark/attention_training/results/wan22/gb200/wan22_no_mask.png b/benchmark/attention_training/results/wan22/gb200/wan22_no_mask.png new file mode 100644 index 000000000..170d126e8 Binary files /dev/null and b/benchmark/attention_training/results/wan22/gb200/wan22_no_mask.png differ diff --git a/benchmark/attention_training/results/wan22/gb200/wan22_no_mask_det_overhead.png b/benchmark/attention_training/results/wan22/gb200/wan22_no_mask_det_overhead.png new file mode 100644 index 000000000..36d413889 Binary files /dev/null and b/benchmark/attention_training/results/wan22/gb200/wan22_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/wan22/gb300/wan22_20260818_123102.csv b/benchmark/attention_training/results/wan22/gb300/wan22_20260818_123102.csv new file mode 100644 index 000000000..654d5d496 --- /dev/null +++ b/benchmark/attention_training/results/wan22/gb300/wan22_20260818_123102.csv @@ -0,0 +1,36 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,0.647,1924.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,2.168,1437.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,2.613,1192.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,0.718,1735.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,0.659,1892.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,2.038,1529.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,2.258,1380.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,2.929,2059.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,9.816,1536.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,11.922,1265.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,3.164,1906.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,2.892,2085.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,9.160,1646.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,10.242,1472.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,10.382,2117.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,38.061,1444.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,43.062,1276.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,11.399,1928.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,10.324,2129.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,33.153,1657.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,38.062,1444.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,23.796,2013.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,86.162,1390.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,101.644,1178.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,26.153,1831.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,22.415,2137.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,76.643,1562.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,90.343,1325.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,59.146,1979.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,214.344,1365.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,253.620,1154.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,64.538,1814.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,60.805,1925.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,192.207,1522.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,229.827,1273.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92600.000,2578.000 diff --git a/benchmark/attention_training/results/wan22/gb300/wan22_no_mask.png b/benchmark/attention_training/results/wan22/gb300/wan22_no_mask.png new file mode 100644 index 000000000..29c15e370 Binary files /dev/null and b/benchmark/attention_training/results/wan22/gb300/wan22_no_mask.png differ diff --git a/benchmark/attention_training/results/wan22/gb300/wan22_no_mask_det_overhead.png b/benchmark/attention_training/results/wan22/gb300/wan22_no_mask_det_overhead.png new file mode 100644 index 000000000..c99c4eee8 Binary files /dev/null and b/benchmark/attention_training/results/wan22/gb300/wan22_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/wan22/h200/wan22_20260818_133020.csv b/benchmark/attention_training/results/wan22/h200/wan22_20260818_133020.csv new file mode 100644 index 000000000..8b9bd83e3 --- /dev/null +++ b/benchmark/attention_training/results/wan22/h200/wan22_20260818_133020.csv @@ -0,0 +1,28 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,1.978,630.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,5.221,597.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,6.355,490.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,1.695,735.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,4.615,675.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,4.901,636.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,9.885,610.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,26.735,564.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,31.295,482.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,8.066,748.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,23.098,653.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,24.022,628.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,38.345,573.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,99.337,553.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,32.309,680.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,88.108,624.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,89.851,612.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,83.526,573.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,218.597,548.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,70.317,681.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,192.201,623.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,195.896,611.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,207.773,563.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,537.143,545.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,174.625,670.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,470.768,622.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,480.075,610.000,0.000,10,,True,False,,NVIDIA H200,1.27.0,92600.000,1071.000 diff --git a/benchmark/attention_training/results/wan22/h200/wan22_no_mask.png b/benchmark/attention_training/results/wan22/h200/wan22_no_mask.png new file mode 100644 index 000000000..b9c6dd271 Binary files /dev/null and b/benchmark/attention_training/results/wan22/h200/wan22_no_mask.png differ diff --git a/benchmark/attention_training/results/wan22/h200/wan22_no_mask_det_overhead.png b/benchmark/attention_training/results/wan22/h200/wan22_no_mask_det_overhead.png new file mode 100644 index 000000000..867cda6dc Binary files /dev/null and b/benchmark/attention_training/results/wan22/h200/wan22_no_mask_det_overhead.png differ diff --git a/benchmark/attention_training/results/wan22/rtxpro6000/wan22_20260818_134432.csv b/benchmark/attention_training/results/wan22/rtxpro6000/wan22_20260818_134432.csv new file mode 100644 index 000000000..eb2e27b50 --- /dev/null +++ b/benchmark/attention_training/results/wan22/rtxpro6000/wan22_20260818_134432.csv @@ -0,0 +1,38 @@ +config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,3.087,404.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,12.999,240.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,12.767,244.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,2.991,417.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,8.649,360.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,13.329,234.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,3.305,377.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,14.157,220.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,458.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,15.234,396.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,452.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,61.328,246.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,60.815,248.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,14.367,420.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,42.529,355.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,61.578,245.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,15.951,378.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,67.135,225.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,56.728,387.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,220.024,250.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,54.066,407.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,152.515,360.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,223.111,246.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,58.690,375.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,240.630,228.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,125.342,382.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,478.876,250.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,118.755,403.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,334.198,358.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,484.704,247.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,130.233,368.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,523.068,229.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,309.743,378.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,1163.473,252.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,294.883,397.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,826.635,354.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,1161.375,252.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,322.109,363.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 +wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,1274.466,230.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92600.000,466.000 diff --git a/benchmark/attention_training/results/wan22/rtxpro6000/wan22_no_mask.png b/benchmark/attention_training/results/wan22/rtxpro6000/wan22_no_mask.png new file mode 100644 index 000000000..9d1b89ba1 Binary files /dev/null and b/benchmark/attention_training/results/wan22/rtxpro6000/wan22_no_mask.png differ diff --git a/benchmark/attention_training/results/wan22/rtxpro6000/wan22_no_mask_det_overhead.png b/benchmark/attention_training/results/wan22/rtxpro6000/wan22_no_mask_det_overhead.png new file mode 100644 index 000000000..c5a4ba4c6 Binary files /dev/null and b/benchmark/attention_training/results/wan22/rtxpro6000/wan22_no_mask_det_overhead.png differ diff --git a/benchmark/sdpa_benchmark_training/runner.py b/benchmark/attention_training/runner.py similarity index 96% rename from benchmark/sdpa_benchmark_training/runner.py rename to benchmark/attention_training/runner.py index 37fe8879a..23d31e6be 100644 --- a/benchmark/sdpa_benchmark_training/runner.py +++ b/benchmark/attention_training/runner.py @@ -9,12 +9,12 @@ Usage: # Run from command line - python -m benchmark.sdpa_benchmark_training.runner --config mlperf - python -m benchmark.sdpa_benchmark_training.runner --config mlperf --dry-run + python -m benchmark.attention_training.runner --config mlperf + python -m benchmark.attention_training.runner --config mlperf --dry-run # Import and use programmatically - from benchmark.sdpa_benchmark_training.runner import BenchmarkRunner - from benchmark.sdpa_benchmark_training.configs import load_config + from benchmark.attention_training.runner import BenchmarkRunner + from benchmark.attention_training.configs import load_config config = load_config("mlperf") runner = BenchmarkRunner() @@ -416,19 +416,19 @@ def main(): epilog=""" Examples: # Run all benchmarks from mlperf config - python -m benchmark.sdpa_benchmark_training.runner --config mlperf + python -m benchmark.attention_training.runner --config mlperf # Dry run (show what would be executed) - python -m benchmark.sdpa_benchmark_training.runner --config mlperf --dry-run + python -m benchmark.attention_training.runner --config mlperf --dry-run # Filter by model name - python -m benchmark.sdpa_benchmark_training.runner --config mlperf --filter llama3.1 + python -m benchmark.attention_training.runner --config mlperf --filter llama3.1 # Filter by backend - python -m benchmark.sdpa_benchmark_training.runner --config mlperf --backend cudnn + python -m benchmark.attention_training.runner --config mlperf --backend cudnn # Skip chart generation - python -m benchmark.sdpa_benchmark_training.runner --config mlperf --no-chart + python -m benchmark.attention_training.runner --config mlperf --no-chart """, ) diff --git a/benchmark/cutedsl_fusion_kernels/results/B200/output_verbose.txt b/benchmark/cutedsl_fusion_kernels/results/B200/output_verbose.txt deleted file mode 100644 index f3c7abdec..000000000 --- a/benchmark/cutedsl_fusion_kernels/results/B200/output_verbose.txt +++ /dev/null @@ -1,71 +0,0 @@ -device: NVIDIA GB200 -shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 -dtype: torch.bfloat16 -torch: 2.12.0a0+5aff3928d8.nv26.05 -transformer_engine: 2.15.0+42b84005 -nvidia-cudnn-frontend: 1.23.0 -nvidia-cutlass-dsl: 4.4.1 -timing mode: CUDA graph replay - -activation: swiglu -shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 -fused GEMM N=8192, fused C input N=None, activation output N=4096 -fused cuDNN avg: 1.550 ms -unfused TE avg: 1.934 ms -speedup vs TE: 1.25x -fused samples: [1.357, 1.425, 1.64, 1.667, 1.663] -TE samples: [2.212, 1.937, 1.814, 1.839, 1.869] - -activation: dswiglu -shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 -fused GEMM N=4096, fused C input N=8192, activation output N=8192 -fused cuDNN avg: 0.870 ms -unfused TE avg: 1.734 ms -speedup vs TE: 1.99x -fused samples: [0.806, 0.806, 0.793, 0.963, 0.98] -TE samples: [2.254, 1.639, 1.626, 1.579, 1.571] - -activation: srelu -shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 -fused GEMM N=4096, fused C input N=None, activation output N=4096 -fused cuDNN avg: 0.739 ms -unfused TE avg: 1.041 ms -speedup vs TE: 1.41x -fused samples: [0.721, 0.718, 0.718, 0.734, 0.803] -TE samples: [1.1, 1.107, 1.023, 1.008, 0.97] - -activation: dsrelu -shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 -fused GEMM N=4096, fused C input N=4096, activation output N=4096 -fused cuDNN avg: 0.781 ms -unfused TE avg: 1.341 ms -speedup vs TE: 1.72x -fused samples: [0.731, 0.729, 0.73, 0.848, 0.866] -TE samples: [1.539, 1.386, 1.27, 1.265, 1.243] - -activation: geglu -shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 -fused GEMM N=8192, fused C input N=None, activation output N=4096 -fused cuDNN avg: 1.544 ms -unfused TE avg: 1.890 ms -speedup vs TE: 1.22x -fused samples: [1.358, 1.523, 1.6, 1.647, 1.595] -TE samples: [2.036, 1.827, 1.842, 1.87, 1.874] - -activation: dgeglu -shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 -fused GEMM N=4096, fused C input N=8192, activation output N=8192 -fused cuDNN avg: 0.889 ms -unfused TE avg: 1.820 ms -speedup vs TE: 2.05x -fused samples: [0.8, 0.795, 0.896, 0.951, 1.003] -TE samples: [2.369, 1.717, 1.668, 1.702, 1.644] - -summary: -activation cudnn_ms te_ms te/cudnn -swiglu 1.550 1.934 1.25x -dswiglu 0.870 1.734 1.99x -srelu 0.739 1.041 1.41x -dsrelu 0.781 1.341 1.72x -geglu 1.544 1.890 1.22x -dgeglu 0.889 1.820 2.05x diff --git a/benchmark/cutedsl_fusion_kernels/results/B200/results.png b/benchmark/cutedsl_fusion_kernels/results/B200/results.png deleted file mode 100644 index 36795201b..000000000 Binary files a/benchmark/cutedsl_fusion_kernels/results/B200/results.png and /dev/null differ diff --git a/benchmark/cutedsl_fusion_kernels/results/B300/output_verbose.txt b/benchmark/cutedsl_fusion_kernels/results/B300/output_verbose.txt deleted file mode 100644 index b19d83aab..000000000 --- a/benchmark/cutedsl_fusion_kernels/results/B300/output_verbose.txt +++ /dev/null @@ -1,71 +0,0 @@ -device: NVIDIA GB300 -shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 -dtype: torch.bfloat16 -torch: 2.12.0a0+5aff3928d8.nv26.05 -transformer_engine: 2.15.0+42b84005 -nvidia-cudnn-frontend: 1.23.0 -nvidia-cutlass-dsl: 4.4.1 -timing mode: CUDA graph replay - -activation: swiglu -shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 -fused GEMM N=8192, fused C input N=None, activation output N=4096 -fused cuDNN avg: 1.319 ms -unfused TE avg: 1.695 ms -speedup vs TE: 1.28x -fused samples: [1.233, 1.243, 1.374, 1.378, 1.369] -TE samples: [1.82, 1.612, 1.603, 1.703, 1.738] - -activation: dswiglu -shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 -fused GEMM N=4096, fused C input N=8192, activation output N=8192 -fused cuDNN avg: 0.740 ms -unfused TE avg: 1.549 ms -speedup vs TE: 2.09x -fused samples: [0.71, 0.716, 0.718, 0.763, 0.793] -TE samples: [1.856, 1.512, 1.464, 1.457, 1.454] - -activation: srelu -shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 -fused GEMM N=4096, fused C input N=None, activation output N=4096 -fused cuDNN avg: 0.675 ms -unfused TE avg: 0.873 ms -speedup vs TE: 1.29x -fused samples: [0.653, 0.654, 0.686, 0.677, 0.705] -TE samples: [0.942, 0.921, 0.85, 0.842, 0.811] - -activation: dsrelu -shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 -fused GEMM N=4096, fused C input N=4096, activation output N=4096 -fused cuDNN avg: 0.686 ms -unfused TE avg: 1.163 ms -speedup vs TE: 1.70x -fused samples: [0.659, 0.666, 0.677, 0.688, 0.743] -TE samples: [1.343, 1.167, 1.116, 1.096, 1.096] - -activation: geglu -shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 -fused GEMM N=8192, fused C input N=None, activation output N=4096 -fused cuDNN avg: 1.317 ms -unfused TE avg: 1.631 ms -speedup vs TE: 1.24x -fused samples: [1.231, 1.245, 1.344, 1.376, 1.389] -TE samples: [1.735, 1.626, 1.58, 1.65, 1.565] - -activation: dgeglu -shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 -fused GEMM N=4096, fused C input N=8192, activation output N=8192 -fused cuDNN avg: 0.751 ms -unfused TE avg: 1.666 ms -speedup vs TE: 2.22x -fused samples: [0.716, 0.717, 0.75, 0.774, 0.795] -TE samples: [2.058, 1.595, 1.569, 1.557, 1.553] - -summary: -activation cudnn_ms te_ms te/cudnn -swiglu 1.319 1.695 1.28x -dswiglu 0.740 1.549 2.09x -srelu 0.675 0.873 1.29x -dsrelu 0.686 1.163 1.70x -geglu 1.317 1.631 1.24x -dgeglu 0.751 1.666 2.22x diff --git a/benchmark/cutedsl_fusion_kernels/results/B300/results.png b/benchmark/cutedsl_fusion_kernels/results/B300/results.png deleted file mode 100644 index 178841447..000000000 Binary files a/benchmark/cutedsl_fusion_kernels/results/B300/results.png and /dev/null differ diff --git a/benchmark/cutedsl_fusion_kernels/results/B200/output.txt b/benchmark/cutedsl_fusion_kernels/results/GB200/output.txt similarity index 52% rename from benchmark/cutedsl_fusion_kernels/results/B200/output.txt rename to benchmark/cutedsl_fusion_kernels/results/GB200/output.txt index eec7c8195..4da2c80a2 100644 --- a/benchmark/cutedsl_fusion_kernels/results/B200/output.txt +++ b/benchmark/cutedsl_fusion_kernels/results/GB200/output.txt @@ -8,9 +8,9 @@ nvidia-cutlass-dsl: 4.4.1 summary: activation cudnn_ms te_ms te/cudnn -swiglu 1.550 1.934 1.25x -dswiglu 0.870 1.734 1.99x -srelu 0.739 1.041 1.41x -dsrelu 0.781 1.341 1.72x -geglu 1.544 1.890 1.22x -dgeglu 0.889 1.820 2.05x +swiglu 1.548 1.882 1.22x +dswiglu 0.884 1.716 1.94x +srelu 0.797 1.043 1.31x +dsrelu 0.811 1.346 1.66x +geglu 1.523 1.913 1.26x +dgeglu 0.892 1.840 2.06x diff --git a/benchmark/cutedsl_fusion_kernels/results/GB200/output_verbose.txt b/benchmark/cutedsl_fusion_kernels/results/GB200/output_verbose.txt new file mode 100644 index 000000000..f6faa7e26 --- /dev/null +++ b/benchmark/cutedsl_fusion_kernels/results/GB200/output_verbose.txt @@ -0,0 +1,71 @@ +device: NVIDIA GB200 +shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 +dtype: torch.bfloat16 +torch: 2.12.0a0+5aff3928d8.nv26.05 +transformer_engine: 2.15.0+42b84005 +nvidia-cudnn-frontend: 1.23.0 +nvidia-cutlass-dsl: 4.4.1 +timing mode: CUDA graph replay + +activation: swiglu +shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 +fused GEMM N=8192, fused C input N=None, activation output N=4096 +fused cuDNN avg: 1.548 ms +unfused TE avg: 1.882 ms +speedup vs TE: 1.22x +fused samples: [1.401, 1.509, 1.625, 1.638, 1.568] +TE samples: [1.968, 1.857, 1.853, 1.863, 1.869] + +activation: dswiglu +shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 +fused GEMM N=4096, fused C input N=8192, activation output N=8192 +fused cuDNN avg: 0.884 ms +unfused TE avg: 1.716 ms +speedup vs TE: 1.94x +fused samples: [0.801, 0.805, 0.917, 0.935, 0.964] +TE samples: [2.234, 1.651, 1.583, 1.548, 1.566] + +activation: srelu +shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 +fused GEMM N=4096, fused C input N=None, activation output N=4096 +fused cuDNN avg: 0.797 ms +unfused TE avg: 1.043 ms +speedup vs TE: 1.31x +fused samples: [0.757, 0.741, 0.775, 0.852, 0.86] +TE samples: [1.137, 1.09, 1.005, 0.989, 0.992] + +activation: dsrelu +shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 +fused GEMM N=4096, fused C input N=4096, activation output N=4096 +fused cuDNN avg: 0.811 ms +unfused TE avg: 1.346 ms +speedup vs TE: 1.66x +fused samples: [0.752, 0.746, 0.823, 0.855, 0.878] +TE samples: [1.548, 1.347, 1.283, 1.275, 1.278] + +activation: geglu +shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 +fused GEMM N=8192, fused C input N=None, activation output N=4096 +fused cuDNN avg: 1.523 ms +unfused TE avg: 1.913 ms +speedup vs TE: 1.26x +fused samples: [1.415, 1.398, 1.515, 1.626, 1.66] +TE samples: [2.082, 1.878, 1.881, 1.867, 1.858] + +activation: dgeglu +shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 +fused GEMM N=4096, fused C input N=8192, activation output N=8192 +fused cuDNN avg: 0.892 ms +unfused TE avg: 1.840 ms +speedup vs TE: 2.06x +fused samples: [0.812, 0.832, 0.9, 0.932, 0.982] +TE samples: [2.319, 1.805, 1.723, 1.691, 1.664] + +summary: +activation cudnn_ms te_ms te/cudnn +swiglu 1.548 1.882 1.22x +dswiglu 0.884 1.716 1.94x +srelu 0.797 1.043 1.31x +dsrelu 0.811 1.346 1.66x +geglu 1.523 1.913 1.26x +dgeglu 0.892 1.840 2.06x diff --git a/benchmark/cutedsl_fusion_kernels/results/GB200/results.png b/benchmark/cutedsl_fusion_kernels/results/GB200/results.png new file mode 100644 index 000000000..f7d94dfaa Binary files /dev/null and b/benchmark/cutedsl_fusion_kernels/results/GB200/results.png differ diff --git a/benchmark/cutedsl_fusion_kernels/results/B300/output.txt b/benchmark/cutedsl_fusion_kernels/results/GB300/output.txt similarity index 52% rename from benchmark/cutedsl_fusion_kernels/results/B300/output.txt rename to benchmark/cutedsl_fusion_kernels/results/GB300/output.txt index 4ebfb6cab..2131c998b 100644 --- a/benchmark/cutedsl_fusion_kernels/results/B300/output.txt +++ b/benchmark/cutedsl_fusion_kernels/results/GB300/output.txt @@ -8,9 +8,9 @@ nvidia-cutlass-dsl: 4.4.1 summary: activation cudnn_ms te_ms te/cudnn -swiglu 1.319 1.695 1.28x -dswiglu 0.740 1.549 2.09x -srelu 0.675 0.873 1.29x -dsrelu 0.686 1.163 1.70x -geglu 1.317 1.631 1.24x -dgeglu 0.751 1.666 2.22x +swiglu 1.319 1.650 1.25x +dswiglu 0.737 1.551 2.10x +srelu 0.676 0.886 1.31x +dsrelu 0.665 1.143 1.72x +geglu 1.329 1.653 1.24x +dgeglu 0.742 1.637 2.20x diff --git a/benchmark/cutedsl_fusion_kernels/results/GB300/output_verbose.txt b/benchmark/cutedsl_fusion_kernels/results/GB300/output_verbose.txt new file mode 100644 index 000000000..20de3a7b5 --- /dev/null +++ b/benchmark/cutedsl_fusion_kernels/results/GB300/output_verbose.txt @@ -0,0 +1,71 @@ +device: NVIDIA GB300 +shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 +dtype: torch.bfloat16 +torch: 2.12.0a0+5aff3928d8.nv26.05 +transformer_engine: 2.15.0+42b84005 +nvidia-cudnn-frontend: 1.23.0 +nvidia-cutlass-dsl: 4.4.1 +timing mode: CUDA graph replay + +activation: swiglu +shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 +fused GEMM N=8192, fused C input N=None, activation output N=4096 +fused cuDNN avg: 1.319 ms +unfused TE avg: 1.650 ms +speedup vs TE: 1.25x +fused samples: [1.208, 1.278, 1.354, 1.385, 1.371] +TE samples: [1.755, 1.632, 1.587, 1.602, 1.674] + +activation: dswiglu +shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 +fused GEMM N=4096, fused C input N=8192, activation output N=8192 +fused cuDNN avg: 0.737 ms +unfused TE avg: 1.551 ms +speedup vs TE: 2.10x +fused samples: [0.704, 0.704, 0.741, 0.771, 0.767] +TE samples: [1.961, 1.464, 1.463, 1.437, 1.432] + +activation: srelu +shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 +fused GEMM N=4096, fused C input N=None, activation output N=4096 +fused cuDNN avg: 0.676 ms +unfused TE avg: 0.886 ms +speedup vs TE: 1.31x +fused samples: [0.646, 0.643, 0.656, 0.716, 0.716] +TE samples: [0.943, 0.952, 0.871, 0.832, 0.832] + +activation: dsrelu +shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 +fused GEMM N=4096, fused C input N=4096, activation output N=4096 +fused cuDNN avg: 0.665 ms +unfused TE avg: 1.143 ms +speedup vs TE: 1.72x +fused samples: [0.65, 0.661, 0.669, 0.66, 0.685] +TE samples: [1.341, 1.132, 1.092, 1.077, 1.074] + +activation: geglu +shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 +fused GEMM N=8192, fused C input N=None, activation output N=4096 +fused cuDNN avg: 1.329 ms +unfused TE avg: 1.653 ms +speedup vs TE: 1.24x +fused samples: [1.21, 1.3, 1.361, 1.395, 1.377] +TE samples: [1.721, 1.662, 1.613, 1.685, 1.582] + +activation: dgeglu +shape: experts=8, tokens/expert=4096, M=32768, K=8192, N=4096 +fused GEMM N=4096, fused C input N=8192, activation output N=8192 +fused cuDNN avg: 0.742 ms +unfused TE avg: 1.637 ms +speedup vs TE: 2.20x +fused samples: [0.711, 0.718, 0.745, 0.746, 0.793] +TE samples: [2.005, 1.565, 1.544, 1.538, 1.531] + +summary: +activation cudnn_ms te_ms te/cudnn +swiglu 1.319 1.650 1.25x +dswiglu 0.737 1.551 2.10x +srelu 0.676 0.886 1.31x +dsrelu 0.665 1.143 1.72x +geglu 1.329 1.653 1.24x +dgeglu 0.742 1.637 2.20x diff --git a/benchmark/cutedsl_fusion_kernels/results/GB300/results.png b/benchmark/cutedsl_fusion_kernels/results/GB300/results.png new file mode 100644 index 000000000..5796bea40 Binary files /dev/null and b/benchmark/cutedsl_fusion_kernels/results/GB300/results.png differ diff --git a/benchmark/sdpa_benchmark_training/configs/qwen3vl_vit.py b/benchmark/sdpa_benchmark_training/configs/qwen3vl_vit.py deleted file mode 100644 index 877478016..000000000 --- a/benchmark/sdpa_benchmark_training/configs/qwen3vl_vit.py +++ /dev/null @@ -1,66 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -""" -Qwen3-VL Vision Encoder (ViT) SDPA Benchmark Configuration - -Benchmarks the self-attention of the Qwen3-VL vision encoder as exercised -by image/video inference. ViT self-attention is bidirectional over the -patchified image tokens, so only ``no_mask`` is benchmarked, forward pass -only (inference workload). - -Architecture (vision tower): - - hidden dim = 1152 - - num_attention_heads = 16 (MHA, no GQA) - - attention head_dim = 72 (16 x 72 = 1152) - - benchmarked head_dim = 80 (72 zero-padded to the next 16-byte - multiple: fp8 kernels require 16B-aligned head dims, and production - integrations run this padded contract. Reported TFLOPS count d=80.) - -Sequence lengths are per-image patch-grid token counts taken from a -production inference trace (native-resolution images; token count = -grid_h x grid_w). Single-image (batch 1) forwards dominate that trace, -and these six lengths span its per-forward FLOPs distribution from the -10th to the 99th percentile: - - ( 8836, 94 x 94 grid) - (15376, 124 x 124 grid) <- most frequent single-image forward - (24336, 156 x 156 grid) - (35344, 188 x 188 grid) <- FLOPs-median forward - (47376, non-square grid, e.g. 168 x 282) - (62500, 250 x 250 grid) - -Usage: - python -m benchmark.sdpa_benchmark_training.runner --config qwen3vl_vit - python -m benchmark.sdpa_benchmark_training.runner --config qwen3vl_vit --dry-run -""" - -from ..config_types import ModelPreset, BenchmarkConfig - -QWEN3VL_VIT = ModelPreset( - name="qwen3vl_vit", - num_q_heads=16, - num_kv_heads=16, - head_dim=80, -) - -CONFIG = BenchmarkConfig( - name="qwen3vl_vit", - models=[QWEN3VL_VIT], - seqlens=[ - (8836, 8836), # 94x94 patch grid - (15376, 15376), # 124x124 (most frequent) - (24336, 24336), # 156x156 - (35344, 35344), # 188x188 (FLOPs median) - (47376, 47376), # non-square grid (168x282) - (62500, 62500), # 250x250 - ], - backends=["cudnn", "cudnn_oss", "flash_attention_4"], - data_types=["bfloat16", "fp8"], - attn_masks=["no_mask"], - profile_pass="fwd", - batch_size=1, - num_iterations=10, - num_warmup_iterations=5, - output_dir="results", -) diff --git a/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/gb200/auto_regressive_dit_20260813_111215.csv b/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/gb200/auto_regressive_dit_20260813_111215.csv deleted file mode 100644 index 28f14d3df..000000000 --- a/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/gb200/auto_regressive_dit_20260813_111215.csv +++ /dev/null @@ -1,36 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,0.214,1319.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,985,62208,9,9,128,128,fwd,False,0.162,1739.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,985,62208,9,9,128,128,fwd,False,0.164,1721.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.214,1372.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.161,1829.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.165,1778.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.391,1503.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.300,1954.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.301,1948.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.721,1628.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.567,2072.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.586,2005.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.417,1658.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.122,2093.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.158,2028.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,0.731,386.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,985,62208,9,9,128,128,fwd,False,0.660,428.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,985,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.731,401.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.660,445.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.736,797.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.660,890.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.774,1517.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.663,1772.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.522,1543.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.313,1789.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,1.207,234.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,1.200,245.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,1.128,520.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,1.032,1138.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.997,1176.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 diff --git a/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/gb200/auto_regressive_dit_no_mask.png b/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/gb200/auto_regressive_dit_no_mask.png deleted file mode 100644 index 622214add..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/gb200/auto_regressive_dit_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/gb300/auto_regressive_dit_20260813_111532.csv b/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/gb300/auto_regressive_dit_20260813_111532.csv deleted file mode 100644 index 949761f38..000000000 --- a/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/gb300/auto_regressive_dit_20260813_111532.csv +++ /dev/null @@ -1,36 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,0.160,1762.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,985,62208,9,9,128,128,fwd,False,0.110,2556.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,985,62208,9,9,128,128,fwd,False,0.117,2405.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.161,1824.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.110,2663.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.118,2494.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.300,1957.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.209,2806.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.221,2656.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.573,2050.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.394,2979.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.426,2759.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.127,2084.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,0.780,3011.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,0.842,2790.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,0.603,468.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,985,62208,9,9,128,128,fwd,False,0.510,554.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,985,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.599,490.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.510,576.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.606,968.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.513,1144.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.630,1865.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.516,2277.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.250,1879.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.018,2306.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,0.522,541.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,0.521,563.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,0.529,1110.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,0.561,2094.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,1.109,2118.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 diff --git a/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/gb300/auto_regressive_dit_no_mask.png b/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/gb300/auto_regressive_dit_no_mask.png deleted file mode 100644 index 401da84ba..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/gb300/auto_regressive_dit_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_20260814_104551.csv b/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_20260814_104551.csv deleted file mode 100644 index 40a75c97c..000000000 --- a/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_20260814_104551.csv +++ /dev/null @@ -1,36 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,2.100,134.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,985,62208,9,9,128,128,fwd,False,1.300,217.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,880.500 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,985,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,2.096,140.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,1.302,225.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,1.961,299.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,1.324,443.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,924.000 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,3.720,316.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,454.500 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,2.469,476.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,7.333,320.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -auto_regressive_dit,auto_regressive_dit,cudnn,fp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,4.818,487.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -auto_regressive_dit,auto_regressive_dit,cudnn,mxfp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,1.053,268.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,985,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,985,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,1.055,278.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,1024,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,2.065,284.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,2048,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,3.509,335.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,4096,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,6.973,337.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -auto_regressive_dit,auto_regressive_dit,cudnn_oss,fp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -auto_regressive_dit,auto_regressive_dit,cudnn_oss,mxfp8,no_mask,1,8192,62208,9,9,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,985,62208,9,9,128,128,fwd,False,2.042,138.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,437.500 -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,1024,62208,9,9,128,128,fwd,False,2.045,144.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,437.500 -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,2048,62208,9,9,128,128,fwd,False,2.066,284.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,437.500 -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,4096,62208,9,9,128,128,fwd,False,4.102,286.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -auto_regressive_dit,auto_regressive_dit,flash_attention_4,bfloat16,no_mask,1,8192,62208,9,9,128,128,fwd,False,8.188,287.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 diff --git a/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_no_mask.png b/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_no_mask.png deleted file mode 100644 index 3e3009889..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/auto_regressive_dit/rtxpro6000/auto_regressive_dit_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/deepseek_v4/b300/deepseek_v4_20260812_173800.csv b/benchmark/sdpa_benchmark_training/results/deepseek_v4/b300/deepseek_v4_20260812_173800.csv deleted file mode 100644 index 33bd9516b..000000000 --- a/benchmark/sdpa_benchmark_training/results/deepseek_v4/b300/deepseek_v4_20260812_173800.csv +++ /dev/null @@ -1,593 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,error_message,gpu_name,cudnn_version,cudnn_backend_version -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,16384,16384,64,1,512,512,fwd,False,3.401,161.000,0.000,10,128,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,16384,16384,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 970, in - graph_bwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 when d_qk != d_v - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,16384,16384,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,16384,16384,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,16384,16384,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,16384,16384,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,16384,16384,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,16384,16384,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,8192,8192,64,1,512,512,fwd,False,1.644,166.000,0.000,10,128,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,8192,8192,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 970, in - graph_bwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 when d_qk != d_v - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,8192,8192,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,8192,8192,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,8192,8192,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,8192,8192,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,8192,8192,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,8192,8192,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,4096,4096,64,1,512,512,fwd,False,0.806,168.000,0.000,10,128,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,4096,4096,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 970, in - graph_bwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 when d_qk != d_v - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,4096,4096,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,4096,4096,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,4096,4096,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,4096,4096,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,4096,4096,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,4096,4096,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,2048,2048,64,1,512,512,fwd,False,0.400,167.000,0.000,10,128,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,2048,2048,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 970, in - graph_bwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 when d_qk != d_v - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,2048,2048,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,2048,2048,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,2048,2048,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,2048,2048,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,2048,2048,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,2048,2048,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,16384,16384,128,1,512,512,fwd,False,6.872,159.000,0.000,10,128,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,16384,16384,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 970, in - graph_bwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 when d_qk != d_v - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,16384,16384,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,16384,16384,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,16384,16384,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,16384,16384,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,16384,16384,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,16384,16384,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,8192,8192,128,1,512,512,fwd,False,3.316,164.000,0.000,10,128,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,8192,8192,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 970, in - graph_bwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 when d_qk != d_v - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,8192,8192,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,8192,8192,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,8192,8192,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,8192,8192,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,8192,8192,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,8192,8192,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,4096,4096,128,1,512,512,fwd,False,1.625,166.000,0.000,10,128,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,4096,4096,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 970, in - graph_bwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 when d_qk != d_v - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,4096,4096,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,4096,4096,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,4096,4096,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,4096,4096,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,4096,4096,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,4096,4096,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,2048,2048,128,1,512,512,fwd,False,0.793,168.000,0.000,10,128,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,2048,2048,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 970, in - graph_bwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 when d_qk != d_v - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,2048,2048,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,2048,2048,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,2048,2048,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,2048,2048,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 746, in - graph_fwd.validate() - File ""/usr/local/lib/python3.12/dist-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,2048,2048,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,2048,2048,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, diff --git a/benchmark/sdpa_benchmark_training/results/deepseek_v4/rtx_pro_6000/deepseek_v4_20260810_143917.csv b/benchmark/sdpa_benchmark_training/results/deepseek_v4/rtx_pro_6000/deepseek_v4_20260810_143917.csv deleted file mode 100644 index f15010237..000000000 --- a/benchmark/sdpa_benchmark_training/results/deepseek_v4/rtx_pro_6000/deepseek_v4_20260810_143917.csv +++ /dev/null @@ -1,593 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,error_message,gpu_name,cudnn_version,cudnn_backend_version -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,16384,16384,64,1,512,512,fwd,False,9.437,58.000,0.000,10,128,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,16384,16384,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,16384,16384,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,16384,16384,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,16384,16384,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,16384,16384,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,16384,16384,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,16384,16384,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,8192,8192,64,1,512,512,fwd,False,4.712,58.000,0.000,10,128,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,8192,8192,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,8192,8192,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,8192,8192,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,8192,8192,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,8192,8192,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,8192,8192,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,8192,8192,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,4096,4096,64,1,512,512,fwd,False,2.347,58.000,0.000,10,128,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,4096,4096,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,4096,4096,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,4096,4096,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,4096,4096,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,4096,4096,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,4096,4096,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,4096,4096,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,2048,2048,64,1,512,512,fwd,False,1.172,57.000,0.000,10,128,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -deepseek_v4,dsv4_flash,cudnn,bfloat16,top_left,2,2048,2048,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,2048,2048,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,fp8,top_left,2,2048,2048,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,2048,2048,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,cudnn,mxfp8,top_left,2,2048,2048,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,2048,2048,64,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_flash,flash_attention_4,bfloat16,top_left,2,2048,2048,64,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,16384,16384,128,1,512,512,fwd,False,18.853,58.000,0.000,10,128,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,16384,16384,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,16384,16384,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,16384,16384,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,16384,16384,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,16384,16384,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,16384,16384,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,16384,16384,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,8192,8192,128,1,512,512,fwd,False,9.394,58.000,0.000,10,128,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,8192,8192,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,8192,8192,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,8192,8192,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,8192,8192,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,8192,8192,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,8192,8192,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,8192,8192,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,4096,4096,128,1,512,512,fwd,False,4.653,58.000,0.000,10,128,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,4096,4096,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,4096,4096,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,4096,4096,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,4096,4096,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,4096,4096,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,4096,4096,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,4096,4096,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,2048,2048,128,1,512,512,fwd,False,2.304,58.000,0.000,10,128,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -deepseek_v4,dsv4_pro,cudnn,bfloat16,top_left,2,2048,2048,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,2048,2048,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,fp8,top_left,2,2048,2048,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,2048,2048,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,cudnn,mxfp8,top_left,2,2048,2048,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: hidden_dim d_qk should be less than or equal to 128 and hidden_dim d_qk should be multiple of 16 unless d_qk == 192 and d_v == 128 (requires cuDNN 9.19+) - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,2048,2048,128,1,512,512,fwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -deepseek_v4,dsv4_pro,flash_attention_4,bfloat16,top_left,2,2048,2048,128,1,512,512,bwd,False,inf,0.000,0.000,10,128,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, diff --git a/benchmark/sdpa_benchmark_training/results/deepseek_v4/rtx_pro_6000/deepseek_v4_top_left.png b/benchmark/sdpa_benchmark_training/results/deepseek_v4/rtx_pro_6000/deepseek_v4_top_left.png deleted file mode 100644 index d76eaf888..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/deepseek_v4/rtx_pro_6000/deepseek_v4_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_20260813_111327.csv b/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_20260813_111327.csv deleted file mode 100644 index a1e758d53..000000000 --- a/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_20260813_111327.csv +++ /dev/null @@ -1,211 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,52.064,1690.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,210.104,1089.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,True,210.020,1089.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,106.708,1649.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,424.053,1079.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,True,411.481,1112.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,fwd,False,36.193,2430.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,bwd,False,123.123,1858.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,bwd,True,122.379,1869.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,75.977,2315.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,238.075,1921.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,239.203,1912.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,fwd,False,39.191,2244.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,False,142.672,1603.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,True,143.499,1594.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,78.181,2250.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,287.702,1590.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,287.959,1588.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,13.439,1636.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,51.091,1119.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,True,48.713,1174.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,24.709,1780.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,104.440,1095.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,True,99.641,1148.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,fwd,False,8.590,2560.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,bwd,False,28.872,1980.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,bwd,True,28.891,1979.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,16.587,2651.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,54.612,2094.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,56.003,2042.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,fwd,False,9.213,2387.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,False,33.043,1730.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,True,33.564,1704.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,18.081,2432.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,64.035,1786.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,64.386,1776.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,3.420,1608.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,11.607,1232.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,12.340,1158.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,6.212,1770.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,23.838,1199.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,23.887,1197.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,fwd,False,2.262,2431.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,bwd,False,7.264,1968.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,bwd,True,7.237,1975.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,4.198,2619.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,13.352,2141.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,13.126,2178.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,fwd,False,2.423,2269.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,False,8.749,1634.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,True,9.093,1572.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,4.416,2490.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,15.923,1795.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,15.980,1789.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,1.000,1375.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,3.325,1075.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,3.239,1103.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.592,1727.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,5.887,1214.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,5.960,1199.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,fwd,False,0.658,2090.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,bwd,False,2.085,1714.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,bwd,True,2.083,1716.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.093,2516.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,3.447,2073.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,3.442,2077.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,fwd,False,0.694,1982.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,False,2.532,1411.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,True,2.536,1410.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.172,2345.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,4.130,1730.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,4.122,1734.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,0.317,1086.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,1.043,857.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,1.002,892.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.460,1495.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,1.692,1056.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,1.622,1102.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,fwd,False,0.205,1679.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,bwd,False,0.671,1331.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,bwd,True,0.672,1330.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.312,2202.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,0.988,1809.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,0.986,1813.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,fwd,False,0.218,1576.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,False,0.841,1062.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,True,0.840,1064.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.330,2081.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,1.182,1511.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,1.182,1512.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,61.154,1438.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,129.001,1364.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,13.849,1588.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,32.224,1365.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,3.695,1488.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,6.852,1605.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,1.037,1326.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.721,1597.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,0.333,1033.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.488,1407.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,51.239,1717.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,203.575,1123.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,True,214.041,1069.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,112.896,1558.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,394.633,1159.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,True,402.368,1137.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,14.004,1570.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,51.913,1101.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,True,52.256,1094.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,23.694,1856.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,104.159,1098.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,True,105.362,1085.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,3.329,1651.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,12.736,1122.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,13.905,1028.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,5.872,1873.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,25.752,1110.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,25.071,1140.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,0.965,1425.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,3.636,983.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,3.834,932.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.515,1814.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,6.164,1160.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,6.302,1134.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,0.306,1122.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,1.171,763.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,1.236,723.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.444,1546.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,1.753,1019.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,1.800,992.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 diff --git a/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_no_mask.png b/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_no_mask.png deleted file mode 100644 index ef5cd0d03..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_no_mask_det_overhead.png b/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_no_mask_det_overhead.png deleted file mode 100644 index 5415372b0..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_no_mask_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_top_left.png b/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_top_left.png deleted file mode 100644 index 1af413508..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_top_left_det_overhead.png b/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_top_left_det_overhead.png deleted file mode 100644 index 1afde4339..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_top_left_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_20260813_111623.csv b/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_20260813_111623.csv deleted file mode 100644 index b5d4574fa..000000000 --- a/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_20260813_111623.csv +++ /dev/null @@ -1,211 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,43.211,2036.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,178.125,1284.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,True,176.417,1296.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,89.086,1975.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,365.130,1253.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,True,353.709,1293.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,fwd,False,26.137,3365.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,bwd,False,95.400,2397.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,bwd,True,96.010,2382.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,51.490,3417.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,187.987,2433.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,188.221,2430.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,fwd,False,29.365,2995.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,False,118.144,1936.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,True,118.114,1936.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,58.925,2986.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,231.814,1973.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,230.744,1982.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,10.836,2029.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,42.972,1331.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,True,41.684,1372.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,21.062,2088.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,89.608,1276.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,True,85.862,1332.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,fwd,False,6.544,3361.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,bwd,False,22.852,2502.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,bwd,True,23.111,2474.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,12.685,3467.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,43.914,2604.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,44.602,2564.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,fwd,False,7.283,3020.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,False,28.892,1979.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,True,28.997,1972.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,14.125,3114.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,56.048,2040.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,56.079,2039.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,2.883,1907.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,11.033,1296.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,10.667,1340.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,5.250,2094.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,21.499,1330.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,20.997,1361.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,fwd,False,1.716,3204.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,bwd,False,6.120,2336.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,bwd,True,6.114,2338.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,3.180,3458.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,11.067,2583.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,11.372,2514.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,fwd,False,1.916,2870.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,False,7.777,1838.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,True,7.783,1837.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,3.541,3105.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,14.037,2037.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,13.931,2052.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,0.819,1678.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,3.108,1150.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,2.987,1197.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.353,2031.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,5.627,1270.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,5.603,1275.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,fwd,False,0.479,2867.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,bwd,False,1.786,2001.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,bwd,True,1.787,2000.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,0.816,3369.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,2.934,2436.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,2.936,2434.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,fwd,False,0.535,2570.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,False,2.310,1548.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,True,2.309,1548.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,0.908,3028.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,3.677,1943.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,3.678,1943.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,0.254,1351.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,0.975,917.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,0.932,959.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.371,1850.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,1.596,1120.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,1.558,1147.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,fwd,False,0.148,2324.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,bwd,False,0.591,1514.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,bwd,True,0.592,1510.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.222,3092.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,0.848,2107.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,0.851,2101.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,fwd,False,0.163,2109.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,False,0.780,1147.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,True,0.780,1146.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.249,2759.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,1.070,1670.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,1.069,1671.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,50.720,1734.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,105.179,1673.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,12.529,1755.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,24.170,1820.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,3.203,1716.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,5.937,1852.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,0.889,1547.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.509,1821.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,0.273,1260.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.402,1710.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,46.128,1907.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,184.078,1242.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,True,196.553,1164.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,94.441,1863.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,349.151,1310.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,True,369.698,1237.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,11.616,1893.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,45.849,1247.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,True,48.123,1188.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,21.495,2046.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,88.427,1293.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,True,93.445,1224.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,3.091,1779.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,12.140,1178.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,12.866,1111.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,5.482,2006.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,21.625,1322.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,23.264,1229.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,0.875,1570.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,3.512,1018.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,3.720,961.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,1.392,1975.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,5.773,1238.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,6.153,1161.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,0.276,1248.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,1.141,783.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,1.206,741.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,0.386,1780.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,1.679,1064.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,1.771,1009.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 diff --git a/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_no_mask.png b/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_no_mask.png deleted file mode 100644 index f48e0d4b2..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_no_mask_det_overhead.png b/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_no_mask_det_overhead.png deleted file mode 100644 index 601f9df31..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_no_mask_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_top_left.png b/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_top_left.png deleted file mode 100644 index b9cd11df5..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_top_left_det_overhead.png b/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_top_left_det_overhead.png deleted file mode 100644 index 80d42889f..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_top_left_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/dsv3/rtxpro6000/dsv3_20260814_104551.csv b/benchmark/sdpa_benchmark_training/results/dsv3/rtxpro6000/dsv3_20260814_104551.csv deleted file mode 100644 index 84cb55aa6..000000000 --- a/benchmark/sdpa_benchmark_training/results/dsv3/rtxpro6000/dsv3_20260814_104551.csv +++ /dev/null @@ -1,211 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,377.898,233.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,465.000 -dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,759.932,231.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,94.885,232.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,189.787,232.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,24.106,228.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,47.558,231.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,6.242,220.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,12.045,228.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,1.680,205.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,3.099,222.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,457.500 -dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,top_left,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,216.928,405.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,428.035,411.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,32768,32768,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,54.868,401.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,107.385,410.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,16384,16384,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,13.363,411.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,26.520,415.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,8192,8192,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,3.576,384.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,6.578,418.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,4096,4096,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,1.037,332.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,1.778,386.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,top_left,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,fp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,2048,2048,128,128,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,cudnn_oss,mxfp8,no_mask,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,fwd,False,249.369,353.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,False,1428.003,160.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,fwd,False,465.772,378.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,False,2883.479,159.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,32768,32768,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,fwd,False,63.695,345.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,False,362.201,158.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,fwd,False,116.811,377.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,False,723.904,158.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,16384,16384,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,fwd,False,15.696,350.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,False,93.107,154.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,fwd,False,28.912,380.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,False,183.122,156.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,8192,8192,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,fwd,False,4.151,331.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,False,24.509,146.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,fwd,False,7.212,381.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,437.500 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,False,46.886,152.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,34.500 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,4096,4096,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,fwd,False,1.170,294.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,34.500 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,False,6.799,131.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,34.500 -dsv3,dsv3,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,fwd,False,2.023,340.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,34.500 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,False,12.310,145.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,34.500 -dsv3,dsv3,flash_attention_4,bfloat16,no_mask,2,2048,2048,128,128,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, diff --git a/benchmark/sdpa_benchmark_training/results/dsv3/rtxpro6000/dsv3_no_mask.png b/benchmark/sdpa_benchmark_training/results/dsv3/rtxpro6000/dsv3_no_mask.png deleted file mode 100644 index c3857db94..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/dsv3/rtxpro6000/dsv3_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/dsv3/rtxpro6000/dsv3_top_left.png b/benchmark/sdpa_benchmark_training/results/dsv3/rtxpro6000/dsv3_top_left.png deleted file mode 100644 index 8ff1eb3a3..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/dsv3/rtxpro6000/dsv3_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/gpt_oss/gb200/gpt_oss_20260813_110803.csv b/benchmark/sdpa_benchmark_training/results/gpt_oss/gb200/gpt_oss_20260813_110803.csv deleted file mode 100644 index c7ca2f1c4..000000000 --- a/benchmark/sdpa_benchmark_training/results/gpt_oss/gb200/gpt_oss_20260813_110803.csv +++ /dev/null @@ -1,106 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,1.611,170.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,3.626,189.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,4.723,145.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,fwd,False,1.509,182.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,bwd,False,91.732,7.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,bwd,True,91.721,7.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,fwd,False,1.561,176.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5043.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,False,107.525,6.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,True,107.527,6.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,0.809,169.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,1.818,188.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,2.372,144.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,fwd,False,0.758,181.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,bwd,False,23.620,14.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,bwd,True,23.616,14.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,fwd,False,0.783,175.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,False,27.692,12.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,True,27.696,12.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.408,167.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,0.899,190.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,1.193,143.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,fwd,False,0.381,179.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,bwd,False,6.287,27.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,bwd,True,6.274,27.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,fwd,False,0.395,172.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,False,7.381,23.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,True,7.379,23.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.207,163.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,0.458,185.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,0.608,139.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,fwd,False,0.193,175.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,bwd,False,1.771,48.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,bwd,True,1.770,48.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,fwd,False,0.201,169.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,False,2.070,41.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,True,2.071,41.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.107,156.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.242,172.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,0.313,133.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,fwd,False,0.100,167.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,bwd,False,0.548,76.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,bwd,True,0.548,76.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,fwd,False,0.104,160.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,False,0.657,63.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,True,0.642,65.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,3.624,76.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,32768,32768,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,32768,32768,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,32768,32768,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,1.810,76.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,16384,16384,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,16384,16384,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,16384,16384,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.906,75.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,8192,8192,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,8192,8192,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,8192,8192,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.454,74.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,4096,4096,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,4096,4096,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,4096,4096,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.228,73.000,0.000,10,128,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,2048,2048,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,2048,2048,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,2048,2048,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,1.818,151.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,4.662,147.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,28.467,24.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,0.912,150.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,2.315,148.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,7.871,43.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.459,149.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,1.154,148.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,2.318,74.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.232,146.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,0.578,146.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,0.759,111.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.120,139.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.292,143.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,0.323,129.000,0.000,10,128,True,False,,NVIDIA GB200,,,2568.000 diff --git a/benchmark/sdpa_benchmark_training/results/gpt_oss/gb200/gpt_oss_top_left.png b/benchmark/sdpa_benchmark_training/results/gpt_oss/gb200/gpt_oss_top_left.png deleted file mode 100644 index 3799e9629..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/gpt_oss/gb200/gpt_oss_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/gpt_oss/gb200/gpt_oss_top_left_det_overhead.png b/benchmark/sdpa_benchmark_training/results/gpt_oss/gb200/gpt_oss_top_left_det_overhead.png deleted file mode 100644 index adf79bba1..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/gpt_oss/gb200/gpt_oss_top_left_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/gpt_oss/gb300/gpt_oss_20260813_111114.csv b/benchmark/sdpa_benchmark_training/results/gpt_oss/gb300/gpt_oss_20260813_111114.csv deleted file mode 100644 index c4b1ef77a..000000000 --- a/benchmark/sdpa_benchmark_training/results/gpt_oss/gb300/gpt_oss_20260813_111114.csv +++ /dev/null @@ -1,106 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,1.426,192.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,3.440,199.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,4.335,158.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,fwd,False,1.266,217.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,bwd,False,75.013,9.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,bwd,True,74.983,9.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,fwd,False,1.301,211.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,False,94.959,7.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,True,95.014,7.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,0.717,191.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,1.726,198.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,2.179,157.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,fwd,False,0.636,215.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,bwd,False,19.413,18.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,bwd,True,19.413,18.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,fwd,False,0.655,209.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,False,24.482,14.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,True,24.483,14.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.362,188.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,0.863,197.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,1.098,155.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,fwd,False,0.321,213.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,bwd,False,5.210,33.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,bwd,True,5.208,33.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,fwd,False,0.331,206.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,False,6.530,26.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,True,6.531,26.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.184,183.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,0.443,191.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,0.563,150.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,fwd,False,0.163,208.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,bwd,False,1.491,57.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,bwd,True,1.490,57.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,fwd,False,0.169,200.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,False,1.848,46.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,True,1.849,46.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.095,174.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.228,183.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,0.290,143.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,fwd,False,0.085,197.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,bwd,False,0.471,88.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,bwd,True,0.471,88.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,fwd,False,0.088,189.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,False,0.577,72.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,True,0.577,72.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,3.610,76.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,32768,32768,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,32768,32768,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,32768,32768,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,1.805,76.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,16384,16384,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,16384,16384,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,16384,16384,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.903,76.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,8192,8192,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,8192,8192,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,8192,8192,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.452,75.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,4096,4096,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,4096,4096,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,4096,4096,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.227,73.000,0.000,10,128,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,2048,2048,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,2048,2048,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,2048,2048,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,1.683,163.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,4.357,157.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,27.854,25.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,0.845,162.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,2.175,157.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,7.680,45.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.425,160.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,1.088,157.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,2.258,75.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.215,157.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,0.549,154.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,0.738,115.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.112,149.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.278,150.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,0.311,134.000,0.000,10,128,True,False,,NVIDIA GB300,,,2578.000 diff --git a/benchmark/sdpa_benchmark_training/results/gpt_oss/gb300/gpt_oss_top_left.png b/benchmark/sdpa_benchmark_training/results/gpt_oss/gb300/gpt_oss_top_left.png deleted file mode 100644 index db5d35a85..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/gpt_oss/gb300/gpt_oss_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/gpt_oss/gb300/gpt_oss_top_left_det_overhead.png b/benchmark/sdpa_benchmark_training/results/gpt_oss/gb300/gpt_oss_top_left_det_overhead.png deleted file mode 100644 index f62648706..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/gpt_oss/gb300/gpt_oss_top_left_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/gpt_oss/rtxpro6000/gpt_oss_20260814_104551.csv b/benchmark/sdpa_benchmark_training/results/gpt_oss/rtxpro6000/gpt_oss_20260814_104551.csv deleted file mode 100644 index b1fb69511..000000000 --- a/benchmark/sdpa_benchmark_training/results/gpt_oss/rtxpro6000/gpt_oss_20260814_104551.csv +++ /dev/null @@ -1,106 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,2.962,93.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,14.303,48.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,453.500 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,fwd,False,2.884,95.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,32768,32768,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,1.482,92.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,7.085,48.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,fwd,False,1.447,95.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,16384,16384,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.762,90.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,3.525,48.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,31.285,5.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,fwd,False,0.732,93.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,8192,8192,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.398,85.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,1.760,48.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,8.342,10.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,fwd,False,0.379,89.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,4096,4096,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.211,79.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.885,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,2.380,17.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,fwd,False,0.207,80.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,fp8,top_left,2,2048,2048,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,2.990,92.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,14.645,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,14.653,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,32768,32768,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,32768,32768,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,32768,32768,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,32768,32768,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,1.502,91.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,465.000 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,7.230,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,7.247,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,16384,16384,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,16384,16384,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,16384,16384,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,16384,16384,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,0.762,89.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,3.595,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,3.606,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,8192,8192,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,8192,8192,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,8192,8192,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,8192,8192,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,0.398,85.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,1.794,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,1.813,47.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,4096,4096,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,4096,4096,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,4096,4096,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,4096,4096,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.213,78.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.899,46.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn_oss,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,0.905,46.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,2048,2048,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,2048,2048,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,fp8,top_left,2,2048,2048,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,64,64,fwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,False,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,cudnn_oss,mxfp8,top_left,2,2048,2048,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,fwd,False,108.666,3.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,437.500 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,False,12.970,53.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,446.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,32768,32768,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,fwd,False,27.307,5.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,False,6.430,53.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,16384,16384,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,fwd,False,7.018,10.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,False,3.195,53.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,8192,8192,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,fwd,False,1.958,17.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,False,1.587,53.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,4096,4096,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,fwd,False,0.564,30.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,False,0.798,52.000,0.000,10,128,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -gpt_oss,gpt_oss,flash_attention_4,bfloat16,top_left,2,2048,2048,128,128,64,64,bwd,True,inf,0.000,0.000,10,128,False,False,Benchmark failed with return code 1.,,,, diff --git a/benchmark/sdpa_benchmark_training/results/gpt_oss/rtxpro6000/gpt_oss_top_left.png b/benchmark/sdpa_benchmark_training/results/gpt_oss/rtxpro6000/gpt_oss_top_left.png deleted file mode 100644 index 1da474042..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/gpt_oss/rtxpro6000/gpt_oss_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/gpt_oss/rtxpro6000/gpt_oss_top_left_det_overhead.png b/benchmark/sdpa_benchmark_training/results/gpt_oss/rtxpro6000/gpt_oss_top_left_det_overhead.png deleted file mode 100644 index 4a47e2abb..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/gpt_oss/rtxpro6000/gpt_oss_top_left_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_20260813_111243.csv b/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_20260813_111243.csv deleted file mode 100644 index 3e793d369..000000000 --- a/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_20260813_111243.csv +++ /dev/null @@ -1,211 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,25.043,1756.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,99.920,1144.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,True,101.371,1128.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,52.640,1671.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,213.760,1070.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,True,199.775,1145.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,fwd,False,16.409,2680.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,bwd,False,57.627,1984.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,bwd,True,58.925,1941.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,35.542,2475.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,114.544,1997.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,116.931,1956.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,fwd,False,19.001,2315.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,False,70.139,1630.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,True,65.649,1742.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,36.902,2384.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,136.710,1673.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,135.948,1682.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,6.306,1744.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,21.672,1319.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,21.626,1322.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,12.474,1763.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,49.498,1155.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,46.283,1235.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,fwd,False,4.164,2641.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,bwd,False,13.390,2135.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,bwd,True,13.938,2051.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,8.683,2533.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,24.767,2308.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,25.971,2201.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,fwd,False,4.490,2449.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,False,15.924,1795.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,True,15.649,1827.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,8.820,2493.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,31.964,1789.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,31.437,1819.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,1.635,1681.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,5.890,1213.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,5.670,1261.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,2.981,1844.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,12.331,1159.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,11.142,1283.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,fwd,False,1.119,2456.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,bwd,False,3.591,1991.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,bwd,True,3.593,1990.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,2.090,2631.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,6.571,2175.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,6.372,2243.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,fwd,False,1.207,2277.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,False,4.238,1687.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,True,4.240,1686.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,2.201,2498.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,7.604,1880.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,7.886,1812.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,0.496,1386.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,1.629,1097.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,1.605,1113.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.815,1687.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,2.946,1213.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,2.897,1234.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,fwd,False,0.328,2098.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,bwd,False,1.058,1689.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,bwd,True,1.055,1694.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.565,2433.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,1.723,2073.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,1.689,2116.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,fwd,False,0.348,1976.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,False,1.282,1394.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,True,1.283,1393.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.591,2326.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,2.064,1731.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,2.046,1746.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.163,1055.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,0.520,860.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,0.513,872.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.231,1486.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.860,1039.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.840,1064.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,fwd,False,0.106,1620.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,bwd,False,0.350,1279.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,bwd,True,0.349,1279.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.160,2150.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.512,1746.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.512,1745.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,fwd,False,0.113,1518.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,False,0.435,1028.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,523.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,True,0.435,1027.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.169,2039.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.618,1447.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.619,1444.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,27.726,1586.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,58.852,1495.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,6.637,1657.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,13.374,1644.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,1.802,1525.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,3.257,1688.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,0.515,1334.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.868,1584.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.171,1002.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.241,1423.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,25.970,1694.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,101.246,1129.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,True,106.933,1069.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,50.734,1734.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,201.998,1132.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,True,206.275,1109.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,6.257,1757.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,24.851,1150.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,25.573,1118.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,12.478,1762.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,48.987,1167.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,50.045,1142.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,1.679,1638.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,6.306,1133.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,6.668,1072.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,2.936,1873.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,12.197,1172.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,12.441,1149.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,0.497,1384.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,1.848,967.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,1.953,915.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.819,1678.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,3.085,1158.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,3.169,1128.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.163,1053.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,0.609,734.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,0.644,694.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.234,1467.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.913,978.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.938,952.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 diff --git a/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_no_mask.png b/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_no_mask.png deleted file mode 100644 index 9b95017e8..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_no_mask_det_overhead.png b/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_no_mask_det_overhead.png deleted file mode 100644 index fe31f2cc2..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_no_mask_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_top_left.png b/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_top_left.png deleted file mode 100644 index bcb2cdd1f..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_top_left_det_overhead.png b/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_top_left_det_overhead.png deleted file mode 100644 index eb56c5903..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/kimiK26/gb200/kimiK26_top_left_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_20260813_111539.csv b/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_20260813_111539.csv deleted file mode 100644 index 8c1b8f451..000000000 --- a/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_20260813_111539.csv +++ /dev/null @@ -1,211 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,21.258,2069.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,88.230,1296.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,True,85.338,1340.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,42.102,2089.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,182.603,1252.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,True,173.766,1316.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,fwd,False,12.728,3455.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,bwd,False,44.328,2580.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,bwd,True,45.123,2534.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,25.272,3481.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,90.030,2540.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,89.650,2551.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,fwd,False,14.173,3103.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,False,56.546,2022.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,True,55.779,2050.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,28.303,3108.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,111.480,2051.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,112.626,2031.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,5.325,2065.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,21.036,1359.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,20.704,1381.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,10.171,2162.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,42.858,1334.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,40.472,1413.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,fwd,False,3.235,3399.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,bwd,False,11.381,2512.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,bwd,True,11.250,2541.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,6.234,3527.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,21.622,2644.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,22.343,2559.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,fwd,False,3.617,3040.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,False,14.563,1963.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,True,14.229,2009.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,6.943,3167.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,27.352,2090.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,26.843,2130.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,1.429,1923.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,5.470,1307.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,5.287,1352.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,2.581,2130.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,10.536,1357.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,10.282,1390.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,fwd,False,0.853,3222.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,bwd,False,3.058,2338.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,bwd,True,3.058,2337.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,1.572,3498.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,5.457,2619.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,5.461,2618.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,fwd,False,0.955,2878.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,False,3.891,1837.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,True,3.889,1838.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,1.756,3130.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,6.834,2092.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,6.833,2092.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,0.411,1674.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,1.554,1150.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,1.506,1186.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.685,2005.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,2.842,1257.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,2.818,1268.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,fwd,False,0.242,2844.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,bwd,False,0.903,1978.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,bwd,True,0.904,1976.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.416,3304.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,1.466,2437.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,1.470,2431.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,fwd,False,0.268,2561.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,False,1.174,1522.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,True,1.175,1521.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.469,2934.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,1.848,1933.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,1.845,1937.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.132,1299.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,0.496,901.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,0.478,935.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.189,1814.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.839,1065.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.809,1105.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,fwd,False,0.078,2202.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,bwd,False,0.304,1469.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,bwd,True,0.305,1467.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.115,2997.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.443,2015.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.443,2019.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,fwd,False,0.086,2005.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,False,0.402,1112.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,True,0.402,1113.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.130,2651.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.558,1600.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.558,1600.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,24.812,1773.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,50.447,1744.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,6.102,1802.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,11.781,1867.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,1.590,1729.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,2.966,1853.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,0.443,1550.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.766,1794.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.141,1219.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.204,1683.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,22.841,1926.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,90.015,1270.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,True,95.986,1191.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,43.048,2043.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,175.377,1304.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,True,188.639,1212.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,5.755,1911.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,22.624,1264.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,24.020,1190.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,10.328,2129.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,43.029,1329.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,45.471,1257.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,1.539,1786.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,6.088,1174.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,6.486,1102.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,2.635,2086.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,10.680,1338.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,11.494,1244.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,0.440,1563.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,1.795,996.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,1.901,940.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,0.716,1921.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,2.918,1225.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,3.101,1152.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.144,1196.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,0.592,755.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,0.625,715.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.197,1745.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,0.869,1028.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,0.918,973.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 diff --git a/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_no_mask.png b/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_no_mask.png deleted file mode 100644 index f147a06bb..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_no_mask_det_overhead.png b/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_no_mask_det_overhead.png deleted file mode 100644 index 05b605d34..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_no_mask_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_top_left.png b/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_top_left.png deleted file mode 100644 index cc7c09e13..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_top_left_det_overhead.png b/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_top_left_det_overhead.png deleted file mode 100644 index 84fb8f8bd..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/kimiK26/gb300/kimiK26_top_left_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/kimiK26/rtxpro6000/kimiK26_20260814_104551.csv b/benchmark/sdpa_benchmark_training/results/kimiK26/rtxpro6000/kimiK26_20260814_104551.csv deleted file mode 100644 index 8718d3b0b..000000000 --- a/benchmark/sdpa_benchmark_training/results/kimiK26/rtxpro6000/kimiK26_20260814_104551.csv +++ /dev/null @@ -1,211 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,188.829,233.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,465.000 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,379.283,232.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,47.716,230.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,94.710,232.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,12.208,225.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,23.907,230.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,3.206,214.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,6.119,225.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.888,194.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,1.583,217.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,top_left,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,108.532,405.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,447.500 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,213.757,412.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,27.449,401.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,53.618,410.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,6.815,403.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,12.891,426.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,1.862,369.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,3.345,411.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.569,302.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,442.000 -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,0.918,374.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,top_left,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,fp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,top_left,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,64,192,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,fwd,False,121.056,363.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,False,714.133,160.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,fwd,False,226.718,388.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,False,1441.186,159.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,fwd,False,30.777,357.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,False,181.089,158.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,fwd,False,56.848,387.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,False,362.741,158.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,440.500 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,fwd,False,7.975,345.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,False,46.621,153.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,fwd,False,14.180,388.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,False,91.948,155.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,450.500 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,fwd,False,2.147,320.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,False,12.369,144.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,459.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,fwd,False,3.635,378.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,False,23.571,152.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,fwd,False,0.653,263.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,False,3.620,123.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -kimiK26,kimiK26,flash_attention_4,bfloat16,top_left,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,fwd,False,1.015,338.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,34.500 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,False,6.217,144.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,34.500 -kimiK26,kimiK26,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,64,192,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, diff --git a/benchmark/sdpa_benchmark_training/results/kimiK26/rtxpro6000/kimiK26_no_mask.png b/benchmark/sdpa_benchmark_training/results/kimiK26/rtxpro6000/kimiK26_no_mask.png deleted file mode 100644 index 87e3340cf..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/kimiK26/rtxpro6000/kimiK26_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/kimiK26/rtxpro6000/kimiK26_top_left.png b/benchmark/sdpa_benchmark_training/results/kimiK26/rtxpro6000/kimiK26_top_left.png deleted file mode 100644 index 47bd12ab2..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/kimiK26/rtxpro6000/kimiK26_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/kimi_k3/b300/kimi_k3_20260812_175226.csv b/benchmark/sdpa_benchmark_training/results/kimi_k3/b300/kimi_k3_20260812_175226.csv deleted file mode 100644 index 4bbf0ed4c..000000000 --- a/benchmark/sdpa_benchmark_training/results/kimi_k3/b300/kimi_k3_20260812_175226.csv +++ /dev/null @@ -1,511 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,error_message,gpu_name,cudnn_version,cudnn_backend_version -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,fwd,False,38.795,1701.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,False,162.976,1052.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,True,157.627,1088.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,fwd,False,80.701,1635.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,False,325.734,1053.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,True,314.510,1091.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,32768,32768,96,96,192,128,fwd,False,23.906,2760.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,32768,32768,96,96,192,128,bwd,False,83.333,2058.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,32768,32768,96,96,192,128,bwd,True,83.955,2043.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,32768,32768,96,96,192,128,fwd,False,47.477,2779.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,32768,32768,96,96,192,128,bwd,False,168.552,2035.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,32768,32768,96,96,192,128,bwd,True,168.149,2040.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,32768,32768,96,96,192,128,fwd,False,24.417,2702.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,32768,32768,96,96,192,128,bwd,False,106.552,1610.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,32768,32768,96,96,192,128,bwd,True,107.064,1602.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,32768,32768,96,96,192,128,fwd,False,51.318,2571.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,32768,32768,96,96,192,128,bwd,False,206.118,1664.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,32768,32768,96,96,192,128,bwd,True,207.268,1655.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,fwd,False,8.975,1838.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,False,37.853,1133.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,True,39.098,1097.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,fwd,False,17.097,1929.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,False,78.072,1099.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,True,75.159,1141.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,16384,16384,96,96,192,128,fwd,False,5.395,3057.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,16384,16384,96,96,192,128,bwd,False,20.704,2071.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,16384,16384,96,96,192,128,bwd,True,20.321,2110.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,16384,16384,96,96,192,128,fwd,False,10.656,3095.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,16384,16384,96,96,192,128,bwd,False,41.492,2067.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,16384,16384,96,96,192,128,bwd,True,40.905,2097.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,16384,16384,96,96,192,128,fwd,False,5.916,2788.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,16384,16384,96,96,192,128,bwd,False,25.190,1702.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,16384,16384,96,96,192,128,bwd,True,25.313,1694.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,16384,16384,96,96,192,128,fwd,False,11.246,2933.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,16384,16384,96,96,192,128,bwd,False,51.280,1672.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,16384,16384,96,96,192,128,bwd,True,51.675,1660.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,fwd,False,2.379,1734.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,False,8.949,1198.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,True,8.862,1210.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,fwd,False,4.307,1914.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,False,18.493,1159.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,True,18.011,1190.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,8192,8192,96,96,192,128,fwd,False,1.411,2922.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,8192,8192,96,96,192,128,bwd,False,4.981,2152.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,8192,8192,96,96,192,128,bwd,True,4.979,2154.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,8192,8192,96,96,192,128,fwd,False,2.623,3143.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,8192,8192,96,96,192,128,bwd,False,9.217,2326.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,8192,8192,96,96,192,128,bwd,True,9.100,2356.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,8192,8192,96,96,192,128,fwd,False,1.556,2651.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,8192,8192,96,96,192,128,bwd,False,6.342,1691.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,8192,8192,96,96,192,128,bwd,True,6.341,1691.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,8192,8192,96,96,192,128,fwd,False,2.822,2922.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,8192,8192,96,96,192,128,bwd,False,11.314,1895.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,8192,8192,96,96,192,128,bwd,True,11.311,1895.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,fwd,False,0.654,1576.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,False,2.505,1070.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,True,2.411,1112.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,fwd,False,1.099,1876.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,False,4.486,1195.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,True,4.406,1217.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,4096,4096,96,96,192,128,fwd,False,0.389,2653.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,4096,4096,96,96,192,128,bwd,False,1.447,1853.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,4096,4096,96,96,192,128,bwd,True,1.447,1852.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,4096,4096,96,96,192,128,fwd,False,0.649,3177.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,4096,4096,96,96,192,128,bwd,False,2.404,2230.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,4096,4096,96,96,192,128,bwd,True,2.402,2231.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,4096,4096,96,96,192,128,fwd,False,0.433,2381.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,4096,4096,96,96,192,128,bwd,False,1.877,1428.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,4096,4096,96,96,192,128,bwd,True,1.877,1428.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,4096,4096,96,96,192,128,fwd,False,0.709,2909.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,4096,4096,96,96,192,128,bwd,False,2.973,1803.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,4096,4096,96,96,192,128,bwd,True,2.972,1804.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,fwd,False,0.207,1246.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,False,0.770,870.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,True,0.738,908.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,fwd,False,0.297,1738.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,False,1.254,1068.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,True,1.203,1113.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,2048,2048,96,96,192,128,fwd,False,0.123,2104.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,2048,2048,96,96,192,128,bwd,False,0.475,1411.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,2048,2048,96,96,192,128,bwd,True,0.475,1411.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,2048,2048,96,96,192,128,fwd,False,0.179,2872.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,2048,2048,96,96,192,128,bwd,False,0.682,1965.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,2048,2048,96,96,192,128,bwd,True,0.682,1965.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,2048,2048,96,96,192,128,fwd,False,0.137,1885.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,2048,2048,96,96,192,128,bwd,False,0.631,1063.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,2048,2048,96,96,192,128,bwd,True,0.630,1065.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,2048,2048,96,96,192,128,fwd,False,0.198,2598.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,2048,2048,96,96,192,128,bwd,False,0.855,1568.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,2048,2048,96,96,192,128,bwd,True,0.854,1569.000,0.000,10,,True,,NVIDIA B300 SXM6 AC,1.27.0,92400.000 -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/home/vagarwalla/attn_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1194, in - import flash_attn.cute.interface as flash_attn_interface - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/__init__.py"", line 12, in - from .interface import ( - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/interface.py"", line 45, in - from flash_attn.cute import utils - File ""/usr/local/lib/python3.12/dist-packages/flash_attn/cute/utils.py"", line 167, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, diff --git a/benchmark/sdpa_benchmark_training/results/kimi_k3/rtx_pro_6000/kimi_k3_20260810_144319.csv b/benchmark/sdpa_benchmark_training/results/kimi_k3/rtx_pro_6000/kimi_k3_20260810_144319.csv deleted file mode 100644 index 0bbe1b6df..000000000 --- a/benchmark/sdpa_benchmark_training/results/kimi_k3/rtx_pro_6000/kimi_k3_20260810_144319.csv +++ /dev/null @@ -1,3011 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,error_message,gpu_name,cudnn_version,cudnn_backend_version -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,fwd,False,291.756,226.000,0.000,10,,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,fwd,False,575.196,229.000,0.000,10,,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,32768,32768,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,32768,32768,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,32768,32768,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,32768,32768,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,32768,32768,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,32768,32768,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,32768,32768,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,32768,32768,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,32768,32768,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,32768,32768,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,32768,32768,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,32768,32768,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,32768,32768,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,32768,32768,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,fwd,False,73.879,223.000,0.000,10,,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,fwd,False,144.025,229.000,0.000,10,,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,16384,16384,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,16384,16384,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,16384,16384,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,16384,16384,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,16384,16384,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,16384,16384,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,16384,16384,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,16384,16384,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,16384,16384,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,16384,16384,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,16384,16384,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,16384,16384,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,16384,16384,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,16384,16384,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,fwd,False,18.148,227.000,0.000,10,,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,fwd,False,35.705,231.000,0.000,10,,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,8192,8192,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,8192,8192,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,8192,8192,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,8192,8192,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,8192,8192,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,8192,8192,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,8192,8192,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,8192,8192,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,8192,8192,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,8192,8192,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,8192,8192,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,8192,8192,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,8192,8192,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,8192,8192,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,fwd,False,4.722,218.000,0.000,10,,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,fwd,False,18.417,112.000,0.000,10,,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,4096,4096,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,4096,4096,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,4096,4096,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,4096,4096,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,4096,4096,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,4096,4096,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,4096,4096,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,4096,4096,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,4096,4096,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,4096,4096,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,4096,4096,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,4096,4096,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,4096,4096,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,4096,4096,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,fwd,False,1.273,203.000,0.000,10,,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,fwd,False,2.289,225.000,0.000,10,,True,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92400.000 -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 971, in - graph_bwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: Num hidden_dim should be less than or equal to 128 and hidden_dim should be multiple of 8 - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,2048,2048,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,2048,2048,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,top_left,2,2048,2048,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for ______________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,2048,2048,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,2048,2048,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,fp8,no_mask,2,2048,2048,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 749, in - graph_fwd.create_execution_plans([cudnn.heur_mode.A, cudnn.heur_mode.FALLBACK]) - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 882, in create_execution_plans - raise cudnn_graph_not_supported(f""no engine — python or backend — proposed a plan for this graph{why}"") -cudnn._compiled_module.cudnnGraphNotSupportedError: no engine — python or backend — proposed a plan for this graph (the backend declined: No valid engine configs for __________________ -{""engineId"":0,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":1,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":2,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":3,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":4,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: KERNEL_CFG knob must be set at: kcfg_idx < 0 -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: resolveSDPAKernelConfig(getPerfKnobs(), opSet, *this->getDeviceProp(), getEngineName(), kcfg, chosen_knob) -Warning: CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN; Reason: process_knob_choices() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":5,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Blackwell is supported for this engine at: !(1000 <= deviceVer && deviceVer < 1200) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":6,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH; Reason: only Hopper is supported for this engine at: !(900 <= deviceVer && deviceVer < 1000) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":7,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -{""engineId"":20,""smVersion"":1200,""knobChoices"":{""CUDNN_KNOB_TYPE_STAGES"":-1,""CUDNN_KNOB_TYPE_KERNEL_CFG"":-1,""CUDNN_KNOB_TYPE_TILE_CGA_M"":-1,""CUDNN_KNOB_TYPE_STREAM_K"":-1,""CUDNN_KNOB_TYPE_TILE_M"":-1,""CUDNN_KNOB_TYPE_TILE_N"":-1}} -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: The matmul should either be an entrance node or occur after the softmax and only allowed 2 at: true -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: tag_mha_graph(graph, mha_mode, is_fp8) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: (CUDNN_STATUS_NOT_INITIALIZED == status) || (CUDNN_STATUS_NOT_SUPPORTED == ((status) / 1000 * 1000)) -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr->isSupported() -Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal() - -) - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,2048,2048,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,2048,2048,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,top_left,2,2048,2048,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,2048,2048,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,2048,2048,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,cudnn,mxfp8,no_mask,2,2048,2048,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 747, in - graph_fwd.validate() - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/cudnn/_pygraph.py"", line 776, in validate - self._lowered_graph.validate() -cudnn._compiled_module.cudnnGraphNotSupportedError: MXFP8 SDPA is only supported on Blackwell Data Center architectures. - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,top_left,2,2048,2048,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,fwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,False,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, -kimi_k3,kimi_k3,flash_attention_4,bfloat16,no_mask,2,2048,2048,96,96,192,128,bwd,True,inf,0.000,0.000,10,,False,"Benchmark failed with return code 1. -stderr: Traceback (most recent call last): - File ""/data/vagarwalla/rtx6000_bench/cudnn-frontend/benchmark/sdpa_benchmark_training/benchmark_single_sdpa.py"", line 1195, in - import flash_attn.cute.interface as flash_attn_interface - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/__init__.py"", line 3, in - from .interface import ( - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/interface.py"", line 33, in - from flash_attn.cute import utils - File ""/data/vagarwalla/rtx6000_bench/venv/lib/python3.12/site-packages/flash_attn/cute/utils.py"", line 44, in - smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False - ^^^^^^^^^^^^^^^^ -AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' - -stdout: ",,, diff --git a/benchmark/sdpa_benchmark_training/results/kimi_k3/rtx_pro_6000/kimi_k3_no_mask.png b/benchmark/sdpa_benchmark_training/results/kimi_k3/rtx_pro_6000/kimi_k3_no_mask.png deleted file mode 100644 index a198474c5..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/kimi_k3/rtx_pro_6000/kimi_k3_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/kimi_k3/rtx_pro_6000/kimi_k3_top_left.png b/benchmark/sdpa_benchmark_training/results/kimi_k3/rtx_pro_6000/kimi_k3_top_left.png deleted file mode 100644 index 55f1c8341..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/kimi_k3/rtx_pro_6000/kimi_k3_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_20260813_111214.csv b/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_20260813_111214.csv deleted file mode 100644 index 964d7ab28..000000000 --- a/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_20260813_111214.csv +++ /dev/null @@ -1,211 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,22.611,1556.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,70.905,1241.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,80.868,1088.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,44.908,1567.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,134.878,1304.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,161.193,1091.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,fwd,False,16.003,2199.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,bwd,False,53.070,1658.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,bwd,True,51.997,1692.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,32.929,2137.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,98.290,1790.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,97.996,1795.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,fwd,False,17.308,2033.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,False,60.591,1452.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,True,64.391,1366.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,34.202,2057.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,124.120,1417.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,123.741,1422.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,5.469,1608.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,17.304,1271.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,19.546,1125.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,10.407,1690.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,33.714,1305.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,38.093,1155.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,fwd,False,4.085,2154.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,bwd,False,13.572,1620.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,bwd,True,13.522,1626.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,7.920,2221.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,24.362,1805.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,23.782,1849.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,fwd,False,4.324,2034.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,False,15.745,1397.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,True,15.973,1377.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,8.211,2143.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,29.449,1493.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,30.229,1455.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,1.417,1552.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,4.240,1297.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,4.825,1139.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.586,1701.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,7.668,1434.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,8.930,1231.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,fwd,False,1.080,2037.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,bwd,False,3.790,1451.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,bwd,True,3.799,1447.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.007,2191.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,6.179,1780.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,6.174,1781.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,fwd,False,1.142,1926.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,False,4.340,1267.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,True,4.339,1267.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.063,2132.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,7.391,1488.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,7.378,1490.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,0.412,1334.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,1.215,1131.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,1.353,1016.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.705,1559.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,2.075,1325.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,2.345,1172.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,fwd,False,0.305,1801.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,bwd,False,1.179,1166.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,bwd,True,1.178,1167.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.538,2045.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,1.743,1577.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,1.743,1577.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,fwd,False,0.321,1716.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,False,1.339,1027.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,True,1.340,1026.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.557,1975.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,2.052,1340.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,2.059,1335.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.130,1061.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,0.391,880.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,0.435,789.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.195,1407.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.617,1114.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.691,994.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.096,1430.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,bwd,False,0.417,824.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,bwd,True,0.417,825.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.148,1855.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.566,1215.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.564,1218.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.102,1350.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,False,0.468,735.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,True,0.468,734.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.157,1751.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.650,1057.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.649,1058.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,23.589,1492.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,46.286,1520.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,32768,32768,64,8,128,128,fwd,False,19.526,1802.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,37.257,1889.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,32768,32768,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,5.709,1541.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,11.008,1598.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,16384,16384,64,8,128,128,fwd,False,5.172,1701.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,9.334,1885.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,16384,16384,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,1.621,1357.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.717,1619.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,8192,8192,64,8,128,128,fwd,False,1.428,1540.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.376,1851.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,8192,8192,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,0.477,1153.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.737,1491.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,4096,4096,64,8,128,128,fwd,False,0.428,1284.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.643,1709.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,4096,4096,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.163,844.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.206,1335.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.152,903.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.180,1526.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,5135.000 -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,2048,2048,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,22.271,1580.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,65.921,1334.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,69.772,1261.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,57.336,1227.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,125.959,1397.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,137.038,1284.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,5.074,1734.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,15.647,1405.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,16.238,1354.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,14.461,1217.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,30.915,1423.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,30.874,1425.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,1.379,1594.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,3.918,1403.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,4.262,1290.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,3.505,1255.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,7.081,1553.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,7.730,1422.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,0.396,1388.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,1.188,1157.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,1.450,948.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.921,1194.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,1.965,1399.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,2.196,1252.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.127,1086.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,0.407,844.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,0.548,628.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.247,1112.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.592,1161.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.725,948.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_no_mask.png b/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_no_mask.png deleted file mode 100644 index 86be70300..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_no_mask_det_overhead.png b/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_no_mask_det_overhead.png deleted file mode 100644 index 013d31e59..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_no_mask_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_top_left.png b/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_top_left.png deleted file mode 100644 index 1b43f57cc..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_top_left_det_overhead.png b/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_top_left_det_overhead.png deleted file mode 100644 index 17b79def9..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_top_left_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_20260813_111515.csv b/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_20260813_111515.csv deleted file mode 100644 index 2ec0d9835..000000000 --- a/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_20260813_111515.csv +++ /dev/null @@ -1,211 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,17.482,2013.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,61.801,1423.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,69.151,1272.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,36.242,1942.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,122.833,1432.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,137.139,1283.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,fwd,False,11.334,3104.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,bwd,False,41.295,2130.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,bwd,True,41.698,2110.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,23.054,3052.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,80.792,2177.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,79.967,2200.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,fwd,False,12.243,2874.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,False,54.784,1606.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,True,55.059,1598.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,24.783,2839.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,106.652,1650.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,107.200,1641.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,4.466,1970.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,14.412,1526.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,17.078,1288.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,8.257,2131.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,29.188,1507.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,32.438,1356.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,fwd,False,2.881,3054.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,bwd,False,10.899,2018.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,bwd,True,10.871,2023.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,5.555,3167.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,19.956,2204.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,19.827,2218.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,fwd,False,3.115,2824.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,False,14.078,1562.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,True,14.072,1563.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,6.004,2930.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,26.001,1691.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,26.960,1631.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,1.159,1897.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,3.848,1429.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,4.262,1290.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.089,2105.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,7.158,1536.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,7.943,1384.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,fwd,False,0.765,2873.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,bwd,False,3.110,1768.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,bwd,True,3.110,1768.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,1.403,3134.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,5.225,2105.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,5.225,2104.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,fwd,False,0.829,2653.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,False,3.933,1398.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,True,3.935,1397.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,1.516,2902.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,6.757,1627.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,6.757,1627.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,0.317,1736.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,1.099,1251.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,1.215,1132.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.540,2038.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,1.929,1425.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,2.195,1252.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,fwd,False,0.218,2525.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,bwd,False,0.994,1384.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,bwd,True,0.994,1383.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.374,2936.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,1.502,1830.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,1.504,1828.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,fwd,False,0.237,2319.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,False,1.211,1135.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,True,1.211,1135.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.405,2718.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,1.888,1456.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,1.891,1454.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.097,1420.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,0.358,961.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,0.391,880.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.147,1874.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.568,1210.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.639,1076.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.070,1960.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,bwd,False,0.365,941.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,bwd,True,0.366,939.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.102,2691.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.498,1381.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.498,1379.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.076,1812.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,False,0.427,806.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,True,0.428,804.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.112,2465.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.596,1153.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.597,1151.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,18.870,1865.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,37.173,1893.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,32768,32768,64,8,128,128,fwd,False,15.273,2304.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,28.551,2465.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,32768,32768,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,4.769,1845.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,9.006,1953.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,16384,16384,64,8,128,128,fwd,False,4.065,2164.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,7.228,2434.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,16384,16384,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,1.292,1702.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.272,1936.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,8192,8192,64,8,128,128,fwd,False,1.144,1923.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,1.861,2364.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,8192,8192,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,0.381,1442.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.588,1871.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,4096,4096,64,8,128,128,fwd,False,0.359,1530.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.510,2155.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,4096,4096,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.131,1052.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.158,1738.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.127,1082.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.145,1894.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,5155.000 -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,2048,2048,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,17.495,2011.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,58.683,1499.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,60.940,1443.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,35.051,2008.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,122.031,1442.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,127.132,1384.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,4.512,1950.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,14.225,1546.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,15.122,1454.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,8.562,2055.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,27.589,1594.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,30.409,1446.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,1.192,1846.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,3.867,1422.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,4.201,1309.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,2.122,2072.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,7.051,1559.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,7.495,1467.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,0.341,1614.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,1.163,1182.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,1.419,969.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,0.554,1985.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,1.925,1428.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,2.174,1264.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.110,1256.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,0.402,855.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,0.536,641.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.148,1861.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,0.581,1183.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,0.715,961.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_no_mask.png b/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_no_mask.png deleted file mode 100644 index 46987881b..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_no_mask_det_overhead.png b/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_no_mask_det_overhead.png deleted file mode 100644 index 419b8c09b..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_no_mask_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_top_left.png b/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_top_left.png deleted file mode 100644 index bedb7a26d..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_top_left_det_overhead.png b/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_top_left_det_overhead.png deleted file mode 100644 index c79bc00ce..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_top_left_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_20260814_104551.csv b/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_20260814_104551.csv deleted file mode 100644 index aad962f18..000000000 --- a/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_20260814_104551.csv +++ /dev/null @@ -1,211 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,90.646,388.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,446.000 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,329.587,267.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,181.404,388.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,587.943,299.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,fwd,False,62.434,564.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,top_left,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,119.257,590.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,22.341,394.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,83.233,264.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,44.602,394.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,147.995,297.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,fwd,False,15.340,573.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,top_left,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,29.320,600.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,5.575,394.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,21.795,252.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,20.777,265.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,10.762,409.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,37.622,292.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,42.323,260.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,fwd,False,4.013,548.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,top_left,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,7.174,613.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,1.513,364.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,6.047,227.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,5.641,244.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,2.751,400.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,9.986,275.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,442.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,10.749,256.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,453.500 -llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,fwd,False,1.098,501.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,932.500 -llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,top_left,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,1.859,591.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,880.500 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.448,307.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,1.837,187.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,1.661,207.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.735,374.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,2.788,246.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,442.000 -llama3.1,llama3.1,cudnn,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,2.960,232.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,fwd,False,0.346,397.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,880.500 -llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,top_left,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.533,515.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,880.500 -llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,88.277,399.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,449.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,251.247,350.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,362.441,243.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,173.685,405.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,492.100,357.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,703.979,250.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,32768,32768,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,32768,32768,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,21.703,405.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,64.888,339.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,94.839,232.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,41.952,419.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,124.413,354.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,178.301,247.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,16384,16384,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,16384,16384,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,5.449,404.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,16.898,325.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,25.199,218.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,10.260,429.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,31.953,344.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,45.851,240.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,8192,8192,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,8192,8192,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,1.488,370.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,4.913,280.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,7.538,182.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,444.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,2.643,416.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,8.301,331.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,465.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,12.262,224.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,4096,4096,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,4096,4096,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.470,293.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,1.604,214.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,442.000 -llama3.1,llama3.1,cudnn_oss,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,2.384,144.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.716,384.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,2.423,284.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn_oss,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,3.609,190.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,2048,2048,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,top_left,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,fp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,2048,2048,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,top_left,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,8,128,128,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,cudnn_oss,mxfp8,no_mask,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,fwd,False,103.978,338.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,False,295.470,298.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,465.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,fwd,False,186.876,377.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,453.500 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,False,608.385,289.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,32768,32768,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,fwd,False,25.870,340.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,False,74.609,295.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,fwd,False,46.455,379.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,457.500 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,False,152.018,289.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,16384,16384,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,fwd,False,6.650,331.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,False,19.194,286.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,442.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,fwd,False,11.352,387.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,False,38.186,288.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,8192,8192,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,fwd,False,1.918,287.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,False,5.219,263.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,fwd,False,3.128,351.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,False,9.911,277.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,440.500 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,4096,4096,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,fwd,False,0.545,253.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,False,1.637,210.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,top_left,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,fwd,False,0.841,327.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,False,2.847,241.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -llama3.1,llama3.1,flash_attention_4,bfloat16,no_mask,2,2048,2048,64,8,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_no_mask.png b/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_no_mask.png deleted file mode 100644 index 20eb574a2..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_no_mask_det_overhead.png b/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_no_mask_det_overhead.png deleted file mode 100644 index 2436fcbe7..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_no_mask_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_top_left.png b/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_top_left.png deleted file mode 100644 index e695bc37b..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_top_left_det_overhead.png b/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_top_left_det_overhead.png deleted file mode 100644 index fc6debd42..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/llama3.1/rtxpro6000/llama3.1_top_left_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/ltx2/gb200/ltx2_20260813_111950.csv b/benchmark/sdpa_benchmark_training/results/ltx2/gb200/ltx2_20260813_111950.csv deleted file mode 100644 index 574d1a958..000000000 --- a/benchmark/sdpa_benchmark_training/results/ltx2/gb200/ltx2_20260813_111950.csv +++ /dev/null @@ -1,46 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,0.433,1429.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,1.185,1305.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,1.324,1168.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,1.752,1673.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,5.118,1432.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,6.501,1127.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,2.934,1721.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,8.668,1456.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,10.683,1182.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,9.297,1612.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,28.501,1314.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,35.923,1043.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,14.684,1580.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,42.348,1370.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,52.690,1101.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,0.463,1337.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,1.948,1505.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,3.190,1583.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,9.385,1596.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,14.043,1652.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,0.594,1041.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,1.139,1358.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,1.277,1211.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,2.536,1156.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,4.767,1537.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,5.118,1432.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,4.243,1190.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,8.166,1546.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,9.057,1394.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,12.746,1175.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,23.970,1563.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,26.511,1413.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,18.936,1225.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,37.710,1538.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,42.775,1356.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 diff --git a/benchmark/sdpa_benchmark_training/results/ltx2/gb200/ltx2_no_mask.png b/benchmark/sdpa_benchmark_training/results/ltx2/gb200/ltx2_no_mask.png deleted file mode 100644 index 05b4edb49..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/ltx2/gb200/ltx2_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/ltx2/gb200/ltx2_no_mask_det_overhead.png b/benchmark/sdpa_benchmark_training/results/ltx2/gb200/ltx2_no_mask_det_overhead.png deleted file mode 100644 index 0f62c5737..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/ltx2/gb200/ltx2_no_mask_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/ltx2/gb300/ltx2_20260813_112243.csv b/benchmark/sdpa_benchmark_training/results/ltx2/gb300/ltx2_20260813_112243.csv deleted file mode 100644 index 8a1a2978f..000000000 --- a/benchmark/sdpa_benchmark_training/results/ltx2/gb300/ltx2_20260813_112243.csv +++ /dev/null @@ -1,46 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,0.338,1830.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,1.103,1402.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,1.214,1273.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,1.469,1996.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,4.823,1520.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,5.864,1250.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,2.432,2076.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,8.192,1541.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,9.738,1296.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,7.183,2086.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,25.202,1486.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,30.032,1247.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,11.146,2082.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,38.796,1495.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,42.290,1372.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,0.370,1670.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,1.622,1807.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,2.670,1891.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,7.965,1881.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,12.340,1880.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,0.340,1819.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,1.085,1425.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,1.247,1240.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,1.503,1951.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,4.684,1564.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,4.983,1471.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,2.467,2047.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,7.885,1601.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,8.507,1484.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,7.267,2062.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,24.456,1532.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,25.543,1466.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,11.464,2024.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,37.318,1554.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,41.553,1396.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 diff --git a/benchmark/sdpa_benchmark_training/results/ltx2/gb300/ltx2_no_mask.png b/benchmark/sdpa_benchmark_training/results/ltx2/gb300/ltx2_no_mask.png deleted file mode 100644 index e4ae44447..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/ltx2/gb300/ltx2_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/ltx2/gb300/ltx2_no_mask_det_overhead.png b/benchmark/sdpa_benchmark_training/results/ltx2/gb300/ltx2_no_mask_det_overhead.png deleted file mode 100644 index ad25966b5..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/ltx2/gb300/ltx2_no_mask_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/ltx2/rtxpro6000/ltx2_20260814_104551.csv b/benchmark/sdpa_benchmark_training/results/ltx2/rtxpro6000/ltx2_20260814_104551.csv deleted file mode 100644 index 6180e5341..000000000 --- a/benchmark/sdpa_benchmark_training/results/ltx2/rtxpro6000/ltx2_20260814_104551.csv +++ /dev/null @@ -1,46 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,1.762,351.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,5.535,279.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,6.448,240.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,7.152,410.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,24.552,298.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,28.715,255.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,12.532,403.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,454.500 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,49.685,254.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,454.500 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,48.998,258.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,36.613,409.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,146.162,256.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,56.543,410.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,191.987,302.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,1.608,385.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,4.373,354.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,6.722,230.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,6.807,431.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,19.509,376.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,30.643,239.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,11.865,426.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,34.180,369.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,52.092,242.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,36.389,412.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,102.591,365.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,152.551,246.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,55.934,415.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,159.273,364.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,cudnn_oss,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,230.014,252.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,fwd,False,1.912,323.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,437.500 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,False,7.236,214.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,437.500 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,6144,6144,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,fwd,False,7.744,379.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,False,32.793,223.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,450.500 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,13376,13376,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,fwd,False,13.517,374.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,False,56.357,224.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,17556,17556,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,fwd,False,39.570,379.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,False,165.016,227.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,30240,30240,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,fwd,False,61.896,375.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,False,254.556,228.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -ltx2,ltx2,flash_attention_4,bfloat16,no_mask,1,37632,37632,32,32,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, diff --git a/benchmark/sdpa_benchmark_training/results/ltx2/rtxpro6000/ltx2_no_mask.png b/benchmark/sdpa_benchmark_training/results/ltx2/rtxpro6000/ltx2_no_mask.png deleted file mode 100644 index 38abc7045..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/ltx2/rtxpro6000/ltx2_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/ltx2/rtxpro6000/ltx2_no_mask_det_overhead.png b/benchmark/sdpa_benchmark_training/results/ltx2/rtxpro6000/ltx2_no_mask_det_overhead.png deleted file mode 100644 index b2ea3ea49..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/ltx2/rtxpro6000/ltx2_no_mask_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/qwen35/gb200/qwen35_20260813_112124.csv b/benchmark/sdpa_benchmark_training/results/qwen35/gb200/qwen35_20260813_112124.csv deleted file mode 100644 index 47ed807d8..000000000 --- a/benchmark/sdpa_benchmark_training/results/qwen35/gb200/qwen35_20260813_112124.csv +++ /dev/null @@ -1,46 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,9.482,1855.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,43.026,1022.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,True,42.785,1028.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,2.370,1856.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,10.109,1088.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,True,10.046,1095.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,0.646,1702.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,2.727,1008.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,True,2.656,1035.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.190,1447.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,0.800,859.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,True,0.801,858.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.060,1140.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,0.271,634.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,True,0.272,632.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,9.625,1828.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,2.486,1769.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,0.714,1541.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.222,1240.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.075,911.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,10.523,1672.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,45.908,958.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,3.128,1406.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,11.621,946.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,1.130,973.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,3.263,842.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.468,588.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,1.404,490.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.221,311.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,0.663,259.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, diff --git a/benchmark/sdpa_benchmark_training/results/qwen35/gb200/qwen35_top_left.png b/benchmark/sdpa_benchmark_training/results/qwen35/gb200/qwen35_top_left.png deleted file mode 100644 index bc1468050..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/qwen35/gb200/qwen35_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/qwen35/gb200/qwen35_top_left_det_overhead.png b/benchmark/sdpa_benchmark_training/results/qwen35/gb200/qwen35_top_left_det_overhead.png deleted file mode 100644 index c0383febe..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/qwen35/gb200/qwen35_top_left_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/qwen35/gb300/qwen35_20260813_112434.csv b/benchmark/sdpa_benchmark_training/results/qwen35/gb300/qwen35_20260813_112434.csv deleted file mode 100644 index f7aed9282..000000000 --- a/benchmark/sdpa_benchmark_training/results/qwen35/gb300/qwen35_20260813_112434.csv +++ /dev/null @@ -1,46 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,8.201,2145.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,37.346,1178.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,True,36.518,1204.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,2.081,2113.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,9.096,1209.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,True,9.112,1207.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,0.543,2025.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,2.433,1130.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,True,2.434,1129.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.153,1796.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,0.717,959.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,True,0.715,961.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.051,1360.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,0.248,693.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,True,0.248,694.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,8.731,2015.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,2.222,1979.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,0.610,1801.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.193,1427.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.067,1031.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,10.415,1689.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,44.443,990.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,3.118,1411.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,11.328,971.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,1.119,983.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,3.162,869.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,0.460,598.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,1.374,500.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.216,318.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,0.646,266.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, diff --git a/benchmark/sdpa_benchmark_training/results/qwen35/gb300/qwen35_top_left.png b/benchmark/sdpa_benchmark_training/results/qwen35/gb300/qwen35_top_left.png deleted file mode 100644 index 45fcc0a9c..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/qwen35/gb300/qwen35_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/qwen35/gb300/qwen35_top_left_det_overhead.png b/benchmark/sdpa_benchmark_training/results/qwen35/gb300/qwen35_top_left_det_overhead.png deleted file mode 100644 index e16b85146..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/qwen35/gb300/qwen35_top_left_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/qwen35/rtxpro6000/qwen35_20260814_104551.csv b/benchmark/sdpa_benchmark_training/results/qwen35/rtxpro6000/qwen35_20260814_104551.csv deleted file mode 100644 index 0d7ea6059..000000000 --- a/benchmark/sdpa_benchmark_training/results/qwen35/rtxpro6000/qwen35_20260814_104551.csv +++ /dev/null @@ -1,46 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,107.255,164.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,446.000 -qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,27.159,162.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,6.970,158.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,450.500 -qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,1.872,147.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,454.500 -qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.568,121.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,48.459,363.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,12.082,364.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,3.284,335.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,456.500 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,1.044,263.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,0.366,188.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,cudnn_oss,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,32768,32768,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,16384,16384,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,8192,8192,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,4096,4096,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,fwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,False,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -qwen35,qwen35,flash_attention_4,bfloat16,top_left,1,2048,2048,32,2,256,256,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, diff --git a/benchmark/sdpa_benchmark_training/results/qwen35/rtxpro6000/qwen35_top_left.png b/benchmark/sdpa_benchmark_training/results/qwen35/rtxpro6000/qwen35_top_left.png deleted file mode 100644 index fc6075657..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/qwen35/rtxpro6000/qwen35_top_left.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/qwen3vl_vit/gb300/qwen3vl_vit_20260814_193600.csv b/benchmark/sdpa_benchmark_training/results/qwen3vl_vit/gb300/qwen3vl_vit_20260814_193600.csv deleted file mode 100644 index cd490d98d..000000000 --- a/benchmark/sdpa_benchmark_training/results/qwen3vl_vit/gb300/qwen3vl_vit_20260814_193600.csv +++ /dev/null @@ -1,31 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -qwen3vl_vit,qwen3vl_vit,cudnn,bfloat16,no_mask,1,8836,8836,16,16,80,80,fwd,False,0.329,1215.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,2464.000 -qwen3vl_vit,qwen3vl_vit,cudnn,fp8,no_mask,1,8836,8836,16,16,80,80,fwd,False,0.258,1547.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,4927.000 -qwen3vl_vit,qwen3vl_vit,cudnn_oss,bfloat16,no_mask,1,8836,8836,16,16,80,80,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -qwen3vl_vit,qwen3vl_vit,cudnn_oss,fp8,no_mask,1,8836,8836,16,16,80,80,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -qwen3vl_vit,qwen3vl_vit,flash_attention_4,bfloat16,no_mask,1,8836,8836,16,16,80,80,fwd,False,0.274,1457.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,2464.000 -qwen3vl_vit,qwen3vl_vit,cudnn,bfloat16,no_mask,1,15376,15376,16,16,80,80,fwd,False,0.975,1241.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,2464.000 -qwen3vl_vit,qwen3vl_vit,cudnn,fp8,no_mask,1,15376,15376,16,16,80,80,fwd,False,0.743,1629.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,4927.000 -qwen3vl_vit,qwen3vl_vit,cudnn_oss,bfloat16,no_mask,1,15376,15376,16,16,80,80,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -qwen3vl_vit,qwen3vl_vit,cudnn_oss,fp8,no_mask,1,15376,15376,16,16,80,80,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -qwen3vl_vit,qwen3vl_vit,flash_attention_4,bfloat16,no_mask,1,15376,15376,16,16,80,80,fwd,False,0.811,1492.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,2464.000 -qwen3vl_vit,qwen3vl_vit,cudnn,bfloat16,no_mask,1,24336,24336,16,16,80,80,fwd,False,2.436,1245.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,2464.000 -qwen3vl_vit,qwen3vl_vit,cudnn,fp8,no_mask,1,24336,24336,16,16,80,80,fwd,False,1.777,1707.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,4927.000 -qwen3vl_vit,qwen3vl_vit,cudnn_oss,bfloat16,no_mask,1,24336,24336,16,16,80,80,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -qwen3vl_vit,qwen3vl_vit,cudnn_oss,fp8,no_mask,1,24336,24336,16,16,80,80,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -qwen3vl_vit,qwen3vl_vit,flash_attention_4,bfloat16,no_mask,1,24336,24336,16,16,80,80,fwd,False,2.054,1476.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,2464.000 -qwen3vl_vit,qwen3vl_vit,cudnn,bfloat16,no_mask,1,35344,35344,16,16,80,80,fwd,False,5.154,1241.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,2464.000 -qwen3vl_vit,qwen3vl_vit,cudnn,fp8,no_mask,1,35344,35344,16,16,80,80,fwd,False,3.717,1721.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,4927.000 -qwen3vl_vit,qwen3vl_vit,cudnn_oss,bfloat16,no_mask,1,35344,35344,16,16,80,80,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -qwen3vl_vit,qwen3vl_vit,cudnn_oss,fp8,no_mask,1,35344,35344,16,16,80,80,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -qwen3vl_vit,qwen3vl_vit,flash_attention_4,bfloat16,no_mask,1,35344,35344,16,16,80,80,fwd,False,4.291,1491.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,2464.000 -qwen3vl_vit,qwen3vl_vit,cudnn,bfloat16,no_mask,1,47376,47376,16,16,80,80,fwd,False,9.175,1253.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,2464.000 -qwen3vl_vit,qwen3vl_vit,cudnn,fp8,no_mask,1,47376,47376,16,16,80,80,fwd,False,6.723,1709.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,4927.000 -qwen3vl_vit,qwen3vl_vit,cudnn_oss,bfloat16,no_mask,1,47376,47376,16,16,80,80,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -qwen3vl_vit,qwen3vl_vit,cudnn_oss,fp8,no_mask,1,47376,47376,16,16,80,80,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -qwen3vl_vit,qwen3vl_vit,flash_attention_4,bfloat16,no_mask,1,47376,47376,16,16,80,80,fwd,False,7.639,1504.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,2464.000 -qwen3vl_vit,qwen3vl_vit,cudnn,bfloat16,no_mask,1,62500,62500,16,16,80,80,fwd,False,15.758,1269.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,2464.000 -qwen3vl_vit,qwen3vl_vit,cudnn,fp8,no_mask,1,62500,62500,16,16,80,80,fwd,False,12.135,1648.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,4927.000 -qwen3vl_vit,qwen3vl_vit,cudnn_oss,bfloat16,no_mask,1,62500,62500,16,16,80,80,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -qwen3vl_vit,qwen3vl_vit,cudnn_oss,fp8,no_mask,1,62500,62500,16,16,80,80,fwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this fwd graph (shape/pass/feature not covered),,,, -qwen3vl_vit,qwen3vl_vit,flash_attention_4,bfloat16,no_mask,1,62500,62500,16,16,80,80,fwd,False,14.051,1423.000,0.000,10,,True,False,,NVIDIA B300 SXM6 AC,1.27.0,92500.000,2464.000 diff --git a/benchmark/sdpa_benchmark_training/results/qwen3vl_vit/gb300/qwen3vl_vit_no_mask.png b/benchmark/sdpa_benchmark_training/results/qwen3vl_vit/gb300/qwen3vl_vit_no_mask.png deleted file mode 100644 index 0f753a0e6..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/qwen3vl_vit/gb300/qwen3vl_vit_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/wan22/gb200/wan22_20260813_112101.csv b/benchmark/sdpa_benchmark_training/results/wan22/gb200/wan22_20260813_112101.csv deleted file mode 100644 index dcef11e44..000000000 --- a/benchmark/sdpa_benchmark_training/results/wan22/gb200/wan22_20260813_112101.csv +++ /dev/null @@ -1,46 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,0.851,1464.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,2.341,1330.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,2.883,1081.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,3.607,1672.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,11.057,1364.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,14.103,1069.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,15.052,1460.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,43.184,1272.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,54.124,1015.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,31.501,1520.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,97.617,1227.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,120.754,992.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,80.892,1447.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,246.078,1189.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,303.132,965.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,0.903,1380.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,3.829,1575.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,13.503,1628.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,33.716,1421.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,84.944,1378.000,0.000,10,,True,False,,NVIDIA GB200,1.27.0,92700.000,2568.000 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,1.134,1099.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,2.138,1457.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,2.278,1367.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,4.965,1215.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,9.754,1546.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,10.520,1433.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,17.817,1234.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,33.943,1619.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,38.880,1413.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,40.567,1181.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,84.447,1418.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,95.404,1255.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,105.615,1108.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,213.532,1370.000,0.000,10,,True,False,,NVIDIA GB200,,,2568.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,241.683,1211.000,0.000,10,,True,False,,NVIDIA GB200,,,149.000 diff --git a/benchmark/sdpa_benchmark_training/results/wan22/gb200/wan22_no_mask.png b/benchmark/sdpa_benchmark_training/results/wan22/gb200/wan22_no_mask.png deleted file mode 100644 index e43163f4a..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/wan22/gb200/wan22_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/wan22/gb200/wan22_no_mask_det_overhead.png b/benchmark/sdpa_benchmark_training/results/wan22/gb200/wan22_no_mask_det_overhead.png deleted file mode 100644 index 37ab5347c..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/wan22/gb200/wan22_no_mask_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/wan22/gb300/wan22_20260813_112353.csv b/benchmark/sdpa_benchmark_training/results/wan22/gb300/wan22_20260813_112353.csv deleted file mode 100644 index ae9d893c4..000000000 --- a/benchmark/sdpa_benchmark_training/results/wan22/gb300/wan22_20260813_112353.csv +++ /dev/null @@ -1,46 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,0.655,1902.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,2.163,1440.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,2.618,1190.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,2.924,2063.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,9.982,1510.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,12.074,1249.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,10.837,2028.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,37.826,1453.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,45.266,1214.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,24.038,1993.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,84.022,1425.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,100.053,1197.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,59.280,1975.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,215.118,1360.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,254.474,1150.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,0.721,1727.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,3.164,1906.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,11.718,1876.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,26.029,1840.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,64.614,1812.000,0.000,10,,True,False,,NVIDIA GB300,1.27.0,92700.000,2578.000 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,True,UNSUPPORTED_CONFIG: no FROST OSS engine serves this bwd graph (shape/pass/feature not covered),,,, -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,0.666,1871.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,2.104,1480.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,2.265,1375.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,2.938,2053.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,9.593,1572.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,10.531,1432.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,10.666,2061.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,34.801,1579.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,38.308,1434.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,24.209,1978.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,82.522,1451.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,92.877,1289.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,61.985,1888.000,0.000,10,,True,False,,NVIDIA GB300,,,2578.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,199.802,1465.000,0.000,10,,True,False,,NVIDIA GB300,,,467.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,232.722,1257.000,0.000,10,,True,False,,NVIDIA GB300,,,149.000 diff --git a/benchmark/sdpa_benchmark_training/results/wan22/gb300/wan22_no_mask.png b/benchmark/sdpa_benchmark_training/results/wan22/gb300/wan22_no_mask.png deleted file mode 100644 index fac00649b..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/wan22/gb300/wan22_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/wan22/gb300/wan22_no_mask_det_overhead.png b/benchmark/sdpa_benchmark_training/results/wan22/gb300/wan22_no_mask_det_overhead.png deleted file mode 100644 index 2e08cf0a0..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/wan22/gb300/wan22_no_mask_det_overhead.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/wan22/rtxpro6000/wan22_20260814_104551.csv b/benchmark/sdpa_benchmark_training/results/wan22/rtxpro6000/wan22_20260814_104551.csv deleted file mode 100644 index 4ab7ce72d..000000000 --- a/benchmark/sdpa_benchmark_training/results/wan22/rtxpro6000/wan22_20260814_104551.csv +++ /dev/null @@ -1,46 +0,0 @@ -config_name,model_name,backend,data_type,attn_mask,batch_size,q_seqlen,kv_seqlen,num_q_heads,num_kv_heads,head_dim_qk,head_dim_vo,profile_pass,deterministic_bwd,time_ms,tflops,max_diff,num_iterations,sliding_window_size,success,skipped,error_message,gpu_name,cudnn_version,cudnn_backend_version,peak_mma_tflops -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,3.259,382.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,12.610,247.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,440.500 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,12.397,251.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,14.813,407.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,59.301,254.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,58.663,257.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,55.165,398.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,457.500 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,213.032,258.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,122.738,390.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,463.671,258.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,301.785,388.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,1125.823,260.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,2.934,425.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,8.587,363.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,13.202,236.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,14.062,429.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,41.090,367.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,61.663,245.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,52.782,416.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,149.470,368.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,220.680,249.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,116.124,412.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,325.406,368.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,479.788,250.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,284.311,412.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,465.000 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,802.493,365.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,466.500 -wan22,wan22_a14b,cudnn_oss,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,1160.301,252.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,1.27.0,92700.000,457.500 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,fwd,False,3.436,363.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,437.500 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,False,14.158,220.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,437.500 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,7800,7800,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,fwd,False,15.948,378.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,439.000 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,False,67.084,225.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,453.500 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,17160,17160,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,fwd,False,57.652,381.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,457.500 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,False,240.702,228.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,32760,32760,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,fwd,False,126.464,379.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,False,522.521,229.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,48360,48360,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,fwd,False,309.913,378.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,False,1273.217,230.000,0.000,10,,True,False,,NVIDIA RTX PRO 6000 Blackwell Server Edition,,,466.500 -wan22,wan22_a14b,flash_attention_4,bfloat16,no_mask,1,75600,75600,40,40,128,128,bwd,True,inf,0.000,0.000,10,,False,False,Benchmark failed with return code 1.,,,, diff --git a/benchmark/sdpa_benchmark_training/results/wan22/rtxpro6000/wan22_no_mask.png b/benchmark/sdpa_benchmark_training/results/wan22/rtxpro6000/wan22_no_mask.png deleted file mode 100644 index 62a5dce7f..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/wan22/rtxpro6000/wan22_no_mask.png and /dev/null differ diff --git a/benchmark/sdpa_benchmark_training/results/wan22/rtxpro6000/wan22_no_mask_det_overhead.png b/benchmark/sdpa_benchmark_training/results/wan22/rtxpro6000/wan22_no_mask_det_overhead.png deleted file mode 100644 index cf28bba7f..000000000 Binary files a/benchmark/sdpa_benchmark_training/results/wan22/rtxpro6000/wan22_no_mask_det_overhead.png and /dev/null differ diff --git a/docs/operations/Attention.md b/docs/operations/Attention.md index ef4e8edf7..3f758aaf9 100644 --- a/docs/operations/Attention.md +++ b/docs/operations/Attention.md @@ -136,40 +136,40 @@ The support matrix is based on the latest cudnn backend version 9.18.1 ## Benchmarks -To run the sdpa benchmarks, refer to [benchmarks/sdpa](https://github.com/NVIDIA/cudnn-frontend/blob/main/benchmark/sdpa_benchmark_training/README.md) folder. Current results: +To run the sdpa benchmarks, refer to [benchmarks/sdpa](https://github.com/NVIDIA/cudnn-frontend/blob/main/benchmark/attention_training/README.md) folder. Current results: ### GB200 - Llama 3.1 Causal (top_left) -![Llama 3.1 Causal on GB200](https://raw.githubusercontent.com/NVIDIA/cudnn-frontend/main/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_top_left.png) +![Llama 3.1 Causal on GB200](https://raw.githubusercontent.com/NVIDIA/cudnn-frontend/main/benchmark/attention_training/results/llama3.1/gb200/llama3.1_top_left.png) - SDPA parameters: `batch=1; num_q_heads=64; num_kv_heads=8; head_dim=128; is_causal=True` - Sequence lengths shown on x-axis - Results obtained on NVIDIA GB200 GPU ### GB200 - Llama 3.1 Non-Causal (no_mask) -![Llama 3.1 Non-Causal on GB200](https://raw.githubusercontent.com/NVIDIA/cudnn-frontend/main/benchmark/sdpa_benchmark_training/results/llama3.1/gb200/llama3.1_no_mask.png) +![Llama 3.1 Non-Causal on GB200](https://raw.githubusercontent.com/NVIDIA/cudnn-frontend/main/benchmark/attention_training/results/llama3.1/gb200/llama3.1_no_mask.png) - SDPA parameters: `batch=1; num_q_heads=64; num_kv_heads=8; head_dim=128; is_causal=False` - Sequence lengths shown on x-axis - Results obtained on NVIDIA GB200 GPU ### GB200 - DeepSeek V3 Causal (top_left) -![DeepSeek V3 Causal on GB200](https://raw.githubusercontent.com/NVIDIA/cudnn-frontend/main/benchmark/sdpa_benchmark_training/results/dsv3/gb200/dsv3_top_left.png) +![DeepSeek V3 Causal on GB200](https://raw.githubusercontent.com/NVIDIA/cudnn-frontend/main/benchmark/attention_training/results/dsv3/gb200/dsv3_top_left.png) - SDPA parameters: `batch=1; num_q_heads=128; num_kv_heads=128; head_dim_qk=192; head_dim_vo=128; is_causal=True` - Sequence lengths shown on x-axis - Results obtained on NVIDIA GB200 GPU ### GB300 - Llama 3.1 Causal (top_left) -![Llama 3.1 Causal on GB300](https://raw.githubusercontent.com/NVIDIA/cudnn-frontend/main/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_top_left.png) +![Llama 3.1 Causal on GB300](https://raw.githubusercontent.com/NVIDIA/cudnn-frontend/main/benchmark/attention_training/results/llama3.1/gb300/llama3.1_top_left.png) - SDPA parameters: `batch=1; num_q_heads=64; num_kv_heads=8; head_dim=128; is_causal=True` - Sequence lengths shown on x-axis - Results obtained on NVIDIA GB300 GPU ### GB300 - Llama 3.1 Non-Causal (no_mask) -![Llama 3.1 Non-Causal on GB300](https://raw.githubusercontent.com/NVIDIA/cudnn-frontend/main/benchmark/sdpa_benchmark_training/results/llama3.1/gb300/llama3.1_no_mask.png) +![Llama 3.1 Non-Causal on GB300](https://raw.githubusercontent.com/NVIDIA/cudnn-frontend/main/benchmark/attention_training/results/llama3.1/gb300/llama3.1_no_mask.png) - SDPA parameters: `batch=1; num_q_heads=64; num_kv_heads=8; head_dim=128; is_causal=False` - Sequence lengths shown on x-axis - Results obtained on NVIDIA GB300 GPU ### GB300 - DeepSeek V3 Causal (top_left) -![DeepSeek V3 Causal on GB300](https://raw.githubusercontent.com/NVIDIA/cudnn-frontend/main/benchmark/sdpa_benchmark_training/results/dsv3/gb300/dsv3_top_left.png) +![DeepSeek V3 Causal on GB300](https://raw.githubusercontent.com/NVIDIA/cudnn-frontend/main/benchmark/attention_training/results/dsv3/gb300/dsv3_top_left.png) - SDPA parameters: `batch=1; num_q_heads=128; num_kv_heads=128; head_dim_qk=192; head_dim_vo=128; is_causal=True` - Sequence lengths shown on x-axis - Results obtained on NVIDIA GB300 GPU diff --git a/llms.txt b/llms.txt index 5439d4a8f..527199701 100644 --- a/llms.txt +++ b/llms.txt @@ -46,7 +46,7 @@ Published documentation: https://docs.nvidia.com/deeplearning/cudnn/latest/devel - [C++ samples (Catch2)](https://github.com/NVIDIA/cudnn-frontend/tree/main/samples/cpp) - [Python notebook tutorials](https://github.com/NVIDIA/cudnn-frontend/tree/main/samples/python) -- [SDPA training benchmark](https://github.com/NVIDIA/cudnn-frontend/tree/main/benchmark/sdpa_benchmark_training) +- [SDPA training benchmark](https://github.com/NVIDIA/cudnn-frontend/tree/main/benchmark/attention_training) ## Optional