sdpa fp8 sm100: fix THD O-descriptor row stride (latent) by vedaanta · Pull Request #577 · NVIDIA/cudnn-frontend

vedaanta · 2026-08-13T07:29:03Z

Summary

One-line latent-bug fix in the SM100 per-tensor FP8 kernel's THD path: the per-batch O-descriptor builder takes the declared per-token element stride of packed O (QH * d_v), but the fp8 kernel passed CFG.TILE_O — coincidentally correct only at QH == 1 with d_v == 128, and otherwise landing every batch ≥ 1's descriptor base at 1/QH of the true offset inside earlier batches' O rows (region b receives batches {b, b+1}'s stores; the last region goes stale). The f16 kernel has carried the correct form (o_tensor.stride[1]) since its envelope work — this aligns the fp8 clone.

Why latent

The engine row declares thd=False and no API path drives fp8 THD today, so no shipping configuration reaches the bug. It should land before any fp8 THD enablement does (see the companion SM107 enablement PR, whose sibling kernel copy inherits this fix).

Evidence

Reproduced and isolated on SM107 silicon (the first hardware ever to run this path) with per-(batch, head) sentinel-V probes:

H=1 × B=N exact, H=N × B=1 exact, H>1 × B>1 corrupt — precisely the QH-scaling signature
with the fix: irregular-varlen e2e (mixed lengths, B=3, H=4) matches the per-sequence fp32 dequant reference; the full fp8 suite is green on both SM100 and SM107

🤖 Generated with Claude Code

Summary by CodeRabbit

Bug Fixes
- Improved variable-length attention processing by using the correct output stride when constructing output descriptors.
- Helps ensure accurate results for supported FP8 workloads.

The per-batch O-descriptor builder takes the DECLARED per-token element stride of packed O (QH * d_v). The fp8 kernel passed CFG.TILE_O - coincidentally correct only at QH == 1 with d_v == 128, otherwise landing every batch >= 1's descriptor base at 1/QH of the true offset inside earlier batches' O rows (region b receives batches {b, b+1}'s stores, the last region goes stale). Latent today - the engine declares thd=False and no API path drives fp8 THD - but real: reproduced and isolated on SM107 silicon via sentinel-V probes, with H=1xB=N and H=NxB=1 exact and H>1xB>1 corrupt. The f16 kernel has carried the same fix since its envelope work; this aligns the fp8 kernel before any THD enablement lands on top. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

coderabbitai · 2026-08-13T07:29:29Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a75d2e7f-9c9d-461f-b976-61126867ce24

📥 Commits

Reviewing files that changed from the base of the PR and between a7b4ca2 and 16e04d7.

📒 Files selected for processing (1)

python/cudnn/sdpa/fwd/kernels/prefill_d128_fp8_sm100.py

📝 Walkthrough

Walkthrough

The THD O-descriptor construction now uses o_tensor.stride[1] for the packed per-token element stride instead of CFG.TILE_O.

Changes

THD output descriptor

Layer / File(s)	Summary
Use the output tensor stride `python/cudnn/sdpa/fwd/kernels/prefill_d128_fp8_sm100.py`	Per-batch THD O-descriptors now use the output tensor’s actual per-token stride instead of the fixed tile width.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Mergeability Score: ⚪ Minimal · up to 16e04

This localized fix corrects FP8 THD output descriptor addressing and is reported validated across the relevant test suites; no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: anerudhan, yanzhuo607

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description explains the bug, impact, rationale, and test evidence, but omits several required template sections and exact test commands.	Add the affected area, checklist status, related issues, API and compatibility impact, and Testing sections with exact commands and results.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the SM100 FP8 THD O-descriptor row-stride fix.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

vedaanta · 2026-08-13T07:35:20Z

@cudnn-ci-bot run frost

cudnn-ci-bot · 2026-08-13T07:35:45Z

🚀 Running mirror pipeline

Branch: cudnn-gh/pr-577-16e04d7
Pipeline: 62481780
Targets: frost

…ard (#579) Two follow-ups to #576: 1. #574 (has_lse specialization for the FP8/MXFP8 SM100 flavors) merged between #577 and #576, so the shared adapter now passes has_lse to the fp8 compile() while the SM107 sibling still had the pre-#574 signature - a TypeError on any Rubin fp8 compile at develop tip. This ports #574's fp8-kernel hunks onto the sibling verbatim (LSE None-specialization, compile(has_lse), the specialized epilogue), restoring signature parity with the shared call site. 2. Responding to review on #576: the 9-stage ring with BF16/FP16 O (~242 KiB) exceeds the STANDARD sm_10x 227 KiB per-CTA opt-in and is legal on GR100 only through the sm107 oversized-SMEM launch mode (function attribute 16), which the required internal cutlass-dsl toolchain enables for its sm_107a kernels - board-validated e4m3->bf16 across the full suite. Rather than rejecting output dtypes that demonstrably work on the target stack, a static import-time guard now accounts the geometry against the GR100 hardware budget so a future stage/width bump fails with a clear message instead of at launch. Validated: SM100 box 34 passed; SM107 board 34 passed (full fp8 e2e suite through the sibling, has_lse and no-lse populations both exercised via the suite's generate_stats matrix). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

… FP8 Two capability enablements for the per-tensor FP8 path: 1. THD/varlen execute. _execute_fp8_thd lands the f16 _execute_thd recipe on the FP8 ABI: packed [1,T,H,D] views (packed contract only, gated in check_support), the [seq_kv | cu_q | cu_k] metadata buffer, the per-sequence O-descriptor array, folded descales, in-kernel amax slots, and the head-major packed LSE [1,QH,T] (bound with has_lse=True). THD samples are declared at LOGICAL dims (B, H, S_max, D); buffers stay packed at execute. Serves every fp8-capable part: the SM100 kernel's O-descriptor stride fix (NVIDIA#577) makes cc10.0/10.3 correct - the irregular varlen e2e passes on SM100 silicon, the first Blackwell validation of this path - and cc10.7 routes to the sibling as usual. 2. d<=128 ENVELOPE for per-tensor FP8 on Rubin (the ViT d=72-in-80 contract's landing zone): compile() takes the ACTUAL head dims into the TMA descriptors while the tile box stays d128 - loads past d zero-fill (FP8 zero is exact), O stores OOB-clip, same semantics as the f16 kernels. Equal head dims, d % 16 == 0 (TMA 16-byte global-stride rule at BPE=1). Blackwell fp8 and MXFP8 keep the exact-d128 gate; the d_qk/d_v compile() parameters are added to BOTH fp8 kernels so the sibling signatures stay in lockstep with the shared adapter call site (the NVIDIA#574 lesson). Direct-adapter surface only; the engine row still declares thd=False and d128 - graph-route exposure is follow-up work. Validation: SM107 board 36 passed (d80 envelope e2e + irregular THD e2e + full fp8 suite); SM100 box 35 passed + 1 skip (envelope is Rubin-gated). Perf work for these shapes (the cga=1 short-sequence config) follows separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

vedaanta added cat-bug Reports of incorrect behavior, crashes, regressions, or unexpected results. orig-nv-eng Reported or requested by NVIDIA engineering. mod-cutedsl CuTeDSL kernels, generated kernels, examples, or related integration work. mod-frost labels Aug 13, 2026

jhjpark approved these changes Aug 13, 2026

View reviewed changes

vedaanta merged commit a305d51 into NVIDIA:develop Aug 13, 2026
1 check passed

This was referenced Aug 13, 2026

sdpa fp8 sm107: port the has_lse specialization; add a static SMEM guard #579

Merged

frost(sdpa): bind THD host prep to the launch stream; plan-time-only THD compile keys (#552) #543

Merged

vedaanta mentioned this pull request Aug 13, 2026

sdpa fp8: dense head-dim envelope + one engine per arch x dtype family #587

Merged

Navigation Menu

Uh oh!

sdpa fp8 sm100: fix THD O-descriptor row stride (latent) - #577

sdpa fp8 sm100: fix THD O-descriptor row stride (latent)#577
vedaanta merged 1 commit into
NVIDIA:developfrom
vedaanta:fp8-thd-odesc-stride-fix

vedaanta commented Aug 13, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Aug 13, 2026 •

edited

Loading

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

vedaanta commented Aug 13, 2026

Uh oh!

cudnn-ci-bot commented Aug 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

vedaanta commented Aug 13, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why latent

Evidence

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

vedaanta commented Aug 13, 2026

Uh oh!

cudnn-ci-bot commented Aug 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vedaanta commented Aug 13, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 13, 2026 •

edited

Loading