Skip to content

samples: skip unsupported test case, remove invalid cudaGraphDestroy - #734

Merged
Anerudhan merged 2 commits into
NVIDIA:developfrom
dextero:develop
Aug 25, 2026
Merged

Anerudhan merged 2 commits into
NVIDIA:developfrom
dextero:develop

Conversation

@dextero

@dextero dextero commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Before submitting

  • I agree to license this contribution under the terms of LICENSE.txt.
  • I ran pre-commit run and committed any formatting changes.
  • I added GitHub labels: one cat-*, one or more mod-*, and one orig-* (see label list).

cat-bugfix mod-frontend orig-external (The UI doesn't appear to let me attach labels)

Affected area

  • Documentation or samples

Summary

  • Skip deterministic SDPA backward in sample when running on Blackwell with tool old cuDNN. This is explicitly marked as unsupported in the code.
  • Remove invalid cudaGraphDestroy(cudnn_cuda_graph_new) in cudagraphs sample. As per documentation of cudaGraphChildGraphNodeGetGraph it's owned by the parent main_cuda_graph and gets destroyed whenever the main one is.

Why

This makes the samples tests pass on older cuDNN.

Related issues

None

API and compatibility impact

None

Testing

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j $(nproc)
build/bin/samples
build/bin/legacy_samples

Summary by CodeRabbit

  • Bug Fixes
    • Improved error handling during CUDA graph cleanup.
    • Added a compatibility skip for deterministic SDPA backward tests on Blackwell hardware with older cuDNN versions.

Marcin Radomski added 2 commits August 25, 2026 10:55
Blackwell + cuDNN<9.18 + deterministic SDPA backward is a combination
explicitly marked as unsupported at
include/cudnn_frontend/node/scaled_dot_product_flash_attention.h:1385.
… sample

In "Cuda graphs with matmul add", `cudnn_cuda_graph_new` is obtained via
`cudaGraphChildGraphNodeGetGraph(cudnn_node_in_main_graph, &cudnn_cuda_graph_new)`.

Per the CUDA Runtime API documentation for `cudaGraphChildGraphNodeGetGraph`:
"This call does not clone the graph. Changes to the graph will be reflected in
the node, and the node retains ownership of the graph."

Destroying `main_cuda_graph` with `cudaGraphDestroy(main_cuda_graph)` destroys
the parent graph and all embedded child graphs owned by its nodes. Calling
`cudaGraphDestroy(cudnn_cuda_graph_new)` afterwards attempts to destroy an
already-destroyed graph handle, returning cudaErrorInvalidValue and leaving a
sticky CUDA runtime error in the process device context that corrupts subsequent
tests executing on that context in monolithic test runners like Catch2.

Remove the invalid `cudaGraphDestroy(cudnn_cuda_graph_new)` call and wrap
destruction calls in `CUDA_CHECK`.
@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c53c8f8e-1e6b-45c0-94c5-3def674b346b

📥 Commits

Reviewing files that changed from the base of the PR and between ae8705e and 659cd39.

📒 Files selected for processing (2)
  • samples/cpp/misc/cudagraphs.cpp
  • samples/cpp/sdpa/fp16_bwd.cpp

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


📝 Walkthrough

Walkthrough

The changes add CUDA status checks during graph cleanup, remove redundant cuDNN graph destruction, and skip unsupported deterministic SDPA backward execution on Blackwell with older cuDNN versions.

Changes

CUDA graph cleanup

Layer / File(s) Summary
Checked graph destruction
samples/cpp/misc/cudagraphs.cpp
cudaGraphExecDestroy and cudaGraphDestroy now use CUDA_CHECK. Explicit destruction of cudnn_cuda_graph_new was removed.

SDPA backward compatibility

Layer / File(s) Summary
Deterministic SDPA compatibility skip
samples/cpp/sdpa/fp16_bwd.cpp
The test skips deterministic SDPA backward execution on Blackwell when cuDNN is older than 9.18.0.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 659cd

This localized sample change skips an unsupported test case and removes an invalid CUDA graph destruction call; no actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: anerudhan

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 2…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes both primary changes: skipping an unsupported test case and removing the invalid cudaGraphDestroy call.
Description check ✅ Passed The description covers the affected area, summary, rationale, related issues, compatibility impact, and testing commands. The Milestone and Projects fields are not addressed, and test results are not …
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 2 files.

Full details: Description check

Explanation

The description covers the affected area, summary, rationale, related issues, compatibility impact, and testing commands. The Milestone and Projects fields are not addressed, and test results are not reported, but the description is otherwise complete and relevant.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@Anerudhan Anerudhan added this to the Frontend 1.28.0 milestone Aug 25, 2026
@Anerudhan Anerudhan added mod-frontend cuDNN frontend APIs, operation graph construction, plans, and user-facing wrappers. orig-external Reported or requested by an external user, customer, or community contributor. cat-cleanup labels Aug 25, 2026
@Anerudhan

Copy link
Copy Markdown
Collaborator

Thanks @dextero for the PR.
Merging

@Anerudhan
Anerudhan merged commit 07d1aa0 into NVIDIA:develop Aug 25, 2026
5 of 6 checks passed
@Anerudhan Anerudhan mentioned this pull request Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cat-cleanup mod-frontend cuDNN frontend APIs, operation graph construction, plans, and user-facing wrappers. orig-external Reported or requested by an external user, customer, or community contributor.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants