feat: handle-less plan deserialization via DeviceProperties (cuDNN 9.8+) - #544
Merged
hwanseoc merged 6 commits intoAug 12, 2026
Merged
Conversation
Add a new Graph::deserialize(blob) overload and supporting API layers that
allow callers to rehydrate a serialized execution plan without holding a
cudnnHandle_t. Instead, a DeviceProperties descriptor (backed by the
CUDNN_ATTR_EXECUTION_PLAN_DEVICEPROP attribute introduced in cuDNN 9.8) is
supplied at graph-construction time and forwarded through the builder chain.
The key motivation is ahead-of-time compilation workflows: a plan can be
built and serialized on a GPU node at build time, then deserialized in a
context where no CUDA context or cuDNN handle exists yet — only a descriptor
of the target device's properties is needed.
Changes by layer:
Layer 1 – ExecutionPlanBuilder_v8 (cudnn_frontend_ExecutionPlan.h)
• New setDeviceProperties() builder method.
• loadFromJson() conditionally sets CUDNN_ATTR_EXECUTION_PLAN_DEVICEPROP
when device_properties is non-null (guarded by #if CUDNN_VERSION >= 90800).
• #else path emits an explicit NOT_SUPPORTED error when device_properties
is set but headers are older than 9.8, avoiding a confusing null-handle
error.
• detail::get_backend_version() < 90800 runtime guard for new-header /
old-runtime combinations.
Layer 2 – Execution_plan_list / plans.h
• New build_plans(std::shared_ptr<const DeviceProperties>, string)
overload via create_cudnn_execution_plan_impl.
Layer 3 – Graph::deserialize (graph_interface.h)
• New deserialize(std::vector<uint8_t> const&, bool enforce_precompiled)
overload; routes through deserialize_plan_impl with device_properties
when no handle is given.
• Existing handle overloads explicitly pass device_prop=nullptr so the
graph's stored device_properties is not consulted (Decision 7).
Layer 4 – Python bindings
• pygraph.h/.cpp: store device_properties in PyGraph; select devprop vs
handle path in PyGraph::deserialize.
• _pygraph.py: fresh-container path now forwards name, kernel_cache,
device_property, and handle only (not datatypes) to avoid changing
classic-deserialize behaviour for existing callers.
Layer 5 – Tests, sample, docs
• test/cpp/serialize.cpp: 14 test cases including the wrong-arch
discriminator (proves devprop is load-bearing), RUNTIME_COMPILATION
behavior-note assertion, enforce_precompiled, and no-devprop guard.
• test/cpp/check_skip_json_lib.cpp: compile-only check for
CUDNN_FRONTEND_SKIP_JSON_LIB builds.
• test/python/test_deviceless_aot_compilation.py: 3 tests including
wrong-arch discriminator, numerical correctness oracle, and
cudnn.backend_version() >= 90800 hard assertion.
• samples/cpp/misc/deviceless_aot_compilation.cpp: updated sample.
• docs/deviceless-ahead-of-time-compilation.md: distinguish handle-free
deserialization from device-free operation (RTC rehydration still
requires a compatible GPU/CUDA environment; execution requires a handle).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…yGraph Two correctness fixes found during adversarial review: 1. pygraph.h: The original constructor used `else if (handle_.has_value())` under the device_properties branch, silently discarding any explicitly supplied handle whenever a DeviceProperties was also provided. Change to an independent `if` (with `else if (device_properties == nullptr)` guarding owned-handle creation) so that callers who supply both — e.g. device_properties for handle-less deserialize and a handle for later execution — do not lose the handle. 2. plans.h: Add an explicit null guard at the top of the build_plans(shared_ptr<const DeviceProperties>, string) overload so that a null device_properties passed directly to that function returns a clean ATTRIBUTE_NOT_SET error rather than propagating into loadFromJson and emitting a confusing 'set either handle or devprop' message. All C++ serialize tests (641 assertions in 14 test cases) and Python deviceless tests (3 passed) continue to pass after these changes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tp5uiuc
marked this pull request as draft
August 10, 2026 23:22
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughThe PR adds handleless UBJSON plan deserialization with configured device properties. It updates execution-plan construction, C++ and Python graph integration, sample code, documentation, and serialization tests. ChangesDeviceless AOT deserialization
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Application
participant Graph
participant ExecutionPlanBuilder_v8
participant cuDNN
Application->>Graph: Deserialize binary plan with DeviceProperties
Graph->>ExecutionPlanBuilder_v8: Load serialized plan
ExecutionPlanBuilder_v8->>cuDNN: Set device-property descriptor
cuDNN-->>ExecutionPlanBuilder_v8: Return execution plan
Application->>Graph: Execute with cuDNN handle
Graph->>cuDNN: Execute plan
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
tp5uiuc
commented
Aug 11, 2026
Comment on lines
+40
to
+43
| Note: the overload accepts `std::vector<uint8_t>` specifically. Passing `std::vector<char>`, | ||
| `std::string`, or a string literal converts via nlohmann's implicit constructor and enters the | ||
| structural-graph `deserialize(json)` overload instead. | ||
|
|
Collaborator
Author
There was a problem hiding this comment.
This is a bit risky from an API perspective. I considered the graph->deserialize(device_properties, data) API as well, but I was unsure if that jives with the cuDNN-FE API as there would be two ways of doing the same thing. i.e. via graph->set_device_properties() and graph->deserialize(...).
If we instead do the following in graph->deserialize(...)
error_t deserialize((std::shared_ptr<const DeviceProperties> devprop, std::vector<uint8_t> const &data, bool const enforce_precompiled = false) {
this->set_device_properties(devprop);
// same rest of the code
}
then we will be overriding previously set device properties.
- docs: remove RTC/rehydration terminology, remove vector<uint8_t> type note; revert wording changes to existing text - include/cudnn_frontend_ExecutionPlan.h: wrap devprop descriptor in a block scope to break consecutive-assignment alignment - python/pygraph/pygraph.cpp: fix inaccurate comment on else branch - samples/cpp/misc/deviceless_aot_compilation.cpp: restore original handle-based step 3; add handle-less devprop path as new step 4 - test/cpp/check_skip_json_lib.cpp: remove file - test/cpp/serialize.cpp: remove DeviceBuf struct and all test sections that call graph.execute(); trim RTC section to behavior-note check only Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the post-hoc behavior-note check with an explicit
select_behavior_notes({RUNTIME_COMPILATION}) call before build, so
the test guarantees an RTC engine was selected rather than hoping the
heuristic picks one.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tp5uiuc
marked this pull request as ready for review
August 11, 2026 02:55
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
test/python/test_deviceless_aot_compilation.py (1)
176-178: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy liftExpose the handle-less state in the test.
The current binding does not expose
handleoris_handle_owner, so an eager internal handle allocation would not fail this test. Add a binding-visible test seam or an equivalent assertion that detects handle creation forpygraph(device_property=dp).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/python/test_deviceless_aot_compilation.py` around lines 176 - 178, Update the test around graph_deser.deserialize and pygraph(device_property=dp) to assert the deserialized graph remains handle-less, using a binding-visible handle or ownership-state seam (or an equivalent observable assertion). Ensure an eagerly allocated internal cuDNN handle causes the test to fail.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/deviceless-ahead-of-time-compilation.md`:
- Around line 31-33: Update the example around the cuDNN handle creation and
graph_deser->execute call: validate the result of cudnnCreate, capture the
execution status from execute, destroy handle afterward, then require the
captured status.
In `@include/cudnn_frontend_ExecutionPlan.h`:
- Around line 329-340: Update the loadFromJson execution-plan attribute
selection around m_execution_plan.handle so CUDNN_ATTR_EXECUTION_PLAN_DEVICEPROP
is used only when device properties are configured and the handle is nullptr;
preserve the explicit handle path when both are present. Revise
setDeviceProperties documentation to state that device properties apply only
without a handle, and that an explicit handle takes precedence.
In `@test/python/test_deviceless_aot_compilation.py`:
- Around line 166-167: Update the pytest.raises assertion around
graph_wrong.deserialize(blob) to accept only cudnn.cudnnGraphNotSupportedError
or RuntimeError, and require the exception message to match the
architecture-mismatch text.
---
Nitpick comments:
In `@test/python/test_deviceless_aot_compilation.py`:
- Around line 176-178: Update the test around graph_deser.deserialize and
pygraph(device_property=dp) to assert the deserialized graph remains
handle-less, using a binding-visible handle or ownership-state seam (or an
equivalent observable assertion). Ensure an eagerly allocated internal cuDNN
handle causes the test to fail.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 240009b0-1021-4821-874d-d9f3ec748c87
📒 Files selected for processing (10)
docs/deviceless-ahead-of-time-compilation.mdinclude/cudnn_frontend/graph_interface.hinclude/cudnn_frontend/plans.hinclude/cudnn_frontend_ExecutionPlan.hpython/cudnn/_pygraph.pypython/pygraph/pygraph.cpppython/pygraph/pygraph.hsamples/cpp/misc/deviceless_aot_compilation.cpptest/cpp/serialize.cpptest/python/test_deviceless_aot_compilation.py
- ExecutionPlan.h: guard CUDNN_ATTR_EXECUTION_PLAN_DEVICEPROP path with m_execution_plan.handle == nullptr; update doc comment to match - docs: validate cudnnCreate, destroy handle after execute in C++ snippet - test/python: narrow pytest.raises(Exception) to (cudnnGraphNotSupportedError, RuntimeError) for wrong-arch test Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Collaborator
Author
|
@cudnn-ci-bot run |
Only allowlisted maintainers can use |
Anerudhan
added
mod-frontend
Collaborator
|
@cudnn-ci-bot backend |
|
cuDNN CI bot commands
Only allowlisted maintainers can use |
Member
|
@cudnn-ci-bot run backend |
|
🚀 Running mirror pipeline Branch: cudnn-gh/pr-544-92fc82d |
hwanseoc
requested changes
Aug 12, 2026
hwanseoc
left a comment
hwanseoc
left a comment
Member
There was a problem hiding this comment.
CI passed and is clean.
Also built on a B200 and ran
tests "[serialize]"— 14 cases- the serialization samples (incl.
Deviceless compilation) — 4 cases test_deviceless_aot_compilation.py+test_kernel_cache.py
against cuDNN 9.24 (C++) / 9.20 (Python) and all seems clean.
Requires two oneline changes but should be good after these are addressed.
…t.raises, drop unused import - ExecutionPlan.h: add `&& m_execution_plan.handle == nullptr` to the pre-9.8 #else branch (line 529) to mirror the >= 9.8 condition at line 503; when a handle is also set, fall through to the handle path instead of erroring - test_deviceless_aot_compilation.py: narrow pytest.raises to cudnn.cudnnGraphNotSupportedError with match="NOT_SUPPORTED"; drop RuntimeError catch-all and unused `import re` Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
hwanseoc
self-requested a review
August 12, 2026 19:54
hwanseoc
approved these changes
Aug 12, 2026
Member
|
@cudnn-ci-bot run backend |
|
🚀 Running mirror pipeline Branch: cudnn-gh/pr-544-4786383 |
This was referenced Aug 12, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before submitting
pre-commit runand committed any formatting changes.cat-*, one or moremod-*, and oneorig-*(see label list).Affected area
Summary
Add a
Graph::deserialize(blob)overload (andpygraph.deserialize(blob)in Python) that rehydrates a serialized execution plan using aDevicePropertiesdescriptor instead of acudnnHandle_t. Requires cuDNN ≥ 9.8 at compile and runtime (CUDNN_ATTR_EXECUTION_PLAN_DEVICEPROP); the API compiles on older headers but returns an error at runtime.Why
Enables ahead-of-time compilation: a plan can be built and serialized on a GPU node at build time and deserialized later in a context where no CUDA context or cuDNN handle has been created yet — only a serialized description of the target device's properties is needed.
Related issues
None.
API and compatibility impact
New, additive only. No existing overload is changed.
Graph::set_device_properties(shared_ptr<const DeviceProperties>)andGraph::deserialize(vector<uint8_t>, enforce_precompiled=false).pygraph(device_property=dp)+pygraph.deserialize(blob)(no handle argument).enforce_precompiled=truerejects blobs that carry no precompiled plan (cudnn_backend_dataabsent).Testing
Tested on L40S, cuDNN 9.20.0, WHEEL lane (
nvidia-cudnn-cu13).Summary by CodeRabbit
New Features
Bug Fixes
Documentation