check status before proceeding to add plan to map - #4
Closed
eqy wants to merge 1 commit into
Closed
Conversation
Anerudhan
reviewed
Jun 5, 2021
zkyue
added a commit
to zkyue/cudnn-frontend
that referenced
this pull request
Jul 18, 2026
Completes the remove/relocate edits from the code review that the inline suggestion blocks could not express on their own: - NVIDIA#2: remove the now-duplicate same_hdim t2r_dKV4 wait before dKV1 - NVIDIA#4: remove the now-duplicate same_hdim t2r_dKV01 wait before dKV3 - NVIDIA#6: switch same_hdim dKV2/dKV3 drain to split T2R + reduce_dKV_from_reg Also repairs line-drift from batch-applying the earlier suggestions (their anchors shifted after the first suggestion inserted lines): - NVIDIA#3: remove a leftover dangling `if cutlass.const_expr(not self.same_hdim_kv):` with no body (IndentationError) left because the original lines were not replaced - NVIDIA#5: remove the leftover old dKV0/dKV1 if/else block that sat before the new unified block, which would otherwise double the reduce/atomic_add and the t2r_dKV01 barrier arrival Final state is the intended split-T2R + signal drain refactor: every reduce site loads dKV into registers, fences, signals the MMA warp that the TMEM columns are free, then does the global atomic_add; same_hdim and not-same_hdim now follow the same protocol. dq is bitwise-identical to ef50ddc; head_dim 576 and 512 both verified race-free (compute-sanitizer racecheck 0).
zkyue
added a commit
to zkyue/cudnn-frontend
that referenced
this pull request
Jul 20, 2026
Completes the remove/relocate edits from the code review that the inline suggestion blocks could not express on their own: - NVIDIA#2: remove the now-duplicate same_hdim t2r_dKV4 wait before dKV1 - NVIDIA#4: remove the now-duplicate same_hdim t2r_dKV01 wait before dKV3 - NVIDIA#6: switch same_hdim dKV2/dKV3 drain to split T2R + reduce_dKV_from_reg Also repairs line-drift from batch-applying the earlier suggestions (their anchors shifted after the first suggestion inserted lines): - NVIDIA#3: remove a leftover dangling `if cutlass.const_expr(not self.same_hdim_kv):` with no body (IndentationError) left because the original lines were not replaced - NVIDIA#5: remove the leftover old dKV0/dKV1 if/else block that sat before the new unified block, which would otherwise double the reduce/atomic_add and the t2r_dKV01 barrier arrival Final state is the intended split-T2R + signal drain refactor: every reduce site loads dKV into registers, fences, signals the MMA warp that the TMEM columns are free, then does the global atomic_add; same_hdim and not-same_hdim now follow the same protocol. dq is bitwise-identical to ef50ddc; head_dim 576 and 512 both verified race-free (compute-sanitizer racecheck 0). Signed-off-by: zky <51477259+zkyue@users.noreply.github.com>
Anerudhan
pushed a commit
that referenced
this pull request
Jul 20, 2026
…m 576) (#396) * Fix latent TMEM WAR race in DSA backward SM100 dKV drain (head_dim 576 path) In the not-same_hdim_kv path (head_dim 576 / head_dim_v 512), the reduce warps read dKV2/dKV3 from TMEM inside store_dKV, after their t2r_dKV4_done arrive. The MMA warp's next-iteration dKV0/dKV1 gemms overwrite the same TMEM columns (tmem_dKV2_offset == tmem_dKV0_offset, tmem_dKV3_offset == tmem_dKV1_offset) with no happens-before edge to those reads: mma_reduce_dKV_pipeline has 2 stages, so the part1 producer_acquire of generation 3i+3 only orders against the consumer_release of generation 3i+1 (the dKV4 generation), and no named barrier separates part2's TMEM reads (generation 3i+2) from the issue of the next part1's overwrites. Correctness currently depends on the tensor core still draining queued work when the reduce warps issue their loads. The window reproduces deterministically under adversarial delay: a 1 us spin inserted in the reduce warps between the two part2 store_dKV calls (no other change) corrupts dkv columns [384:512) on every run at S=2048 H=64 D=576 topk=512 bf16 (rel L2 vs fp32 autograd 1.0e0 vs 5.6e-3 baseline); a 10 us spin before both calls corrupts [256:384) as well. Delays at already-synchronized points (after part1's register-staged T2Rs, or before the MMA warp's dKV0 issue) never corrupt. Fix, mirroring part1's existing register-staging pattern: T2R dKV2/dKV3 into registers, fence, arrive on a new t2r_dKV23_done named barrier (id 9), and only then run the global-memory atomic reduction; the MMA warp waits on that barrier before issuing the next iteration's dKV0 (skipped on the first iteration, balanced by a final arrive after the loop, like t2r_dKV4_done). Register peak is unchanged (part1 already holds two fragments). The same_hdim_kv (head_dim 512) path is untouched and compiles identically. Measured on B200 (S=8192 H=64 topk=1024 bf16, ABAB paired): head_dim 576: 5.398 -> 5.840 ms median (+8.0%), the cost of no longer overlapping the next iteration's dKV issue with the dKV2/dKV3 readback; head_dim 512: +0.03% (noise). dq stays bitwise-identical to develop on both head dims; dkv/d_sink match the fp32 autograd reference at the baseline rel L2 on both, and the adversarial-delay matrix that trips develop is clean on the fixed kernel. Signed-off-by: zky <51477259+zkyue@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Jie Fang <jief@nvidia.com> Signed-off-by: zky <51477259+zkyue@users.noreply.github.com> * Complete review remove/relocate edits and repair suggestion line-drift Completes the remove/relocate edits from the code review that the inline suggestion blocks could not express on their own: - #2: remove the now-duplicate same_hdim t2r_dKV4 wait before dKV1 - #4: remove the now-duplicate same_hdim t2r_dKV01 wait before dKV3 - #6: switch same_hdim dKV2/dKV3 drain to split T2R + reduce_dKV_from_reg Also repairs line-drift from batch-applying the earlier suggestions (their anchors shifted after the first suggestion inserted lines): - #3: remove a leftover dangling `if cutlass.const_expr(not self.same_hdim_kv):` with no body (IndentationError) left because the original lines were not replaced - #5: remove the leftover old dKV0/dKV1 if/else block that sat before the new unified block, which would otherwise double the reduce/atomic_add and the t2r_dKV01 barrier arrival Final state is the intended split-T2R + signal drain refactor: every reduce site loads dKV into registers, fences, signals the MMA warp that the TMEM columns are free, then does the global atomic_add; same_hdim and not-same_hdim now follow the same protocol. dq is bitwise-identical to ef50ddc; head_dim 576 and 512 both verified race-free (compute-sanitizer racecheck 0). Signed-off-by: zky <51477259+zkyue@users.noreply.github.com> --------- Signed-off-by: zky <51477259+zkyue@users.noreply.github.com> Co-authored-by: Jie Fang <jief@nvidia.com>
1 task
YangXu1990uiuc
added a commit
to YangXu1990uiuc/cudnn-frontend
that referenced
this pull request
Aug 17, 2026
…d ints cudnn.create_handle() is the only way to make a handle in the Python API, so every real caller already holds a first-class Handle (verified across flashinfer / sglang / the FE's own code; torch uses the C++ frontend, not this module). A raw backend int silently opted out of the Handle's device/stream tracking and device-scoped build, so keeping it as an equal citizen was a second, incompatible concept on every handle API. to_backend_handle / set_stream / get_stream / destroy_handle / execute(handle=) now require a cudnn.Handle (or None) and raise TypeError on a bare int. A framework holding a foreign cudnnHandle_t wraps it once -- cudnn.Handle(backend_handle, ordinal, stream) -- so it becomes first-class instead of a bare int. deserialize keeps its classic (handle, data) vs (data) overload by unwrapping only a Handle and leaving the blob alone. Fixes a stale handle:int annotation on execute() and a duplicate return in destroy_handle. Design doc Hard-constraint NVIDIA#4 updated; the raw-int unit tests now assert rejection. Verified: test_set_stream_cache + test_dispatch (64 passed), and a real create_handle -> build -> execute on GPU (rel-L2 1.6e-3, raw int rejected, destroy clears). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
YangXu1990uiuc
added a commit
that referenced
this pull request
Aug 17, 2026
…kend handle, device, stream}) (#612) * Make cudnn.set_stream idempotent: skip the backend call when the stream is unchanged cudnnSetStream is not free. For a non-null stream, cudnn::ops::SetStream (backend src/graph/src/context.cpp) issues several CUDA driver queries on EVERY call — green-context detection (cuStreamGetGreenCtx), cudaStreamGetPriority, cudaDeviceGetStreamPriorityRange, plus a cudaEventRecord device check when the stream changes — to maintain cuDNN's internal per-priority / per-green-context stream pool. It does this even when the stream has not changed (there is no unchanged-stream early return). On Blackwell that is ~2.4us/call (measured), and a framework that calls set_stream before every execute pays it every iteration. Cache the last stream per handle in the Python layer and skip the backend call when it is unchanged, so a steady-state single-stream loop pays it once. destroy_handle forgets the entry so a reused handle address is not wrongly skipped. Assumes a handle is not driven from two streams concurrently (the normal single-stream case; a caller that does needs its own handle per stream regardless). This closes most of the per-op host-overhead gap between routing a plain GEMM through cuDNN and calling cuBLAS directly (the cudnn backend execute itself is already at cuBLAS parity). A complementary backend fix — an unchanged-stream early return in SetStream — would help all callers (including framework code that calls cudnnSetStream directly); filed separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Skip the discarded per-call context in graph.execute() execute() built a caller ExecutionContext (a cudnnGetStream round-trip + an object alloc) at the top of every call, but only used it when the plan was not yet built. In steady state the plan is built, so the context was computed and thrown away on every execute — a ~2.9us tax that made execute() slower than execute_plan_at_index() for the identical plan. Move the context build inside the `not _is_built` branch, where it is the only user. No API/behavior change; the JIT-build path still gets the caller's handle/stream. On SM100, 256^3 bf16 single-plan matmul this closes the whole execute()-vs-execute_plan_at_index() gap (16.3 -> 10.5 us), matching execute_plan_at_index; test_matmul_bias_relu 34 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Make cudnn.create_handle() return a first-class Handle The backend cudnnHandle_t binds a device and carries the stream, but on the FE side the handle was a bare int with nowhere to hang per-handle state, so that state accreted as side tables (the _handle_to_stream dict) and per-engine device queries (frost's current_device()). create_handle() now returns a cudnn.Handle owning {backend_handle, device, stream}. The naming anticipates the front end being "cudnn" and today's cuDNN becoming "cudnn backend": this object is the handle; the wrapped cudnnHandle_t is its backend_handle. - The backend handle is handed to C++ EXPLICITLY -- to_backend_handle(h) at the named handoffs (_execute*, backend_graph) and unwrap_handles(args, kwargs) at the opaque passthroughs (get_workspace_size, cuda-graph, deserialize). A reader can grep `backend_handle` and trace the plumbing top-to-bottom without an IDE. An inventory confirmed every handle->C++ handoff is in _pygraph/__init__ (the __getattr__ delegation carries no handle), so the set is closed. Handle has NO __index__: the only path to the backend is those explicit calls, and a Handle reaching a binding unconverted fails loudly. No C++ binding changes. - Dunders are minimal (no int coercion; __eq__/__hash__/__bool__ at object defaults) so the handle stays a valid dict key, stays truthy in `if handle:`, and does not raise on wrapper.py's `== 'auto'`. - stream lives on Handle.stream (absorbing the write-only _handle_to_stream cache); get_stream() reads it with no cudnnGetStream round-trip, which also removes the live query _resolve_stream did on every python-engine execute. - device is a lazy DeviceInfo (compute_capability + packed sm_version, sm_count, smem-optin, oversized-smem, L2, name) sourced from the frost driver introspector and cached per ordinal -- one device-info surface for the FE. Foreign raw-int handles (framework-created via the C API) keep working: stream falls back to the _handle_to_stream registry keyed by int(handle), and a live cudnnGetStream. Design + call-site inventory in docs/handle_first_class_design.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * frost: fix oversized-SMEM query crashing all frost GEMM on CUDA<13.4 cuda-python oversized_shared_memory_per_block() passed a bare attribute ordinal (150, CU_DEVICE_ATTRIBUTE_MAX_OVERSIZED_SHARED_MEMORY_PER_BLOCK, added in CUDA 13.4) to cuda-python's cuDeviceGetAttribute. That binding is strongly typed on the attribute -- it reads attrib.value -- so a bare int (for an enum member the installed cuda-python does not carry; 13.0.2 tops out at 148) raises "'int' object has no attribute 'value'". The query is on the tile-selection hot path (_sm_smem_budget_bytes_of), so this one call took down every frost GEMM kernel: on develop tip the frost gemm suite is 5641 failed / 163 passed, all with that single signature; the query was introduced in #593. Gate on the driver's CUDA version instead: the attribute arrived in 13.4, so a driver older than that has no such mode -> 0 by design (not an error), and the enum member -- which an older cuda-python lacks -- is never touched. From 13.4 the attribute is real, so query it via the proper enum and let a genuine failure raise rather than masking it as 0. This keeps "expected absence" (below 13.4) distinct from an unexpected driver error, and needs no ctypes / bare-ordinal workaround. Validated: frost gemm suite 5804 passed / 0 failed after the fix (was 5641 failed / 163 passed); test_public_execute_flavors.py 30 passed on py3.12 (fe-jax, driver 13.2 -> returns 0). Build-time + lru_cached: 0.38us first call, 50ns cached, never on the execute path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * frost(gemm): bake the plan for the handle's device, not the ambient one The FROST GEMM engine read every device-derived kernel constant (arch, ab_stages, grid_num_clusters, sm_count, the SMEM/L2 budgets) off frost.device.current_device() -- a workaround from when the graph handed it no device. With a first-class cudnn.Handle carrying a device, source them from the handle instead, so a plan is baked for the GPU the handle is on rather than whatever CUDA device happens to be current at build time. Every one of those constants already funnels through current_device() / resolve_device(None), so rather than thread an ordinal through ~20 signatures, scope it once: build_device(ordinal) is a context manager (like torch.cuda.device()) that overrides current_device() for the build. FrostGemmEngine .build_plan wraps build_gemm_plan() in `with build_device(ctx.handle.device.ordinal)`. tile_config._sm_count() -- the one query that bypassed current_device() (it used torch.cuda.current_device) -- is re-routed through frost.device so it honours the scope too. Grep `build_device`/`_build_device` to trace it end to end. _check_plan_device is unchanged: it is the EXECUTE-time launch guard and must read the LIVE current device (where the launch is going) vs the baked device; the override is a build scope only, unset at execute. VariantPack.device likewise stays on the live device (operand views, read at execute). Validated: frost gemm test_public_execute_flavors + test_stream_respect 32 passed (no regression, SM100); test_build_device.py scopes a build to a different-arch real GPU (L40S/H100/A100) and asserts every constant reports that device -- the multi-GPU behaviour proven on parley without two Blackwells. Foreign raw-int handles (or none) carry no device -> None -> classic current-device. Follow-up: same hinge wrap for the linear-attention and sdpa frost engines (they also read buffers.current_sm()/current_device_id() at build). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * device: extract the device-fact layer to a common cudnn._device.DeviceInfo Frost had its own device introspection (frost/device.py querying the driver for compute capability, SM count, the SMEM/L2 ceilings, ...), a workaround from when the FE had no device concept to hand it. Move that layer up to a common cudnn._device.DeviceInfo: each fact is a @cached_property queried from the driver once and cached ON the instance, with one instance per CUDA ordinal (device_info(ordinal), lru-cached), so a GPU's facts are asked for once and shared. Handle.device is that object. This inverts the direction: the driver queries used to live in frost and DeviceInfo (Handle.device) delegated down to them; now the common layer owns the queries + cache, and frost/device.py's fact functions become thin shims onto device_info(ordinal). Frost's ~24-file / 65-site call surface is unchanged (still frost.device.compute_capability(ord)), but it now reads the same DeviceInfo the handle exposes -- one device concept, not a per-engine introspection stack. A later step can repoint those sites at handle.device.* directly; this ownership move is the enabling half. frost/device.py keeps only its runtime concerns (current_device / build_device / resolve_device / device_context), importing the driver machinery from cudnn._device. Validated: handle.device facts + frost shims read the same instance; cache lives on the DeviceInfo instance (test_device_info.py, 3 passed); Handle-core set_stream 4 + matmul/conv/rope 41; build_device cross-device redirect 3; frost gemm test_public_execute_flavors + test_stream_respect 32 -- all pass, no regression. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Give the passthrough methods explicit signatures; drop unwrap_handles get_workspace_size, get_workspace_size_plan_at_index and populate/update_cuda_graph were (*args, **kwargs) passthroughs, which meant the handle could be at any position -- so the handle->backend conversion had to scan every arg (unwrap_handles). The C++ overloads are just optional trailing args, so ONE explicit Python signature per method (handle=None, override_uids/shapes/strides=None) forwards to them with no duplication, and the handle is unwrapped by name via to_backend_handle(). The methods now self-document and a reader can see exactly where the backend handle is extracted. (test_api_signature_parity only covers __init__/tensor, so these are free to make explicit.) deserialize is the one genuinely ambiguous classic overload -- (data) vs (handle, data, enforce_precompiled) -- so it stays a passthrough, unwrapping just its first positional (the only place a handle can be; to_backend_handle is a no-op on the data blob). unwrap_handles is removed. Validated: test_deviceless_aot_compilation (deserialize, positional handle) + set_stream + device_info + matmul 44 passed; build_device 3; frost gemm 32. The 4 test_block_scale_quantize_dynamic_shape failures are pre-existing (the override-shape backend feature needs cuDNN >= 9.21; the local .so is 9.20) -- identical on HEAD. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * frost(linear-attention): bake gdn/gdn2/kda plans for the handle's device Like the frost GEMM engine, the GDN/GDN2/KDA engines read their one device-baked constant -- num_sm, for the split-K work distribution -- off the ambient device (multiprocessor_count(current_device_id()), a buffers probe that bypasses the build-device scope). Wrap each build_plan in `with build_device(ctx.handle.device .ordinal)` and re-route num_sm onto frost.device.current_device(), so the plan is sized for the handle's GPU rather than whatever CUDA device is current at build. The sdpa frost engines need no such change: their build bakes no device constant from current_device (arch gating lives in check_support), and the lone torch.cuda.current_device() tags a TensorDesc's operand device, which is correctly the live device (as VariantPack.device is). Validated: test_la.py 359 passed / 462 skipped / 0 failed on SM100 (frost opted in) -- no regression across the gdn/gdn2/kda forward + backward engines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * env: single owner for CUDA driver/runtime versions (cudnn/_env.py) Version facts are process-global, not per-device: the installed driver and the linked runtime each have one version for the whole process regardless of which GPU a handle is bound to. They had accreted as re-queries in each consumer -- the DeviceInfo oversized-SMEM gate re-called cuDriverGetVersion, and the cutile GDN/KDA check_support each re-implemented a cudaRuntimeGetVersion probe + version gate. Collect them into cudnn/_env.py. This mirrors the backend convention: cuDNN exposes its own versions as argument-less globals (cudnnGetVersion/cudnnGetCudartVersion), never off a handle or the DEVICEPROP descriptor. cuDNN's own version stays there (cudnn.backend_version()); _env owns only the CUDA-side versions that were otherwise duplicated. This is the environment tier below the per-ordinal DeviceInfo and the per-handle Handle -- a process-global fact placed on either would be duplicated per ordinal / per handle. - driver_version() replaces the inline cuDriverGetVersion in the DeviceInfo oversized-SMEM gate. - runtime_version() replaces the duplicated cudaRuntimeGetVersion + gate in the cutile GDN/KDA engines; the decline outcome is unchanged (an unavailable runtime declines exactly like a too-old one). ~100 ns and off the execute hot path, so the lru_cache is for a single owner returning a stable constant, not for speed (measured: raw cuda-python query ~110 ns, cache hit ~43 ns -- invisible against a build/compile path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * frost(linear-attention): bake kernel num_sm for the build-scope device The GDN/GDN2/KDA kernels bake num_sm / max_active_clusters (the persistent grid size) as a compile-time constant. They read it from buffers.current_device_id() -- a raw cudaGetDevice that does NOT honour the build_device() scope -- while the engine (tier 1) and the tile/arch codegen (tier 2) read it through frost.device.current_device(), which does. So inside a build_device(A) scope while the process is live on B, the engine and tiles bake for A but the kernel grid bakes for B: one build, two GPUs. Point the 9 kernel sites at frost.device.current_device() so all three tiers follow the one handle-sourced scope. Same value in the common case (build device == current device); consistent under a cross-GPU build scope. _check_plan_device stays the execute-time launch guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * handle: fix stale create_handle docstring (no __index__) The docstring still described the handle forwarding to the backend "via __index__". That path was dropped for the explicit to_backend_handle() handoff -- Handle deliberately has no int-coercing dunder, so an unconverted Handle reaching a binding fails loudly. Correct the docstring to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * execute: build the per-execute ExecutionContext ~220 ns cheaper The python-engine execute path builds an ExecutionContext every call. Two stateless trims to that hot path -- no caching / invalidation surface: - ExecutionContext was a frozen dataclass; its generated __init__ sets each of the three fields through object.__setattr__ (the immutability tax), ~387 ns. A NamedTuple is equally immutable (an engine still cannot rebind ctx.stream) but constructs in ~242 ns for the same three read-only fields. Nothing treats the ctx as a dataclass (no replace()/fields()/is_dataclass), and it is never compared or hashed, so the switch is transparent to engines. - _resolve_stream re-ran `import cudnn` on every call (~69 ns for the sys.modules re-lookup); hoist it to a module-level import (already safe -- _pygraph does `from cudnn import _pybind_module` at module scope, and cudnn.get_stream is resolved at call time, not import time). _build_context: 647 -> 427 ns/execute. With the Handle stream-resolve and the removed discarded rebuild, the python-engine execute path now saves ~4.0 us vs develop. Measured on parley (host timing). Validated: frost gemm 5805 passed / 0 failed, test_la 359 / 0 (SM100). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * handle: address codex + CodeRabbit review findings - Seed Handle.stream from the backend's actual stream at create_handle (a fresh handle runs on stream 0). It was None, so a python plan resolved the stream to torch's current while a backend plan on the same handle ran on stream 0 -- divergent ordering under a non-default current stream. - destroy_handle clears Handle.backend_handle after destruction, so a double-destroy or a later set_stream cannot hand a released cudnnHandle_t back to C++. __init__ accepts a None backend handle; __repr__ renders it. - Do not cache the stream for a foreign raw-int handle: its owner may call cudnnSetStream out-of-band, so a cached "unchanged" skip could leave the wrong stream. The idempotency fast path stays only on Handle (handle.stream); the _handle_to_stream registry is removed. Foreign destroy_handle forwards to the backend (the classic destroy-destroys contract). - Gate the oversized-SMEM attribute on binding support, not just driver version: a CUDA 13.4+ driver with an older cuda-python has driver support but no CUdevice_attribute enum member, which raised AttributeError instead of the intended 0 fallback. getattr(...) is None now short-circuits too. - deserialize unwraps a Handle passed as the handle_ keyword, not just the first positional (the pybind overload names the arg handle_). - Restore create_handle/destroy_handle/get_stream/set_stream to cudnn.__all__ (they moved from pybind symbols to Python wrappers and fell out of the export list), and export Handle/DeviceInfo. - Docs: _handle.py no longer claims __index__ coercion; the design doc notes the properties.cpp binding rename so "no .cpp changes" -> "the C++ handle ABI is unchanged". test_set_stream_cache rewritten for the new semantics (Handle-only idempotency, foreign always-set, destroy clears the backend handle, double-destroy safe). Validated (SM100): set_stream 5, device_info 3, matmul/conv/rope, deviceless-AOT (create_handle + deserialize), native lowering 15, frost gemm 95, test_la 359. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * frost(gemm): pin cute.compile target to the build_device scope arch The build_device() scope already baked every device-derived frost GEMM constant (ab_stages, grid clusters, sm_count, SMEM/L2 budgets, target SM selection) for the handle's GPU, but the cute.compile TARGET still came from the ambient CUDA device: cutedsl derives it from torch.cuda.get_device_capability() when no --gpu-arch is passed. A build for handle-GPU-A while GPU-B was current therefore baked A's constants into a B-targeted kernel. _frost_compile_options() now pins `--gpu-arch sm_<scope>` into the cute.compile() options string (rendered into the content-hashed source, so a cross-arch kernel can no longer collide in the JIT cache). The pin is honoured on the public nvidia-cutlass-dsl >= 4.7 (frost's CUTEDSL_MIN_VERSION, where compile_and_cache / get_arch_enum consult compile_options.gpu_arch before the env arch) AND on internal RCs. The support probe reuses buffers.cutedsl_too_old so an internal RC's own 0.x numbering is judged new, not old (else a capable internal build would be wrongly disabled). On a public wheel below the floor the option is inert and cutedsl targets an arch captured at import time, which we can neither set nor reliably read; a handle-scoped build there fails loud rather than bake scope constants into a possibly-mis-targeted kernel (an unscoped build makes no cross-device promise and is unchanged). frost declines sub-floor wheels as too-old before reaching here, so the refusal is belt-and-suspenders. frost.device gains ambient_device() (the scope-free live device, the extracted body of current_device()) and build_scope_device() (the active scope ordinal or None, for the fail-loud guard). check_support gating and the linear-attention lazy-compile still read the ambient arch; documented as holes that only diverge on a sub-floor handle-scoped build. Verified on SM100 (cutedsl 4.7): test_matmul bf16 sweep 677 passed / 337 skipped with the pin baked in; compiling one graph as sm_100a and sm_103a both succeed while sm_90a fails in the arch-specific NVVM backend (proving the option reaches the target); the sub-floor fail-loud is unit-checked by forcing the support probe false (scoped build raises, unscoped passes). Also forced through flashinfer's unified GEMM fuzzer on the cudnn backend (this build shimmed into flashinfer's venv): 731 passed / 0 failed / 151 xfailed across bf16/fp8/nvfp4/mxfp4/mxfp8 mm+bmm. Addresses codex review (internal-RC support; import-time-arch fallback). note to self: claude::11323ca1-07bc-4fc4-8ec7-ba95d8f061d8 — "PR #612 first-class Handle + A'" cwd /home/scratch.yanxu_libs/cudnn_frontend · worktree /home/scratch.yanxu_gpu/fe-handle Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * frost(gemm): enforce the cutedsl floor in check_support, like the LA engines FrostGemmEngine.check_support (probe_supported) analysed the graph and picked a template but never checked the cutedsl version, so on a wheel below CUTEDSL_MIN_VERSION (4.7) the GEMM engine still accepted the graph and tried to compile -- unlike the linear-attention engines, which decline a too-old wheel up front. Below the floor that either faults deep in cute or, worse, runs unpinned: surfaced live driving this build through flashinfer's GEMM fuzzer on its pinned cutedsl 4.5.2 with FROST engines on, where the frost plan (no backend knobs) then tripped flashinfer's autotuner. Gate probe_supported on buffers.cutedsl_state() / cutedsl_too_old the same way, so a sub-floor wheel declines to the backend cleanly. Internal RCs pass (cutedsl_too_old judges only the public wheel). This also makes the --gpu-arch target pin from the previous commit always available by compile time, so its sub-floor fail-loud is pure belt-and-suspenders. Verified: declines frost on flashinfer's cutedsl 4.5.2 (the fuzzer config that tripped the autotuner now passes via the backend); no-op on 4.7 where probe_supported still accepts. note to self: claude::11323ca1-07bc-4fc4-8ec7-ba95d8f061d8 — "PR #612 first-class Handle + A'" cwd /home/scratch.yanxu_libs/cudnn_frontend · worktree /home/scratch.yanxu_gpu/fe-handle Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: drop the session-provenance line from the handle design doc Remove the internal note-to-self (session id + absolute local working path) from the published design doc; provenance lives in the PR and git history. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * handle: accept only cudnn.Handle on the Python API, reject raw backend ints cudnn.create_handle() is the only way to make a handle in the Python API, so every real caller already holds a first-class Handle (verified across flashinfer / sglang / the FE's own code; torch uses the C++ frontend, not this module). A raw backend int silently opted out of the Handle's device/stream tracking and device-scoped build, so keeping it as an equal citizen was a second, incompatible concept on every handle API. to_backend_handle / set_stream / get_stream / destroy_handle / execute(handle=) now require a cudnn.Handle (or None) and raise TypeError on a bare int. A framework holding a foreign cudnnHandle_t wraps it once -- cudnn.Handle(backend_handle, ordinal, stream) -- so it becomes first-class instead of a bare int. deserialize keeps its classic (handle, data) vs (data) overload by unwrapping only a Handle and leaving the blob alone. Fixes a stale handle:int annotation on execute() and a duplicate return in destroy_handle. Design doc Hard-constraint #4 updated; the raw-int unit tests now assert rejection. Verified: test_set_stream_cache + test_dispatch (64 passed), and a real create_handle -> build -> execute on GPU (rel-L2 1.6e-3, raw int rejected, destroy clears). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * handle: address review — drop Handle.__slots__, trim the ExecutionContext comment Per @Anerudhan's review: Handle is created once per (device, stream), not on a hot path, so __slots__ buys nothing worth the restriction; and the NamedTuple-vs-dataclass rationale on ExecutionContext is trimmed to one line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
hxbai
pushed a commit
to hxbai/cudnn-frontend
that referenced
this pull request
Aug 18, 2026
…kend handle, device, stream}) (NVIDIA#612) * Make cudnn.set_stream idempotent: skip the backend call when the stream is unchanged cudnnSetStream is not free. For a non-null stream, cudnn::ops::SetStream (backend src/graph/src/context.cpp) issues several CUDA driver queries on EVERY call — green-context detection (cuStreamGetGreenCtx), cudaStreamGetPriority, cudaDeviceGetStreamPriorityRange, plus a cudaEventRecord device check when the stream changes — to maintain cuDNN's internal per-priority / per-green-context stream pool. It does this even when the stream has not changed (there is no unchanged-stream early return). On Blackwell that is ~2.4us/call (measured), and a framework that calls set_stream before every execute pays it every iteration. Cache the last stream per handle in the Python layer and skip the backend call when it is unchanged, so a steady-state single-stream loop pays it once. destroy_handle forgets the entry so a reused handle address is not wrongly skipped. Assumes a handle is not driven from two streams concurrently (the normal single-stream case; a caller that does needs its own handle per stream regardless). This closes most of the per-op host-overhead gap between routing a plain GEMM through cuDNN and calling cuBLAS directly (the cudnn backend execute itself is already at cuBLAS parity). A complementary backend fix — an unchanged-stream early return in SetStream — would help all callers (including framework code that calls cudnnSetStream directly); filed separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Skip the discarded per-call context in graph.execute() execute() built a caller ExecutionContext (a cudnnGetStream round-trip + an object alloc) at the top of every call, but only used it when the plan was not yet built. In steady state the plan is built, so the context was computed and thrown away on every execute — a ~2.9us tax that made execute() slower than execute_plan_at_index() for the identical plan. Move the context build inside the `not _is_built` branch, where it is the only user. No API/behavior change; the JIT-build path still gets the caller's handle/stream. On SM100, 256^3 bf16 single-plan matmul this closes the whole execute()-vs-execute_plan_at_index() gap (16.3 -> 10.5 us), matching execute_plan_at_index; test_matmul_bias_relu 34 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Make cudnn.create_handle() return a first-class Handle The backend cudnnHandle_t binds a device and carries the stream, but on the FE side the handle was a bare int with nowhere to hang per-handle state, so that state accreted as side tables (the _handle_to_stream dict) and per-engine device queries (frost's current_device()). create_handle() now returns a cudnn.Handle owning {backend_handle, device, stream}. The naming anticipates the front end being "cudnn" and today's cuDNN becoming "cudnn backend": this object is the handle; the wrapped cudnnHandle_t is its backend_handle. - The backend handle is handed to C++ EXPLICITLY -- to_backend_handle(h) at the named handoffs (_execute*, backend_graph) and unwrap_handles(args, kwargs) at the opaque passthroughs (get_workspace_size, cuda-graph, deserialize). A reader can grep `backend_handle` and trace the plumbing top-to-bottom without an IDE. An inventory confirmed every handle->C++ handoff is in _pygraph/__init__ (the __getattr__ delegation carries no handle), so the set is closed. Handle has NO __index__: the only path to the backend is those explicit calls, and a Handle reaching a binding unconverted fails loudly. No C++ binding changes. - Dunders are minimal (no int coercion; __eq__/__hash__/__bool__ at object defaults) so the handle stays a valid dict key, stays truthy in `if handle:`, and does not raise on wrapper.py's `== 'auto'`. - stream lives on Handle.stream (absorbing the write-only _handle_to_stream cache); get_stream() reads it with no cudnnGetStream round-trip, which also removes the live query _resolve_stream did on every python-engine execute. - device is a lazy DeviceInfo (compute_capability + packed sm_version, sm_count, smem-optin, oversized-smem, L2, name) sourced from the frost driver introspector and cached per ordinal -- one device-info surface for the FE. Foreign raw-int handles (framework-created via the C API) keep working: stream falls back to the _handle_to_stream registry keyed by int(handle), and a live cudnnGetStream. Design + call-site inventory in docs/handle_first_class_design.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * frost: fix oversized-SMEM query crashing all frost GEMM on CUDA<13.4 cuda-python oversized_shared_memory_per_block() passed a bare attribute ordinal (150, CU_DEVICE_ATTRIBUTE_MAX_OVERSIZED_SHARED_MEMORY_PER_BLOCK, added in CUDA 13.4) to cuda-python's cuDeviceGetAttribute. That binding is strongly typed on the attribute -- it reads attrib.value -- so a bare int (for an enum member the installed cuda-python does not carry; 13.0.2 tops out at 148) raises "'int' object has no attribute 'value'". The query is on the tile-selection hot path (_sm_smem_budget_bytes_of), so this one call took down every frost GEMM kernel: on develop tip the frost gemm suite is 5641 failed / 163 passed, all with that single signature; the query was introduced in NVIDIA#593. Gate on the driver's CUDA version instead: the attribute arrived in 13.4, so a driver older than that has no such mode -> 0 by design (not an error), and the enum member -- which an older cuda-python lacks -- is never touched. From 13.4 the attribute is real, so query it via the proper enum and let a genuine failure raise rather than masking it as 0. This keeps "expected absence" (below 13.4) distinct from an unexpected driver error, and needs no ctypes / bare-ordinal workaround. Validated: frost gemm suite 5804 passed / 0 failed after the fix (was 5641 failed / 163 passed); test_public_execute_flavors.py 30 passed on py3.12 (fe-jax, driver 13.2 -> returns 0). Build-time + lru_cached: 0.38us first call, 50ns cached, never on the execute path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * frost(gemm): bake the plan for the handle's device, not the ambient one The FROST GEMM engine read every device-derived kernel constant (arch, ab_stages, grid_num_clusters, sm_count, the SMEM/L2 budgets) off frost.device.current_device() -- a workaround from when the graph handed it no device. With a first-class cudnn.Handle carrying a device, source them from the handle instead, so a plan is baked for the GPU the handle is on rather than whatever CUDA device happens to be current at build time. Every one of those constants already funnels through current_device() / resolve_device(None), so rather than thread an ordinal through ~20 signatures, scope it once: build_device(ordinal) is a context manager (like torch.cuda.device()) that overrides current_device() for the build. FrostGemmEngine .build_plan wraps build_gemm_plan() in `with build_device(ctx.handle.device.ordinal)`. tile_config._sm_count() -- the one query that bypassed current_device() (it used torch.cuda.current_device) -- is re-routed through frost.device so it honours the scope too. Grep `build_device`/`_build_device` to trace it end to end. _check_plan_device is unchanged: it is the EXECUTE-time launch guard and must read the LIVE current device (where the launch is going) vs the baked device; the override is a build scope only, unset at execute. VariantPack.device likewise stays on the live device (operand views, read at execute). Validated: frost gemm test_public_execute_flavors + test_stream_respect 32 passed (no regression, SM100); test_build_device.py scopes a build to a different-arch real GPU (L40S/H100/A100) and asserts every constant reports that device -- the multi-GPU behaviour proven on parley without two Blackwells. Foreign raw-int handles (or none) carry no device -> None -> classic current-device. Follow-up: same hinge wrap for the linear-attention and sdpa frost engines (they also read buffers.current_sm()/current_device_id() at build). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * device: extract the device-fact layer to a common cudnn._device.DeviceInfo Frost had its own device introspection (frost/device.py querying the driver for compute capability, SM count, the SMEM/L2 ceilings, ...), a workaround from when the FE had no device concept to hand it. Move that layer up to a common cudnn._device.DeviceInfo: each fact is a @cached_property queried from the driver once and cached ON the instance, with one instance per CUDA ordinal (device_info(ordinal), lru-cached), so a GPU's facts are asked for once and shared. Handle.device is that object. This inverts the direction: the driver queries used to live in frost and DeviceInfo (Handle.device) delegated down to them; now the common layer owns the queries + cache, and frost/device.py's fact functions become thin shims onto device_info(ordinal). Frost's ~24-file / 65-site call surface is unchanged (still frost.device.compute_capability(ord)), but it now reads the same DeviceInfo the handle exposes -- one device concept, not a per-engine introspection stack. A later step can repoint those sites at handle.device.* directly; this ownership move is the enabling half. frost/device.py keeps only its runtime concerns (current_device / build_device / resolve_device / device_context), importing the driver machinery from cudnn._device. Validated: handle.device facts + frost shims read the same instance; cache lives on the DeviceInfo instance (test_device_info.py, 3 passed); Handle-core set_stream 4 + matmul/conv/rope 41; build_device cross-device redirect 3; frost gemm test_public_execute_flavors + test_stream_respect 32 -- all pass, no regression. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Give the passthrough methods explicit signatures; drop unwrap_handles get_workspace_size, get_workspace_size_plan_at_index and populate/update_cuda_graph were (*args, **kwargs) passthroughs, which meant the handle could be at any position -- so the handle->backend conversion had to scan every arg (unwrap_handles). The C++ overloads are just optional trailing args, so ONE explicit Python signature per method (handle=None, override_uids/shapes/strides=None) forwards to them with no duplication, and the handle is unwrapped by name via to_backend_handle(). The methods now self-document and a reader can see exactly where the backend handle is extracted. (test_api_signature_parity only covers __init__/tensor, so these are free to make explicit.) deserialize is the one genuinely ambiguous classic overload -- (data) vs (handle, data, enforce_precompiled) -- so it stays a passthrough, unwrapping just its first positional (the only place a handle can be; to_backend_handle is a no-op on the data blob). unwrap_handles is removed. Validated: test_deviceless_aot_compilation (deserialize, positional handle) + set_stream + device_info + matmul 44 passed; build_device 3; frost gemm 32. The 4 test_block_scale_quantize_dynamic_shape failures are pre-existing (the override-shape backend feature needs cuDNN >= 9.21; the local .so is 9.20) -- identical on HEAD. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * frost(linear-attention): bake gdn/gdn2/kda plans for the handle's device Like the frost GEMM engine, the GDN/GDN2/KDA engines read their one device-baked constant -- num_sm, for the split-K work distribution -- off the ambient device (multiprocessor_count(current_device_id()), a buffers probe that bypasses the build-device scope). Wrap each build_plan in `with build_device(ctx.handle.device .ordinal)` and re-route num_sm onto frost.device.current_device(), so the plan is sized for the handle's GPU rather than whatever CUDA device is current at build. The sdpa frost engines need no such change: their build bakes no device constant from current_device (arch gating lives in check_support), and the lone torch.cuda.current_device() tags a TensorDesc's operand device, which is correctly the live device (as VariantPack.device is). Validated: test_la.py 359 passed / 462 skipped / 0 failed on SM100 (frost opted in) -- no regression across the gdn/gdn2/kda forward + backward engines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * env: single owner for CUDA driver/runtime versions (cudnn/_env.py) Version facts are process-global, not per-device: the installed driver and the linked runtime each have one version for the whole process regardless of which GPU a handle is bound to. They had accreted as re-queries in each consumer -- the DeviceInfo oversized-SMEM gate re-called cuDriverGetVersion, and the cutile GDN/KDA check_support each re-implemented a cudaRuntimeGetVersion probe + version gate. Collect them into cudnn/_env.py. This mirrors the backend convention: cuDNN exposes its own versions as argument-less globals (cudnnGetVersion/cudnnGetCudartVersion), never off a handle or the DEVICEPROP descriptor. cuDNN's own version stays there (cudnn.backend_version()); _env owns only the CUDA-side versions that were otherwise duplicated. This is the environment tier below the per-ordinal DeviceInfo and the per-handle Handle -- a process-global fact placed on either would be duplicated per ordinal / per handle. - driver_version() replaces the inline cuDriverGetVersion in the DeviceInfo oversized-SMEM gate. - runtime_version() replaces the duplicated cudaRuntimeGetVersion + gate in the cutile GDN/KDA engines; the decline outcome is unchanged (an unavailable runtime declines exactly like a too-old one). ~100 ns and off the execute hot path, so the lru_cache is for a single owner returning a stable constant, not for speed (measured: raw cuda-python query ~110 ns, cache hit ~43 ns -- invisible against a build/compile path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * frost(linear-attention): bake kernel num_sm for the build-scope device The GDN/GDN2/KDA kernels bake num_sm / max_active_clusters (the persistent grid size) as a compile-time constant. They read it from buffers.current_device_id() -- a raw cudaGetDevice that does NOT honour the build_device() scope -- while the engine (tier 1) and the tile/arch codegen (tier 2) read it through frost.device.current_device(), which does. So inside a build_device(A) scope while the process is live on B, the engine and tiles bake for A but the kernel grid bakes for B: one build, two GPUs. Point the 9 kernel sites at frost.device.current_device() so all three tiers follow the one handle-sourced scope. Same value in the common case (build device == current device); consistent under a cross-GPU build scope. _check_plan_device stays the execute-time launch guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * handle: fix stale create_handle docstring (no __index__) The docstring still described the handle forwarding to the backend "via __index__". That path was dropped for the explicit to_backend_handle() handoff -- Handle deliberately has no int-coercing dunder, so an unconverted Handle reaching a binding fails loudly. Correct the docstring to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * execute: build the per-execute ExecutionContext ~220 ns cheaper The python-engine execute path builds an ExecutionContext every call. Two stateless trims to that hot path -- no caching / invalidation surface: - ExecutionContext was a frozen dataclass; its generated __init__ sets each of the three fields through object.__setattr__ (the immutability tax), ~387 ns. A NamedTuple is equally immutable (an engine still cannot rebind ctx.stream) but constructs in ~242 ns for the same three read-only fields. Nothing treats the ctx as a dataclass (no replace()/fields()/is_dataclass), and it is never compared or hashed, so the switch is transparent to engines. - _resolve_stream re-ran `import cudnn` on every call (~69 ns for the sys.modules re-lookup); hoist it to a module-level import (already safe -- _pygraph does `from cudnn import _pybind_module` at module scope, and cudnn.get_stream is resolved at call time, not import time). _build_context: 647 -> 427 ns/execute. With the Handle stream-resolve and the removed discarded rebuild, the python-engine execute path now saves ~4.0 us vs develop. Measured on parley (host timing). Validated: frost gemm 5805 passed / 0 failed, test_la 359 / 0 (SM100). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * handle: address codex + CodeRabbit review findings - Seed Handle.stream from the backend's actual stream at create_handle (a fresh handle runs on stream 0). It was None, so a python plan resolved the stream to torch's current while a backend plan on the same handle ran on stream 0 -- divergent ordering under a non-default current stream. - destroy_handle clears Handle.backend_handle after destruction, so a double-destroy or a later set_stream cannot hand a released cudnnHandle_t back to C++. __init__ accepts a None backend handle; __repr__ renders it. - Do not cache the stream for a foreign raw-int handle: its owner may call cudnnSetStream out-of-band, so a cached "unchanged" skip could leave the wrong stream. The idempotency fast path stays only on Handle (handle.stream); the _handle_to_stream registry is removed. Foreign destroy_handle forwards to the backend (the classic destroy-destroys contract). - Gate the oversized-SMEM attribute on binding support, not just driver version: a CUDA 13.4+ driver with an older cuda-python has driver support but no CUdevice_attribute enum member, which raised AttributeError instead of the intended 0 fallback. getattr(...) is None now short-circuits too. - deserialize unwraps a Handle passed as the handle_ keyword, not just the first positional (the pybind overload names the arg handle_). - Restore create_handle/destroy_handle/get_stream/set_stream to cudnn.__all__ (they moved from pybind symbols to Python wrappers and fell out of the export list), and export Handle/DeviceInfo. - Docs: _handle.py no longer claims __index__ coercion; the design doc notes the properties.cpp binding rename so "no .cpp changes" -> "the C++ handle ABI is unchanged". test_set_stream_cache rewritten for the new semantics (Handle-only idempotency, foreign always-set, destroy clears the backend handle, double-destroy safe). Validated (SM100): set_stream 5, device_info 3, matmul/conv/rope, deviceless-AOT (create_handle + deserialize), native lowering 15, frost gemm 95, test_la 359. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * frost(gemm): pin cute.compile target to the build_device scope arch The build_device() scope already baked every device-derived frost GEMM constant (ab_stages, grid clusters, sm_count, SMEM/L2 budgets, target SM selection) for the handle's GPU, but the cute.compile TARGET still came from the ambient CUDA device: cutedsl derives it from torch.cuda.get_device_capability() when no --gpu-arch is passed. A build for handle-GPU-A while GPU-B was current therefore baked A's constants into a B-targeted kernel. _frost_compile_options() now pins `--gpu-arch sm_<scope>` into the cute.compile() options string (rendered into the content-hashed source, so a cross-arch kernel can no longer collide in the JIT cache). The pin is honoured on the public nvidia-cutlass-dsl >= 4.7 (frost's CUTEDSL_MIN_VERSION, where compile_and_cache / get_arch_enum consult compile_options.gpu_arch before the env arch) AND on internal RCs. The support probe reuses buffers.cutedsl_too_old so an internal RC's own 0.x numbering is judged new, not old (else a capable internal build would be wrongly disabled). On a public wheel below the floor the option is inert and cutedsl targets an arch captured at import time, which we can neither set nor reliably read; a handle-scoped build there fails loud rather than bake scope constants into a possibly-mis-targeted kernel (an unscoped build makes no cross-device promise and is unchanged). frost declines sub-floor wheels as too-old before reaching here, so the refusal is belt-and-suspenders. frost.device gains ambient_device() (the scope-free live device, the extracted body of current_device()) and build_scope_device() (the active scope ordinal or None, for the fail-loud guard). check_support gating and the linear-attention lazy-compile still read the ambient arch; documented as holes that only diverge on a sub-floor handle-scoped build. Verified on SM100 (cutedsl 4.7): test_matmul bf16 sweep 677 passed / 337 skipped with the pin baked in; compiling one graph as sm_100a and sm_103a both succeed while sm_90a fails in the arch-specific NVVM backend (proving the option reaches the target); the sub-floor fail-loud is unit-checked by forcing the support probe false (scoped build raises, unscoped passes). Also forced through flashinfer's unified GEMM fuzzer on the cudnn backend (this build shimmed into flashinfer's venv): 731 passed / 0 failed / 151 xfailed across bf16/fp8/nvfp4/mxfp4/mxfp8 mm+bmm. Addresses codex review (internal-RC support; import-time-arch fallback). note to self: claude::11323ca1-07bc-4fc4-8ec7-ba95d8f061d8 — "PR NVIDIA#612 first-class Handle + A'" cwd /home/scratch.yanxu_libs/cudnn_frontend · worktree /home/scratch.yanxu_gpu/fe-handle Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * frost(gemm): enforce the cutedsl floor in check_support, like the LA engines FrostGemmEngine.check_support (probe_supported) analysed the graph and picked a template but never checked the cutedsl version, so on a wheel below CUTEDSL_MIN_VERSION (4.7) the GEMM engine still accepted the graph and tried to compile -- unlike the linear-attention engines, which decline a too-old wheel up front. Below the floor that either faults deep in cute or, worse, runs unpinned: surfaced live driving this build through flashinfer's GEMM fuzzer on its pinned cutedsl 4.5.2 with FROST engines on, where the frost plan (no backend knobs) then tripped flashinfer's autotuner. Gate probe_supported on buffers.cutedsl_state() / cutedsl_too_old the same way, so a sub-floor wheel declines to the backend cleanly. Internal RCs pass (cutedsl_too_old judges only the public wheel). This also makes the --gpu-arch target pin from the previous commit always available by compile time, so its sub-floor fail-loud is pure belt-and-suspenders. Verified: declines frost on flashinfer's cutedsl 4.5.2 (the fuzzer config that tripped the autotuner now passes via the backend); no-op on 4.7 where probe_supported still accepts. note to self: claude::11323ca1-07bc-4fc4-8ec7-ba95d8f061d8 — "PR NVIDIA#612 first-class Handle + A'" cwd /home/scratch.yanxu_libs/cudnn_frontend · worktree /home/scratch.yanxu_gpu/fe-handle Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: drop the session-provenance line from the handle design doc Remove the internal note-to-self (session id + absolute local working path) from the published design doc; provenance lives in the PR and git history. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * handle: accept only cudnn.Handle on the Python API, reject raw backend ints cudnn.create_handle() is the only way to make a handle in the Python API, so every real caller already holds a first-class Handle (verified across flashinfer / sglang / the FE's own code; torch uses the C++ frontend, not this module). A raw backend int silently opted out of the Handle's device/stream tracking and device-scoped build, so keeping it as an equal citizen was a second, incompatible concept on every handle API. to_backend_handle / set_stream / get_stream / destroy_handle / execute(handle=) now require a cudnn.Handle (or None) and raise TypeError on a bare int. A framework holding a foreign cudnnHandle_t wraps it once -- cudnn.Handle(backend_handle, ordinal, stream) -- so it becomes first-class instead of a bare int. deserialize keeps its classic (handle, data) vs (data) overload by unwrapping only a Handle and leaving the blob alone. Fixes a stale handle:int annotation on execute() and a duplicate return in destroy_handle. Design doc Hard-constraint NVIDIA#4 updated; the raw-int unit tests now assert rejection. Verified: test_set_stream_cache + test_dispatch (64 passed), and a real create_handle -> build -> execute on GPU (rel-L2 1.6e-3, raw int rejected, destroy clears). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * handle: address review — drop Handle.__slots__, trim the ExecutionContext comment Per @Anerudhan's review: Handle is created once per (device, stream), not on a hot path, so __slots__ buys nothing worth the restriction; and the NamedTuple-vs-dataclass rationale on ExecutionContext is trimmed to one line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.