Repository navigation
Create DNN handles before command buffers capture convolutions - #1387
Open
magaonka-amd wants to merge 1 commit into
Open
magaonka-amd wants to merge 1 commit into
magaonka-amd wants to merge 1 commit into
Conversation
- With CONVOLUTION in command buffers, a convolution's first run is recorded inside a stream capture. On a device that has no MIOpen handle yet, the handle was created there: its hipMalloc is not allowed during capture, and the process aborts. - ConvolutionThunk::Initialize now creates the DNN handle, before any capture. - Creating a handle later can still break other devices' captures: HIP without ROCm/rocm-systems#12274 lets a synchronous call invalidate every capture in the process. The HIP version does not tell whether openxla#12274 is present, so a runtime check decides, once per process. It starts a capture on one device and makes a synchronous call on another. Unless the capture survives, every device's MIOpen handle is created at startup.
magaonka-amd
force-pushed
the
fix/conv-cmdbuf-dnn-handle-init
branch
from
November 2, 2026 16:50
14bb635 to
90f9a89
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Summary of Changes
Create the DNN (MIOpen) handle before command buffers capture convolutions:
ConvolutionThunk::Initializecreates the device's DNN handle.PlatformUtil::GetStreamExecutorsalso creates every device's MIOpen handle atstartup, unless HIP limits capture invalidation to the calling device. A runtime check decides this once per
process (
xla/stream_executor/rocm/rocm_capture_invalidation.{h,cc}).🎯 Justification
With
--xla_gpu_enable_command_buffer=+CONVOLUTION, a convolution's first run is recorded inside a stream capture.A device without a MIOpen handle creates it there. Creating it calls
hipMalloc, which HIP rejects during capture,and the process aborts. This happens on every GPU except the one that ran autotuning, and on a single GPU when the
executable comes from a compilation cache.
Creating the handle in
Initializekeeps it out of the device's own capture. With several GPUs that is not alwaysenough: HIP without ROCm/rocm-systems#12274 lets the handle's synchronous HIP calls invalidate the captures running
on every other device. On such a HIP, all handles are created at startup, after every device is initialized
(inside each device's
Init()it can deadlock in comgr).The HIP version cannot tell whether openxla#12274 is present:
developreported 7.18 before it merged, and it had beenreverted once before. So the check tests the behaviour. It starts a capture on one device and makes a synchronous
call (
hipMemset) on another device from a second thread. It first checks that the same call on the capture's owndevice does invalidate the capture. If that control fails, or anything is inconsistent, the handles are created at
startup.
🚀 Kind of Contribution
🐛 Bug Fix
📊 Benchmark (for Performance Improvements)
Not a performance change. The check runs once per process, at client creation. On a HIP without openxla#12274, each
multi-GPU process creates a MIOpen handle per device at startup, also when it has no convolutions.
🧪 Unit Tests:
convolution_thunk_test:Initializerequests the DNN handle and fails without one.rocm_capture_invalidation_test:current-device change behind.
platform_util_multigpu_test(2 GPUs): afterGetStreamExecutors, device 1 creates its DNN handle while device 0records a command buffer, and the recording succeeds.
🧪 Execution Tests:
MI355X (gfx950), ROCm 10.2, through JAX with
+CONVOLUTION:fix(hip): implement device-scoped invalidation for stream capture rocm-systems#12274, and with the stock runtime reporting version 7.18.
With an empty MIOpen find database, capture also needs ROCm/rocm-libraries#12265 (or
MIOPEN_NAIVE_TIMEOUT=0).