Skip to content

Create DNN handles before command buffers capture convolutions - #1387

Open
magaonka-amd wants to merge 1 commit into
mainfrom
fix/conv-cmdbuf-dnn-handle-init
Open

magaonka-amd wants to merge 1 commit into
mainfrom
fix/conv-cmdbuf-dnn-handle-init

Conversation

@magaonka-amd

@magaonka-amd magaonka-amd commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

📝 Summary of Changes

Create the DNN (MIOpen) handle before command buffers capture convolutions:

  • ConvolutionThunk::Initialize creates the device's DNN handle.
  • With more than one device, PlatformUtil::GetStreamExecutors also creates every device's MIOpen handle at
    startup, unless HIP limits capture invalidation to the calling device. A runtime check decides this once per
    process (xla/stream_executor/rocm/rocm_capture_invalidation.{h,cc}).

🎯 Justification

With --xla_gpu_enable_command_buffer=+CONVOLUTION, a convolution's first run is recorded inside a stream capture.
A device without a MIOpen handle creates it there. Creating it calls hipMalloc, which HIP rejects during capture,
and the process aborts. This happens on every GPU except the one that ran autotuning, and on a single GPU when the
executable comes from a compilation cache.

Creating the handle in Initialize keeps it out of the device's own capture. With several GPUs that is not always
enough: HIP without ROCm/rocm-systems#12274 lets the handle's synchronous HIP calls invalidate the captures running
on every other device. On such a HIP, all handles are created at startup, after every device is initialized
(inside each device's Init() it can deadlock in comgr).

The HIP version cannot tell whether openxla#12274 is present: develop reported 7.18 before it merged, and it had been
reverted once before. So the check tests the behaviour. It starts a capture on one device and makes a synchronous
call (hipMemset) on another device from a second thread. It first checks that the same call on the capture's own
device does invalidate the capture. If that control fails, or anything is inconsistent, the handles are created at
startup.

🚀 Kind of Contribution

🐛 Bug Fix

📊 Benchmark (for Performance Improvements)

Not a performance change. The check runs once per process, at client creation. On a HIP without openxla#12274, each
multi-GPU process creates a MIOpen handle per device at startup, also when it has no convolutions.

🧪 Unit Tests:

  • convolution_thunk_test: Initialize requests the DNN handle and fails without one.
  • rocm_capture_invalidation_test:
    • When the probe fails, every device gets its DNN handle (fake probe, no GPU needed).
    • On a GPU, the probe reports that a same-device call invalidates a capture, and leaves no capture, error or
      current-device change behind.
  • platform_util_multigpu_test (2 GPUs): after GetStreamExecutors, device 1 creates its DNN handle while device 0
    records a command buffer, and the recording succeeds.

🧪 Execution Tests:

MI355X (gfx950), ROCm 10.2, through JAX with +CONVOLUTION:

With an empty MIOpen find database, capture also needs ROCm/rocm-libraries#12265 (or MIOPEN_NAIVE_TIMEOUT=0).

- With CONVOLUTION in command buffers, a convolution's first run is
  recorded inside a stream capture. On a device that has no MIOpen
  handle yet, the handle was created there: its hipMalloc is not allowed
  during capture, and the process aborts.
- ConvolutionThunk::Initialize now creates the DNN handle, before any
  capture.
- Creating a handle later can still break other devices' captures: HIP
  without ROCm/rocm-systems#12274 lets a synchronous call invalidate
  every capture in the process. The HIP version does not tell whether
  openxla#12274 is present, so a runtime check decides, once per process. It
  starts a capture on one device and makes a synchronous call on
  another. Unless the capture survives, every device's MIOpen handle is
  created at startup.
@magaonka-amd
magaonka-amd force-pushed the fix/conv-cmdbuf-dnn-handle-init branch from 14bb635 to 90f9a89 Compare November 2, 2026 16:50

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant