Repository navigation
feat(rocprofiler-sdk): kernel-dispatch completion signals in device memory - #14460
Draft
mjkvaak-amd wants to merge 2 commits into
Draft
mjkvaak-amd wants to merge 2 commits into
mjkvaak-amd wants to merge 2 commits into
Conversation
…emory Kernel tracing gives every dispatch an SDK completion signal in host memory, so with barrier-bit packets (graph replay) each kernel waits on a CP write across PCIe. On the inline interposition path, allocate these signals from the agent's signal-capable pool (hsa_amd_signal_batch_create), never write them from the host, and read timestamps with a single streaming load. rocprofiler_flush_buffer now waits, bounded, for completions already registered so records are not dropped at stop. ROCPROFILER_DEVICE_SIGNALS=0 disables; falls back to host signals when the runtime or agent lacks support. Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
- Device signals only for non-last dispatches of a batch, created HSA_AMD_SIGNAL_AMD_GPU_ONLY with the owning agent as sole consumer (no KFD event, mapped to one GPU). The last dispatch keeps a host signal, so the monitor's wait and completion check read host memory. - Retire a device signal only if it holds idle_value - 1; otherwise warn, read timestamps through ROCr and keep the slot out of the pool. Debug builds check the value on acquire. - Read value and timestamps from one snapshot; cross-check the first raw read against hsa_amd_profiling_get_dispatch_time and fall back to it on disagreement. Null-check the tick conversion function. - Select only GPU-located signal-capable pools and log the choice. - Per-agent device pools in an immutable list: no lock per dispatch. - Flush waits for the batches with device signals registered before it, by id (pending_batches), 1 ms without progress on the device and 100 ms while emitting; only for buffers receiving kernel-dispatch records and only when device signals are in use. - Null-checked teardown with created/destroyed accounting. - Unit tests for pending_batches and the device signal source; document ROCPROFILER_DEVICE_SIGNALS; changelog. Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
github-actions
Bot
added
the
documentation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Kernel tracing gives every dispatch an SDK completion signal in host memory. With barrier-bit packets (HIP graph replay) each kernel then waits on a CP write across PCIe. For small kernels this wait dominates and sets a per-kernel floor under both rocprofv3 and torch.profiler.
Technical Details
Inline interposition path (
queue_interposition.cpp) only:hsa_amd_signal_batch_create. Falls back to host signals when the runtime lacks the API or the agent has no such pool.ROCPROFILER_DEVICE_SIGNALS=0disables.hsa_amd_profiling_convert_tick_to_system_domain. One fence per completed batch.rocprofiler_flush_bufferwaits, bounded, for completions already registered. A flush right after a device sync (Kineto flushes before stopping the context) no longer drops the last records.The legacy path (
ROCPROFILER_QUEUE_INTERPOSITION=0,queue.cpp) is unchanged.Related: CLR #13424 (device graph signals when no profiler is attached), CLR #13507 (signal hints). The signal-less KFD dispatch log is the long-term alternative.
Issue Tracking
N/A
Test Plan
--kernel-traceand torch.profiler, stock develop vs this branch; record counts and timestamp sanity (duplicates, overlaps, non-positive durations); eager launches.ROCPROFILER_DEVICE_SIGNALS=0.rocprofiler-sdk-lib-tests: hsa_barrier, queue_interposition, signal_pool, queue_controller, rocprofiler_lib.Test Result
ROCPROFILER_DEVICE_SIGNALS=0matches develop.Submission Checklist