[rocprofiler-sdk] Prevent queue-interposition doorbell over-advance

[minseobshin11]

Motivation

The inline queue-interposition path intermittently hangs HIP applications under kernel_dispatch tracing. Two reported symptoms — a multi-threaded producer deadlock (threads stuck in the doorbell path) and an async signal handler waiting forever on a completion signal pinned at value=1 — are both caused by a single bug in the doorbell handling. This PR fixes that root cause.

Technical Details

process_doorbell_impl virtualizes the queue write pointer: producers advance a shadow index (virtual_wptr) and the real hardware doorbell is only meant to be advanced by the SDK after it has actually submitted packets (real_wdid == next_submit_pos). Two early-exit paths violated this: scan_pos >= wptr_end (nothing new to scan) and drained == 0 (next slot claimed by a producer but not yet written).

Both rang the hardware doorbell with the application's virtualized value (virtual_wptr - 1), which can point past what the SDK has published. This advertises unpublished (and possibly unwritten) ring slots to the GPU, so the hardware read index (real_rdid) overshoots the shadow submit index. The overshoot then (1) wedges wait_for_free_slot in an unsigned-underflow spin (submit_pos - real_rdid underflows to ~UINT64_MAX) while holding gate_lock, so every other producer blocks on that lock — the producer deadlock; and (2) leaves a dispatch's completion signal at its +1 bump (never retired in order), so the async handler waits forever — the value=1 stall.

Fix:

• Add ring_published_doorbell(), which rings the doorbell with the last index the SDK has actually submitted (next_submit_pos - 1), and use it in both early-exit paths instead of forwarding the virtual value. Liveness is preserved because a not-yet-written slot is published by its own producer's later doorbell, and process_doorbell_impl always scans up to the current virtual_wptr.
• Guard the wait_for_free_slot subtraction so it can never underflow-spin (return when real_rdid >= submit_pos) — defense-in-depth.

Single file changed: source/lib/rocprofiler-sdk/hsa/queue_interposition.cpp.

JIRA ID

Resolves AIPROFSDK-910

Test Plan

• Reproduced a permanent hang on MI300X (gfx942, ROCm 7.2.3) on a clean develop build using high queue contention: HSA_XNACK=1 HIP_VISIBLE_DEVICES=0 rocprofv3 --kernel-trace -- transpose 16 800 800 1024 1024.
• Confirmed the mechanism via gdb (gate_lock holder spinning in wait_for_free_slot; *real_rdid > *real_wdid == next_submit_pos) and via temporary in-tree instrumentation that captured the over-advancing doorbell ring (value+1 > real_wdid) immediately preceding the overshoot.
• Positive control: the unfixed build hangs within a few runs under the same harness.
• Fix validation campaign across multiple contention profiles (16/32 threads, single- and multi-GPU, XNACK on/off, frequent-sync, long soak), plus a correctness check that the captured kernel_dispatch record count matches the expected number of dispatches (no dropped/duplicated kernels).

Test Result

• Unfixed (control): hangs reliably under the high-contention repro.
• Fixed: 447/447 runs passed across all profiles — 0 hangs, 0 over-advance events, 0 invariant violations, 0 still waiting on signal warnings — and dispatch counts matched expected on every run.
• CI verification of the affected rocprofiler-systems tests (unified-memory-output-sys-run, kfd-events-sys-run) to be confirmed via the TheRock test workflow.

Submission Checklist

• [V] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[github-actions]

Code Coverage Report

Code Coverage Report

Tests Only

Samples Only

Tests + Samples