[venkat1361]

Motivation

• Intermittent roctx-pause-resume hang.

Thread 0  (SDK: dispatch + serializer)                     | Thread 1  (HSA async completion)
-----------------------------------------------------------+----------------------------------
roctx RESUME  ->  serializer.enable()                      |
Q0 dispatches kernels  0, 1, ... , 10                      |
                                                           | completing Q0 kernels 0..9
                                                           |
roctx PAUSE   ->  serializer.disable()                     |
  set_barrier(): Q0 still has 1 kernel in flight (#10)     |
  => new transition Barrier B  [handle:1]  ARMED (=1)      |
                                                           |
Q1 dispatches 1 kernel + the transition packet             |
  BARRIER_AND  dep_signal = handle:1   (Q1 blocked)        |
                                                           |
                                                           | Q0 kernel #10 completes
                                                           |   -> kernel_completion_signal()
                                                           |   -> Barrier B count hits 0
                                                           |   => handle:1 CLEARED (=0), B.complete()
                                                           |
roctx RESUME  ->  serializer.enable()                      |
  clear_complete_barriers(): B is complete()               |
  -> ~hsa_barrier() DESTROYS handle:1                      |
     (but Q1's transition packet STILL waits on it!)       |
  Q1 has 1 kernel in flight while serialization was off    |
  -> new Barrier B': hsa_signal_create REUSES handle:1     |
  -> set_barrier(): handle:1 ARMED again (=1)              |
                                                           |
*** DEADLOCK ***                                           |
  Q1's old packet waits on handle:1 - now armed by B'      |
  -> Q1 never runs; serializer idle, no token owner        |
-----------------------------------------------------------+----------------------------------

handle:1 lifecycle:  ARMED(B)  ->  CLEARED(0)  ->  DESTROYED  ->  REUSED+ARMED(B')
                     ^------------- same handle, re-armed under Q1's stale packet

Root cause:  handle:1 was destroyed (and reused) while Q1's transition packet still
             referenced it, so Q1 ends up waiting on an unrelated, armed barrier.
Fix:    keep B until Q1 has executed past the dispatch carrying its packet
             -- safe_to_retire() = complete() AND no queue still references the signal.
             

Technical Details

A serializer barrier owns an HSA signal that its queues' transition (barrier-AND) packets wait on. The hang occurred when a barrier was retired (signal destroyed) while a queue still held an un-executed transition packet; HSA recycled the freed handle into the next barrier, and if that one was armed the queue blocked on the wrong barrier → deadlock.

Fix: defer retirement until every transition packet has actually executed:

• hsa_barrier tracks per-queue the dispatch id of the transition packet it handed out (_barrier_enqueued is now a queue -> dispatch_id map).
• Queue exposes monotonic serialized_dispatched / serialized_completed counters so a barrier can tell whether a queue has executed past a given dispatch.
• drain_queue(queue) clears a queue's tag once serialized_completed reaches its token; it early-exits without taking the lock while the barrier is still armed, since a transition packet can't have executed before its barrier clears. remove_queue() clears the tag on teardown
• safe_to_destroy() = complete() and no outstanding tags. clear_complete_barriers() retires a barrier (destroying its signal) only when safe_to_destroy(), so the signal is freed only once nothing references it.
• The destructor enforces this as an invariant: it logs a ROCP_ERROR if a barrier is ever destroyed with outstanding tags (should be unreachable given the gating + remove_queue on teardown) and then destroys the signal normally, no leak.

JIRA ID

ROCM-25255

Test Plan

• Tests (tests/hsa_barrier.cpp): retirement gated on drain, teardown escape, and signal-handle-not-recycled (fails on the old code, passes on the fix).

Test Result

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[Copilot]

Pull request overview

This PR addresses an intermittent roctx pause/resume deadlock in rocprofiler-sdk’s HSA profiling serializer by preventing premature retirement (signal destruction) of serializer transition barriers while any queue may still have an unexecuted transition packet referencing the barrier’s signal.

Changes:

• Add per-queue monotonic serialized_dispatched/serialized_completed counters to detect when a queue has executed past the dispatch that carried a barrier transition packet.
• Extend hsa_barrier with per-queue “transition enqueued” tracking and safe_to_retire() gating so barriers are only retired after transition packets drain (plus a destructor backstop to avoid handle reuse).
• Add regression tests covering deferred retirement, queue teardown escape, and non-recycling of barrier signal handles while referenced.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/hsa_barrier.cpp	Adds regression tests validating barrier retirement is deferred until transition packets drain and handles aren’t recycled while referenced.
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/queue.hpp	Introduces serialized dispatch/completion counters used to determine transition packet drain progress.
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/queue.cpp	Increments the new serialized counters at dispatch and completion to drive drain detection.
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/profile_serializer.cpp	Uses safe_to_retire() when clearing barriers and notifies barriers on serialized completions to drop drained queue tags.
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/hsa_barrier.hpp	Adds drain notification API, safe_to_retire(), handle accessor, and changes enqueued tracking to a queue→dispatch-id map.
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/hsa_barrier.cpp	Implements deferred retirement via drain tracking, safe_to_retire(), and a destructor backstop that avoids handle reuse.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[meserve-amd]

I think this is fix is overall correct, but I want to get rid of the leak-to-cover-up behavior. Some other comments around naming and comments. Overall I think some more explanation could be warranted around what the "notify_drain" is checking for.

[meserve-amd]

Looks good now. Thanks for addressing everything.

I recommend you ask @bwelton for a second review; they probably have the best understanding here.

[bwelton]

I am ok with these changes for the most part, i would suggest moving the counters into the serializer itself since its the only one that actually cares about them. I also think we may need to include this draining/serialization destruction in QueueController when we are destroying queues (however i am not sure that actually matters in practice).