[minseobshin11]

Motivation

PR #8007 (AIPROFSDK-887) allows a device thread trace context to be started before hsa_init() by deferring the request and replaying it once HSA is registered. On develop, the new thread-trace-api-agent-before-hsa-init-test intermittently hangs the GPU (HW Exception ... reason :GPU Hang) because the deferred start programs the thread trace (SQTT) hardware too early during HSA initialization. This PR fixes the hang while keeping the deferred-start feature.

Technical Details

The deferred-start replay ran inside thread_trace::initialize(), which executes during HSA table registration before queue_controller_init() and queue interposition are set up. Programming the SQTT hardware at that point intermittently wedges the GPU command processor. The replay is moved out of initialize() (now only builds per-agent resources) into a new thread_trace::start_active_contexts(), invoked at the end of the HSA registration path after queue_controller_init() and queue interposition. This matches the deferred-start ordering already used by device counting (device_counting_service_hsa_registration) and PC sampling (post_hsa_init_start_active_service). Changed files: thread_trace/core.cpp, thread_trace/core.hpp, registration.cpp.

Test Plan

Reproduced on MI300X (gfx942, 8 GPUs) using the CI test environment (ATT_START_BEFORE_HSA_INIT=1), which hung within a few runs. After the fix: ran the failing test 100x in a loop, ran thread-trace-api-agent-before-hsa-init-test and thread-trace-api-agent-start-stop-before-hsa-init-test 25x each under ctest --repeat until-fail, ran the full thread-trace integration suite, and confirmed trace data is still collected end-to-end with the trace decoder (latency validated, identical to the normal agent test).

Test Result

Before: GPU hang within 2-4 runs (~25%). After: 100/100 loop runs clean, both before-hsa-init tests pass 25/25 under ctest, full thread-trace suite passes (10/10, 2 disabled), and the decoder confirms trace data is collected. No regressions.

Submission Checklist

• [V] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

[therock-pr-bot]

❌ PR Check — Action Required

Check	Status	Details
🌿 Branch Name	✅ Pass	—
📝 PR Title/Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	❌ Fail	Error: Source/code files changed without an accompanying unit test. Expected: add at least one test file named like test_<name>.py / test_<name>.cpp (or <name>_test.*). Current: code file(s) changed: projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/registration.cpp, projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/thread_trace/core.cpp, projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/thread_trace/core.hpp; no test file found
🔎 pre-commit	⏳ Pending	⏳ Still running…
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

⚠️ 1 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

• ❌ Unit Test

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

[therock-pr-bot]

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

• ❌ Unit Test

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

[Copilot]

Pull request overview

This PR addresses an intermittent GPU hang when a device thread trace context is started before hsa_init() by changing when deferred thread-trace starts are replayed during HSA registration, ensuring SQTT hardware programming happens only after the queue infrastructure is fully initialized.

Changes:

• Updates thread_trace::initialize() to only build per-agent resources and no longer replay deferred starts.
• Adds thread_trace::start_active_contexts() to replay pre-hsa_init() device thread-trace starts after queue_controller_init() and queue interposition are established.
• Invokes thread_trace::start_active_contexts() at the end of the HSA registration path in registration.cpp.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/thread_trace/core.hpp	Documents the new split between resource initialization and deferred-start replay via start_active_contexts().
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/thread_trace/core.cpp	Moves deferred-start replay out of initialize() into the new start_active_contexts() to avoid early SQTT programming.
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/registration.cpp	Calls rocprofiler::thread_trace::start_active_contexts() after queue controller initialization and queue interposition setup.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.