[saleelk]

Motivation

Two host-side costs remain for signals in device memory after #14115:

1. XGMI-attached GPUs get no shadow. perf(rocr): resolve device-memory batch signals without reading the slot #14115 keeps a host-side copy of each batch member's
   Signal* so Signal::Convert resolves a device-memory batch handle without reading the slot's
   id and core_signal across the link. The copy was only made when
   GpuAgent::PcieStoresNeedFlush() is true, which is !xgmi_cpu_gpu_. On XGMI-attached systems,
   every signal API call on a device-memory batch member still read two fields across XGMI.
2. Reading timestamps costs two round trips. SharedSignal::GetRawTs reads start_ts and
   end_ts with two 8-byte loads, so hsa_amd_profiling_get_dispatch_time and
   hsa_amd_profiling_get_async_copy_time cost two uncached reads per device-memory signal. CLR's
   built-in profiler (GPU_CLR_PROFILE_OUTPUT) attaches a signal to every dispatch, so this cost is
   paid once per kernel.

Technical Details

• hsa_amd_signal_batch_create shadows every batch whose arena is in a discrete GPU's device pool
  (SignalBatch::Create takes the decision as shadow_members). The host reads a discrete GPU's
  memory across PCIe or XGMI either way. host_access_via_hdp is unchanged: it still decides HDP
  flushes after stores and emulated read-modify-writes, which only apply over PCIe.
• An APU's device pools (properties().Integrated) are not shadowed: the host shares that memory,
  so reading the slot costs no link round trip. Before, an APU's batches were shadowed when its CPU
  link was not reported as XGMI.
• SharedSignal::GetRawTs reads both timestamps with one 16-byte SSE2 load on x86-64. They fill one
  aligned 16-byte chunk of amd_signal_t. static_asserts pin the layout and the alignment the
  load relies on: alignof(amd_signal_t) >= 16 and amd_signal at a 16-byte offset in
  SharedSignal. Both signal allocators align to SharedSignal. Other architectures keep the two
  loads.

A host read of device memory on MI300X: 8-byte load 1.1 us, 16-byte load 1.1 us (one transaction),
32- or 64-byte load 2.2 us (two transactions).

Test Plan

• rocrtst *Signal* and IPC tests.
• Shadow stress (300 device batches, concurrent member loads while batches are recreated).
• Per-call signal API cost on host and device batch signals.
• hip_launch_latency -k -g with GPU_CLR_PROFILE_OUTPUT set, CLR device-memory queue signals on
  and off, against develop's ROCr; kernel durations from the traces compared.
• A pageable-copy stress test under the profiler with HSA_ENABLE_SDMA=1 and 0, checking copy
  durations in the trace (hsa_amd_profiling_get_async_copy_time).

Test Result

MI300X (PCIe host):

• rocrtst: 14/14 pass. Stress: pass (11k batch recreations, 15.7M concurrent member loads, 0
  mismatches).
• Per call on a device-memory batch signal, median of 15 rounds of 5000 calls, develop and this
  branch interleaved: hsa_amd_profiling_get_dispatch_time 2.24-2.47 -> 1.15-1.27 us;
  hsa_signal_load_relaxed unchanged at 1.06-1.25 us. Host-memory signals: 6-7 ns both.
• Profiled launches with device-memory signals: 100-node graph enqueue 463 -> 351 us, single
  launch + sync 11.7 -> 10.5 us. Host-memory signals unchanged.
• Traces: no zero or negative kernel or copy durations; host-signal kernel median identical
  (3.328 us) on both ROCr builds; copy durations consistent with SDMA on and off.

Not run on an XGMI-attached system or an APU. On XGMI, device batches now take the shadow path
that #14115 measured over PCIe, where dropping the slot's id and core_signal reads took
hsa_signal_load_relaxed from 3.5 to 1.1 us; the XGMI saving is two reads across XGMI per call,
not measured here.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

[therock-pr-bot]

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	⚠️ Warning	Error: Source/code files changed without an accompanying unit test. Expected: add at least one test file named like test_<name>.py / test_<name>.cpp (or <name>_test.*). Current: code file(s) changed: projects/rocr-runtime/runtime/hsa-runtime/core/inc/signal.h, projects/rocr-runtime/runtime/hsa-runtime/core/runtime/hsa_ext_amd.cpp, projects/rocr-runtime/runtime/hsa-runtime/core/runtime/signal.cpp; no test file found
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[cfreeamd]

🤖 AI-assisted pre-review (for @cfreeamd)

The SSE timestamp load and the shadow-allocation refactor are structurally sound; amd_signal_t is aligned(64) and amd_signal is SharedSignal's first member, so the _mm_load_si128 address is 32-byte aligned at runtime.

[systems-assistant]

TheRock Submodule Bump Activity

Newest first

• 2026-10-30 — Included in TheRock via ROCm/TheRock#9592.