[PGO][HIP] HSA-introspection device profile drain + GPU PGO tests - #203056
Merged
Conversation
|
✅ With the latest revision this PR passed the Python code formatter. |
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
🐧 Linux x64 Test Results
✅ The build succeeded and all tests passed. |
lfmeadow
force-pushed
the
upstream-pgo-hsa-drain
branch
2 times, most recently
from
June 13, 2026 21:53
35260c9 to
575f979
Compare
2 tasks
lfmeadow
force-pushed
the
upstream-pgo-hsa-drain
branch
3 times, most recently
from
June 14, 2026 19:22
023bf69 to
a054c9d
Compare
lfmeadow
marked this pull request as ready for review
June 14, 2026 19:27
llvmorg-github-actions
Bot
added
compiler-rt
backend:AMDGPU
clang:driver
|
@llvm/pr-subscribers-clang-driver @llvm/pr-subscribers-pgo Author: Larry Meadows (lfmeadow) ChangesSummaryFollow-up to #202095 (now landed). #202095's host-shadow device-profile drain can This adds a supplemental, Linux-only HSA-introspection drain that runs after It is purely additive — it does not modify #202095's host-shadow drain or its
Why a separate test harnessThere are no AMD GPUs in upstream CI, so these Test plan
Patch is 99.22 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/203056.diff 26 Files Affected:
diff --git a/clang/lib/Driver/ToolChains/Clang.cpp b/clang/lib/Driver/ToolChains/Clang.cpp
index c2ac478d84929..3b8bc46820af6 100644
--- a/clang/lib/Driver/ToolChains/Clang.cpp
+++ b/clang/lib/Driver/ToolChains/Clang.cpp
@@ -9658,6 +9658,21 @@ void LinkerWrapper::ConstructJob(Compilation &C, const JobAction &JA,
(TC->getTriple().isAMDGPU() || TC->getTriple().isNVPTX()))
LinkerArgs.emplace_back("-lompdevice");
+ // With PGO/coverage instrumentation, GPU device code references the
+ // device profile runtime (__llvm_profile_instrument_gpu and the
+ // __llvm_profile_sections bounds table emitted by
+ // InstrProfilingPlatformGPU). The offload device link does not otherwise
+ // pull it in, so forward the static device profile runtime to the GPU
+ // device linker. The archive is arch-suffixed, so pass its full path
+ // rather than a -l name.
+ if (ToolChain::needsProfileRT(Args) &&
+ (TC->getTriple().isAMDGPU() || TC->getTriple().isNVPTX())) {
+ std::string ProfileRT =
+ TC->getCompilerRT(Args, "profile", ToolChain::FT_Static);
+ if (TC->getVFS().exists(ProfileRT))
+ LinkerArgs.emplace_back(Args.MakeArgString(ProfileRT));
+ }
+
// For SPIR-V, pass some extra flags to `spirv-link`, the out-of-tree
// SPIR-V linker. `spirv-link` isn't called in LTO mode so restrict these
// flags to normal compilation.
diff --git a/clang/lib/Driver/ToolChains/HIPAMD.cpp b/clang/lib/Driver/ToolChains/HIPAMD.cpp
index 01cb23d0aa230..1bd4e073b4e27 100644
--- a/clang/lib/Driver/ToolChains/HIPAMD.cpp
+++ b/clang/lib/Driver/ToolChains/HIPAMD.cpp
@@ -19,6 +19,7 @@
#include "clang/Options/Options.h"
#include "llvm/Support/FileSystem.h"
#include "llvm/Support/Path.h"
+#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/TargetParser/TargetParser.h"
using namespace clang::driver;
@@ -142,6 +143,25 @@ void AMDGCN::Linker::constructLldCommand(Compilation &C, const JobAction &JA,
LldArgs.push_back("--no-whole-archive");
+ // With PGO/coverage instrumentation, instrumented device code references the
+ // device profile runtime (__llvm_profile_instrument_gpu and the
+ // __llvm_profile_sections bounds table emitted by InstrProfilingPlatformGPU).
+ // The new-offload-driver path injects this in LinkerWrapper::ConstructJob,
+ // but HIP using the traditional offload path (e.g. on Windows, which does not
+ // route device linking through clang-linker-wrapper) reaches the device link
+ // here instead. Forward the static device profile runtime to this lld device
+ // link so the runtime is pulled in regardless of offload-driver/host OS. The
+ // archive is arch-suffixed, so pass its full path rather than a -l name.
+ if (ToolChain::needsProfileRT(Args)) {
+ std::string ProfileRT =
+ TC.getCompilerRT(Args, "profile", ToolChain::FT_Static);
+ // Use the ToolChain VFS (matches the new-offload-driver path in
+ // Clang.cpp) so overlay/virtual filesystems used by the driver are
+ // honored; llvm::sys::fs bypasses them and can wrongly skip the runtime.
+ if (TC.getVFS().exists(ProfileRT))
+ LldArgs.push_back(Args.MakeArgString(ProfileRT));
+ }
+
const char *Lld = Args.MakeArgStringRef(getToolChain().GetProgramPath("lld"));
C.addCommand(std::make_unique<Command>(JA, *this, ResponseFileSupport::None(),
Lld, LldArgs, Inputs, Output));
diff --git a/clang/lib/Driver/ToolChains/Linux.cpp b/clang/lib/Driver/ToolChains/Linux.cpp
index 512788d235fec..00ae53af4865f 100644
--- a/clang/lib/Driver/ToolChains/Linux.cpp
+++ b/clang/lib/Driver/ToolChains/Linux.cpp
@@ -906,13 +906,28 @@ void Linux::addOffloadRTLibs(unsigned ActiveKinds, const ArgList &Args,
Args.MakeArgString(StringRef("-L") + RocmInstallation->getLibPath()));
// For HIP device PGO, link clang_rt.profile_rocm when available. It is a
- // self-contained superset of clang_rt.profile, emitted first so the base
- // archive stays inert.
- if ((ActiveKinds & Action::OFK_HIP) && needsProfileRT(Args) &&
+ // self-contained superset of clang_rt.profile, emitted first (before the
+ // base archive added by addProfileRTLibs) so the base archive stays inert.
+ //
+ // This is intentionally not gated on Action::OFK_HIP. HIP host objects are
+ // routinely linked into a shared library or executable from pre-compiled
+ // .o files (e.g. RCCL's librccl.so), a link command that carries no HIP
+ // offload action yet still needs the device-counter drain. Gating on
+ // OFK_HIP would silently drop the drain for those object-only links and
+ // the resulting .profraw would contain host counters only. profile_rocm is
+ // self-contained and both its hipModuleLoad interceptor and its
+ // device-collection drain self-skip at runtime when the process has no
+ // resident device code, so linking it into a non-HIP instrumented binary is
+ // harmless. It is only present on ROCm-equipped toolchains in the first
+ // place (the getVFS().exists check below).
+ if (needsProfileRT(Args) &&
getVFS().exists(getCompilerRT(Args, "profile_rocm", FT_Static))) {
CmdArgs.push_back(getCompilerRTArgString(Args, "profile_rocm"));
// Force-retain the constructor-only hipModuleLoad* interceptor object; its
// constructor self-skips when the program does not use hipModuleLoad.
+ // Pulling this object in also pulls the device-counter drain
+ // (__llvm_profile_hip_collect_device_data) from the same translation unit,
+ // which InstrProfilingFile.c invokes through a weak reference at exit.
CmdArgs.push_back("-u");
CmdArgs.push_back("__llvm_profile_offload_register_dynamic_module");
}
diff --git a/clang/lib/Driver/ToolChains/MSVC.cpp b/clang/lib/Driver/ToolChains/MSVC.cpp
index 0796bdff96d46..9a7df6af7727c 100644
--- a/clang/lib/Driver/ToolChains/MSVC.cpp
+++ b/clang/lib/Driver/ToolChains/MSVC.cpp
@@ -598,19 +598,26 @@ void MSVCToolChain::addOffloadRTLibs(unsigned ActiveKinds, const ArgList &Args,
CmdArgs.append({Args.MakeArgString(StringRef("-libpath:") +
RocmInstallation->getLibPath()),
"amdhip64.lib"});
+ }
- // For HIP device PGO, link clang_rt.profile_rocm when available. It is a
- // self-contained superset of clang_rt.profile, emitted first so the base
- // archive stays inert (avoiding a /MD-vs-/MT CRT mix in the host image).
- if (needsProfileRT(Args) &&
- getVFS().exists(getCompilerRT(Args, "profile_rocm", FT_Static))) {
- CmdArgs.push_back(getCompilerRTArgString(Args, "profile_rocm"));
- // Force the linker to retain the constructor-only hipModuleLoad*
- // interceptor object from clang_rt.profile_rocm (see Linux.cpp). The
- // constructor self-skips for programs that do not use hipModuleLoad.
- CmdArgs.push_back(
- "-include:__llvm_profile_offload_register_dynamic_module");
- }
+ // For HIP device PGO, link clang_rt.profile_rocm when available. It is a
+ // self-contained superset of clang_rt.profile, emitted first so the base
+ // archive stays inert (avoiding a /MD-vs-/MT CRT mix in the host image).
+ //
+ // Not gated on Action::OFK_HIP: HIP host objects are routinely linked into a
+ // DLL or executable from pre-compiled .obj files, a link that carries no HIP
+ // offload action yet still needs the device-counter drain (see Linux.cpp for
+ // the full rationale). profile_rocm self-skips at runtime when the process
+ // has no resident device code, and is only present on ROCm-equipped
+ // toolchains (the getVFS().exists check below).
+ if (needsProfileRT(Args) &&
+ getVFS().exists(getCompilerRT(Args, "profile_rocm", FT_Static))) {
+ CmdArgs.push_back(getCompilerRTArgString(Args, "profile_rocm"));
+ // Force the linker to retain the constructor-only hipModuleLoad*
+ // interceptor object from clang_rt.profile_rocm (see Linux.cpp). The
+ // constructor self-skips for programs that do not use hipModuleLoad.
+ CmdArgs.push_back(
+ "-include:__llvm_profile_offload_register_dynamic_module");
}
}
diff --git a/clang/test/Driver/hip-profile-rocm-runtime.hip b/clang/test/Driver/hip-profile-rocm-runtime.hip
index fc82db4fc13c0..9346f05dedf42 100644
--- a/clang/test/Driver/hip-profile-rocm-runtime.hip
+++ b/clang/test/Driver/hip-profile-rocm-runtime.hip
@@ -25,9 +25,23 @@
// RUN: | FileCheck -check-prefix=HIP-NOPGO %s
// HIP-NOPGO-NOT: libclang_rt.profile_rocm.a
-// A non-HIP host link with PGO does not link the ROCm device-profile runtime.
+// An object-only host link with PGO (no HIP offload action) still links the
+// ROCm device-profile runtime when it is available in the toolchain. HIP host
+// code is frequently linked into a library/executable from pre-compiled
+// objects, a link that carries no OFK_HIP yet still needs the device drain.
// RUN: %clang -### --target=x86_64-unknown-linux \
// RUN: -fprofile-instr-generate -resource-dir=%t %t.o 2>&1 \
// RUN: | FileCheck -check-prefix=HOST-PGO %s
+// HOST-PGO: "{{.*}}libclang_rt.profile_rocm.a"
+// HOST-PGO: "-u" "__llvm_profile_offload_register_dynamic_module"
// HOST-PGO: "{{.*}}libclang_rt.profile.a"
-// HOST-PGO-NOT: libclang_rt.profile_rocm.a
+
+// On a lean toolchain that ships only the base profile runtime (no
+// profile_rocm), nothing extra is linked and the link still succeeds.
+// RUN: rm -rf %t2 && mkdir -p %t2/lib/x86_64-unknown-linux
+// RUN: touch %t2/lib/x86_64-unknown-linux/libclang_rt.profile.a
+// RUN: %clang -### --target=x86_64-unknown-linux \
+// RUN: -fprofile-instr-generate -resource-dir=%t2 %t.o 2>&1 \
+// RUN: | FileCheck -check-prefix=NO-ROCM-RT %s
+// NO-ROCM-RT: "{{.*}}libclang_rt.profile.a"
+// NO-ROCM-RT-NOT: libclang_rt.profile_rocm.a
diff --git a/compiler-rt/lib/profile/InstrProfilingPlatformROCm.cpp b/compiler-rt/lib/profile/InstrProfilingPlatformROCm.cpp
index d0d9b1ea8f61d..b1db1d8a74041 100644
--- a/compiler-rt/lib/profile/InstrProfilingPlatformROCm.cpp
+++ b/compiler-rt/lib/profile/InstrProfilingPlatformROCm.cpp
@@ -66,6 +66,15 @@ struct OffloadSectionShadowGroup;
static int processDeviceOffloadPrf(void *DeviceOffloadPrf, const char *Target,
const OffloadSectionShadowGroup *Sections);
+#if defined(__linux__) && !defined(_WIN32)
+// Record a drained section-bounds tuple so the supplemental HSA-introspection
+// pass (Linux only) skips any code object the host-shadow path already
+// drained. Defined alongside the HSA drain below; forward-declared here so
+// processDeviceOffloadPrf can register every successful host-shadow drain.
+static void profRecordDrainedBounds(const void *Data, const void *Counters,
+ const void *Names);
+#endif
+
static int isVerboseMode() {
static int IsVerbose = -1;
if (IsVerbose == -1)
@@ -1119,8 +1128,14 @@ static int processDeviceOffloadPrf(void *DeviceOffloadPrf, const char *Target,
if (ret != 0) {
PROF_ERR("%s\n", "failed to write device profile using shared API");
- } else if (isVerboseMode()) {
- PROF_NOTE("%s\n", "Successfully wrote device profile using shared API");
+ } else {
+#if defined(__linux__) && !defined(_WIN32)
+ // Dedup against the supplemental HSA pass: this section is now drained, so
+ // the HSA walk must not drain the same device code object again.
+ profRecordDrainedBounds(DevDataBegin, DevCntsBegin, DevNamesBegin);
+#endif
+ if (isVerboseMode())
+ PROF_NOTE("%s\n", "Successfully wrote device profile using shared API");
}
return ret;
@@ -1148,72 +1163,635 @@ static int isHipAvailable(void) {
return pHipMemcpy != nullptr && pHipGetSymbolAddress != nullptr;
}
-/* -------------------------------------------------------------------------- */
-/* Collect device-side profile data */
-/* -------------------------------------------------------------------------- */
+/* ========================================================================== */
+/* Supplemental HSA-introspection drain (Linux only) */
+/* */
+/* The host-shadow drain above only sees device code objects registered */
+/* host-side (__hipRegisterVar shadows) or loaded through an intercepted */
+/* hipModuleLoad* call. Device code linked by the offload device linker with */
+/* no host-side shadow -- e.g. RCCL, whose many device functions are glued */
+/* into a single kernel with no source module -- is invisible to it. This */
+/* pass walks every GPU agent's loaded executables via HSA, finds each */
+/* __llvm_profile_sections table directly on the device, and drains the ones */
+/* the host-shadow pass did not already handle (deduped by the device */
+/* section-bounds tuple). It reuses processDeviceOffloadPrf() for the */
+/* copy/relocate/write so the on-disk profraw layout is identical. */
+/* ========================================================================== */
+#if defined(__linux__) && !defined(_WIN32)
-extern "C" int __llvm_profile_hip_collect_device_data(void) {
- if (NumShadowVariables == 0 && NumDynamicModules == 0)
+/* Minimal HSA type/enum stubs. compiler-rt cannot depend on ROCm headers at
+ * build time, so mirror just the handful of HSA declarations the drain needs.
+ * Values match hsa/hsa.h and hsa/hsa_ven_amd_loader.h. */
+typedef uint32_t prof_hsa_status_t;
+#define PROF_HSA_STATUS_SUCCESS ((prof_hsa_status_t)0x0)
+#define PROF_HSA_STATUS_INFO_BREAK ((prof_hsa_status_t)0x1)
+
+typedef struct {
+ uint64_t handle;
+} prof_hsa_agent_t;
+typedef struct {
+ uint64_t handle;
+} prof_hsa_executable_t;
+typedef struct {
+ uint64_t handle;
+} prof_hsa_executable_symbol_t;
+
+typedef uint32_t prof_hsa_agent_info_t;
+#define PROF_HSA_AGENT_INFO_NAME ((prof_hsa_agent_info_t)0)
+#define PROF_HSA_AGENT_INFO_DEVICE ((prof_hsa_agent_info_t)17)
+
+typedef uint32_t prof_hsa_device_type_t;
+#define PROF_HSA_DEVICE_TYPE_GPU ((prof_hsa_device_type_t)1)
+
+typedef uint32_t prof_hsa_symbol_kind_t;
+#define PROF_HSA_SYMBOL_KIND_VARIABLE ((prof_hsa_symbol_kind_t)0)
+
+typedef uint32_t prof_hsa_executable_symbol_info_t;
+#define PROF_HSA_EXECUTABLE_SYMBOL_INFO_TYPE \
+ ((prof_hsa_executable_symbol_info_t)0)
+#define PROF_HSA_EXECUTABLE_SYMBOL_INFO_NAME_LENGTH \
+ ((prof_hsa_executable_symbol_info_t)1)
+#define PROF_HSA_EXECUTABLE_SYMBOL_INFO_NAME \
+ ((prof_hsa_executable_symbol_info_t)2)
+#define PROF_HSA_EXECUTABLE_SYMBOL_INFO_VARIABLE_ADDRESS \
+ ((prof_hsa_executable_symbol_info_t)21)
+
+#define PROF_HSA_EXTENSION_AMD_LOADER ((uint16_t)0x201)
+
+typedef uint32_t prof_hsa_loader_storage_type_t;
+
+typedef struct {
+ prof_hsa_agent_t agent;
+ prof_hsa_executable_t executable;
+ prof_hsa_loader_storage_type_t code_object_storage_type;
+ const void *code_object_storage_base;
+ size_t code_object_storage_size;
+ size_t code_object_storage_offset;
+ const void *segment_base;
+ size_t segment_size;
+} prof_hsa_loader_segment_descriptor_t;
+
+typedef prof_hsa_status_t (*hsa_init_ty)(void);
+typedef prof_hsa_status_t (*hsa_iterate_agents_ty)(
+ prof_hsa_status_t (*)(prof_hsa_agent_t, void *), void *);
+typedef prof_hsa_status_t (*hsa_agent_get_info_ty)(prof_hsa_agent_t,
+ prof_hsa_agent_info_t,
+ void *);
+typedef prof_hsa_status_t (*hsa_executable_iterate_agent_symbols_ty)(
+ prof_hsa_executable_t, prof_hsa_agent_t,
+ prof_hsa_status_t (*)(prof_hsa_executable_t, prof_hsa_agent_t,
+ prof_hsa_executable_symbol_t, void *),
+ void *);
+typedef prof_hsa_status_t (*hsa_executable_symbol_get_info_ty)(
+ prof_hsa_executable_symbol_t, prof_hsa_executable_symbol_info_t, void *);
+typedef prof_hsa_status_t (*hsa_system_get_major_extension_table_ty)(uint16_t,
+ uint16_t,
+ size_t,
+ void *);
+typedef prof_hsa_status_t (*hsa_loader_query_segment_descriptors_ty)(
+ prof_hsa_loader_segment_descriptor_t *, size_t *);
+
+/* First two members of hsa_ven_amd_loader_1_00_pfn_t. Only
+ * query_segment_descriptors is used; query_host_address keeps the offset. */
+typedef struct {
+ void *query_host_address;
+ hsa_loader_query_segment_descriptors_ty query_segment_descriptors;
+} prof_hsa_loader_pfn_t;
+
+static hsa_iterate_agents_ty pHsaIterateAgents = nullptr;
+static hsa_agent_get_info_ty pHsaAgentGetInfo = nullptr;
+static hsa_executable_iterate_agent_symbols_ty pHsaExecIterAgentSyms = nullptr;
+static hsa_executable_symbol_get_info_ty pHsaSymGetInfo = nullptr;
+static hsa_loader_query_segment_descriptors_ty pQuerySegDescs = nullptr;
+
+/* 0 = not yet attempted, 1 = ready, -1 = unavailable. Accessed with acquire/
+ * release atomics: a thread observing HsaRuntimeState==1 (acquire) also sees
+ * the fully-written p* function pointers (published before the release store
+ * of HsaRuntimeState=1 below). */
+static int HsaRuntimeState = 0;
+
+static int setHsaRuntimeState(int S) {
+ __atomic_store_n(&HsaRuntimeState, S, __ATOMIC_RELEASE);
+ return S > 0 ? 0 : -1;
+}
+
+/* Resolve HSA entry points (and the AMD loader extension) once, and confirm
+ * HIP's hipMemcpy is reachable for the device-to-host copies. HIP itself is
+ * resolved by the shared ensureHipLoaded() above. */
+static int loadHsaRuntimePointers(void) {
+ int State = __atomic_load_n(&HsaRuntimeState, __ATOMIC_ACQUIRE);
+ if (State)
+ return State > 0 ? 0 : -1;
+
+ if (!__interception::DynamicLoaderAvailable()) {
+ if (isVerboseMode())
+ PROF_NOTE("%s", "Dynamic library loading not available - "
+ "HSA device profiling disabled\n");
+ return setHsaRuntimeState(-1);
+ }
+
+ void *Hsa = __interception::OpenLibrary("libhsa-runtime64.so");
+ if (!Hsa)
+ Hsa = __interception::OpenLibrary("libhsa-runtime64.so.1");
+ if (!Hsa) {
+ if (isVerboseMode())
+ PROF_NOTE("%s", "libhsa-runtime64.so not loadable - "
+ "HSA device profiling disabled\n");
+ return setHsaRuntimeState(-1);
+ }
+
+ hsa_init_ty pHsaInit =
+ (hsa_init_ty)__interception::LookupSymbol(Hsa, "hsa_init");
+ hsa_system_get_major_extension_table_ty pGetExtTable =
+ (hsa_system_get_major_extension_table_ty)__interception::LookupSymbol(
+ Hsa, "hsa_system_get_major_extension_table");
+ pHsaIterateAgents = (hsa_iterate_agents_ty)__interception::LookupSymbol(
+ Hsa, "hsa_iterate_agents");
+ pHsaAgentGetInfo = (hsa_agent_get_info_ty)__interception::LookupSymbol(
+ Hsa, "hsa_agent_get_info");
+ pHsaExecIterAgentSyms =
+ (hsa_executable_iterate_agent_symbols_ty)__interception::LookupSymbol(
+ Hsa, "hsa_executable_iterate_agent_symbols");
+ pHsaSymGetInfo =
+ (hsa_executable_symbol_get_info_ty)__interception::LookupSymbol(
+ Hsa, "hsa_executable_symbol_get_info");
+
+ if (!pHsaInit || !pGetExtTable || !pHsaIterateAgents || !pHsaAgentGetInfo ||
+ !pHsaExecIterAgentSyms || !pHsaSymGetInfo) {
+ PROF_WARN("%s",
+ "required HSA symbols missing - HSA device profiling disabled\n");
+ return setHsaRuntimeState(-1);
+ }
+
+ /* Bring HSA up (idempotent, refcounted). This runs lazily on the first drain
+ * rather than from the library constructor, so merely loading the
+ * instrumented library does not initialize HSA in the process -- which would
+ * break fork-based callers that deliberately keep HIP/HSA uninitialized in
+ * the parent (see the constructor note at the end of the HSA block). In the
+ * common case the drain runs from the profile write path while HSA is still
+ * alive; if it only runs after HSA's own atexit(hsa_shut_down) has executed,
+ * this simply re-initializes HSA (the process is exiting anyway). */
+ prof_hsa_status_t St = pHsaInit();
+ if (St != PROF_HSA_STATUS_SUCCESS && St != PROF_HSA_STATUS_INFO_BREAK) {
+ if (isVerboseMode())
+ PROF_NOTE("hsa_init failed (0x%x) - HSA device ...
[truncated]
|
|
@llvm/pr-subscribers-backend-amdgpu Author: Larry Meadows (lfmeadow) ChangesSummaryFollow-up to #202095 (now landed). #202095's host-shadow device-profile drain can This adds a supplemental, Linux-only HSA-introspection drain that runs after It is purely additive — it does not modify #202095's host-shadow drain or its
Why a separate test harnessThere are no AMD GPUs in upstream CI, so these Test plan
Patch is 99.22 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/203056.diff 26 Files Affected:
diff --git a/clang/lib/Driver/ToolChains/Clang.cpp b/clang/lib/Driver/ToolChains/Clang.cpp
index c2ac478d84929..3b8bc46820af6 100644
--- a/clang/lib/Driver/ToolChains/Clang.cpp
+++ b/clang/lib/Driver/ToolChains/Clang.cpp
@@ -9658,6 +9658,21 @@ void LinkerWrapper::ConstructJob(Compilation &C, const JobAction &JA,
(TC->getTriple().isAMDGPU() || TC->getTriple().isNVPTX()))
LinkerArgs.emplace_back("-lompdevice");
+ // With PGO/coverage instrumentation, GPU device code references the
+ // device profile runtime (__llvm_profile_instrument_gpu and the
+ // __llvm_profile_sections bounds table emitted by
+ // InstrProfilingPlatformGPU). The offload device link does not otherwise
+ // pull it in, so forward the static device profile runtime to the GPU
+ // device linker. The archive is arch-suffixed, so pass its full path
+ // rather than a -l name.
+ if (ToolChain::needsProfileRT(Args) &&
+ (TC->getTriple().isAMDGPU() || TC->getTriple().isNVPTX())) {
+ std::string ProfileRT =
+ TC->getCompilerRT(Args, "profile", ToolChain::FT_Static);
+ if (TC->getVFS().exists(ProfileRT))
+ LinkerArgs.emplace_back(Args.MakeArgString(ProfileRT));
+ }
+
// For SPIR-V, pass some extra flags to `spirv-link`, the out-of-tree
// SPIR-V linker. `spirv-link` isn't called in LTO mode so restrict these
// flags to normal compilation.
diff --git a/clang/lib/Driver/ToolChains/HIPAMD.cpp b/clang/lib/Driver/ToolChains/HIPAMD.cpp
index 01cb23d0aa230..1bd4e073b4e27 100644
--- a/clang/lib/Driver/ToolChains/HIPAMD.cpp
+++ b/clang/lib/Driver/ToolChains/HIPAMD.cpp
@@ -19,6 +19,7 @@
#include "clang/Options/Options.h"
#include "llvm/Support/FileSystem.h"
#include "llvm/Support/Path.h"
+#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/TargetParser/TargetParser.h"
using namespace clang::driver;
@@ -142,6 +143,25 @@ void AMDGCN::Linker::constructLldCommand(Compilation &C, const JobAction &JA,
LldArgs.push_back("--no-whole-archive");
+ // With PGO/coverage instrumentation, instrumented device code references the
+ // device profile runtime (__llvm_profile_instrument_gpu and the
+ // __llvm_profile_sections bounds table emitted by InstrProfilingPlatformGPU).
+ // The new-offload-driver path injects this in LinkerWrapper::ConstructJob,
+ // but HIP using the traditional offload path (e.g. on Windows, which does not
+ // route device linking through clang-linker-wrapper) reaches the device link
+ // here instead. Forward the static device profile runtime to this lld device
+ // link so the runtime is pulled in regardless of offload-driver/host OS. The
+ // archive is arch-suffixed, so pass its full path rather than a -l name.
+ if (ToolChain::needsProfileRT(Args)) {
+ std::string ProfileRT =
+ TC.getCompilerRT(Args, "profile", ToolChain::FT_Static);
+ // Use the ToolChain VFS (matches the new-offload-driver path in
+ // Clang.cpp) so overlay/virtual filesystems used by the driver are
+ // honored; llvm::sys::fs bypasses them and can wrongly skip the runtime.
+ if (TC.getVFS().exists(ProfileRT))
+ LldArgs.push_back(Args.MakeArgString(ProfileRT));
+ }
+
const char *Lld = Args.MakeArgStringRef(getToolChain().GetProgramPath("lld"));
C.addCommand(std::make_unique<Command>(JA, *this, ResponseFileSupport::None(),
Lld, LldArgs, Inputs, Output));
diff --git a/clang/lib/Driver/ToolChains/Linux.cpp b/clang/lib/Driver/ToolChains/Linux.cpp
index 512788d235fec..00ae53af4865f 100644
--- a/clang/lib/Driver/ToolChains/Linux.cpp
+++ b/clang/lib/Driver/ToolChains/Linux.cpp
@@ -906,13 +906,28 @@ void Linux::addOffloadRTLibs(unsigned ActiveKinds, const ArgList &Args,
Args.MakeArgString(StringRef("-L") + RocmInstallation->getLibPath()));
// For HIP device PGO, link clang_rt.profile_rocm when available. It is a
- // self-contained superset of clang_rt.profile, emitted first so the base
- // archive stays inert.
- if ((ActiveKinds & Action::OFK_HIP) && needsProfileRT(Args) &&
+ // self-contained superset of clang_rt.profile, emitted first (before the
+ // base archive added by addProfileRTLibs) so the base archive stays inert.
+ //
+ // This is intentionally not gated on Action::OFK_HIP. HIP host objects are
+ // routinely linked into a shared library or executable from pre-compiled
+ // .o files (e.g. RCCL's librccl.so), a link command that carries no HIP
+ // offload action yet still needs the device-counter drain. Gating on
+ // OFK_HIP would silently drop the drain for those object-only links and
+ // the resulting .profraw would contain host counters only. profile_rocm is
+ // self-contained and both its hipModuleLoad interceptor and its
+ // device-collection drain self-skip at runtime when the process has no
+ // resident device code, so linking it into a non-HIP instrumented binary is
+ // harmless. It is only present on ROCm-equipped toolchains in the first
+ // place (the getVFS().exists check below).
+ if (needsProfileRT(Args) &&
getVFS().exists(getCompilerRT(Args, "profile_rocm", FT_Static))) {
CmdArgs.push_back(getCompilerRTArgString(Args, "profile_rocm"));
// Force-retain the constructor-only hipModuleLoad* interceptor object; its
// constructor self-skips when the program does not use hipModuleLoad.
+ // Pulling this object in also pulls the device-counter drain
+ // (__llvm_profile_hip_collect_device_data) from the same translation unit,
+ // which InstrProfilingFile.c invokes through a weak reference at exit.
CmdArgs.push_back("-u");
CmdArgs.push_back("__llvm_profile_offload_register_dynamic_module");
}
diff --git a/clang/lib/Driver/ToolChains/MSVC.cpp b/clang/lib/Driver/ToolChains/MSVC.cpp
index 0796bdff96d46..9a7df6af7727c 100644
--- a/clang/lib/Driver/ToolChains/MSVC.cpp
+++ b/clang/lib/Driver/ToolChains/MSVC.cpp
@@ -598,19 +598,26 @@ void MSVCToolChain::addOffloadRTLibs(unsigned ActiveKinds, const ArgList &Args,
CmdArgs.append({Args.MakeArgString(StringRef("-libpath:") +
RocmInstallation->getLibPath()),
"amdhip64.lib"});
+ }
- // For HIP device PGO, link clang_rt.profile_rocm when available. It is a
- // self-contained superset of clang_rt.profile, emitted first so the base
- // archive stays inert (avoiding a /MD-vs-/MT CRT mix in the host image).
- if (needsProfileRT(Args) &&
- getVFS().exists(getCompilerRT(Args, "profile_rocm", FT_Static))) {
- CmdArgs.push_back(getCompilerRTArgString(Args, "profile_rocm"));
- // Force the linker to retain the constructor-only hipModuleLoad*
- // interceptor object from clang_rt.profile_rocm (see Linux.cpp). The
- // constructor self-skips for programs that do not use hipModuleLoad.
- CmdArgs.push_back(
- "-include:__llvm_profile_offload_register_dynamic_module");
- }
+ // For HIP device PGO, link clang_rt.profile_rocm when available. It is a
+ // self-contained superset of clang_rt.profile, emitted first so the base
+ // archive stays inert (avoiding a /MD-vs-/MT CRT mix in the host image).
+ //
+ // Not gated on Action::OFK_HIP: HIP host objects are routinely linked into a
+ // DLL or executable from pre-compiled .obj files, a link that carries no HIP
+ // offload action yet still needs the device-counter drain (see Linux.cpp for
+ // the full rationale). profile_rocm self-skips at runtime when the process
+ // has no resident device code, and is only present on ROCm-equipped
+ // toolchains (the getVFS().exists check below).
+ if (needsProfileRT(Args) &&
+ getVFS().exists(getCompilerRT(Args, "profile_rocm", FT_Static))) {
+ CmdArgs.push_back(getCompilerRTArgString(Args, "profile_rocm"));
+ // Force the linker to retain the constructor-only hipModuleLoad*
+ // interceptor object from clang_rt.profile_rocm (see Linux.cpp). The
+ // constructor self-skips for programs that do not use hipModuleLoad.
+ CmdArgs.push_back(
+ "-include:__llvm_profile_offload_register_dynamic_module");
}
}
diff --git a/clang/test/Driver/hip-profile-rocm-runtime.hip b/clang/test/Driver/hip-profile-rocm-runtime.hip
index fc82db4fc13c0..9346f05dedf42 100644
--- a/clang/test/Driver/hip-profile-rocm-runtime.hip
+++ b/clang/test/Driver/hip-profile-rocm-runtime.hip
@@ -25,9 +25,23 @@
// RUN: | FileCheck -check-prefix=HIP-NOPGO %s
// HIP-NOPGO-NOT: libclang_rt.profile_rocm.a
-// A non-HIP host link with PGO does not link the ROCm device-profile runtime.
+// An object-only host link with PGO (no HIP offload action) still links the
+// ROCm device-profile runtime when it is available in the toolchain. HIP host
+// code is frequently linked into a library/executable from pre-compiled
+// objects, a link that carries no OFK_HIP yet still needs the device drain.
// RUN: %clang -### --target=x86_64-unknown-linux \
// RUN: -fprofile-instr-generate -resource-dir=%t %t.o 2>&1 \
// RUN: | FileCheck -check-prefix=HOST-PGO %s
+// HOST-PGO: "{{.*}}libclang_rt.profile_rocm.a"
+// HOST-PGO: "-u" "__llvm_profile_offload_register_dynamic_module"
// HOST-PGO: "{{.*}}libclang_rt.profile.a"
-// HOST-PGO-NOT: libclang_rt.profile_rocm.a
+
+// On a lean toolchain that ships only the base profile runtime (no
+// profile_rocm), nothing extra is linked and the link still succeeds.
+// RUN: rm -rf %t2 && mkdir -p %t2/lib/x86_64-unknown-linux
+// RUN: touch %t2/lib/x86_64-unknown-linux/libclang_rt.profile.a
+// RUN: %clang -### --target=x86_64-unknown-linux \
+// RUN: -fprofile-instr-generate -resource-dir=%t2 %t.o 2>&1 \
+// RUN: | FileCheck -check-prefix=NO-ROCM-RT %s
+// NO-ROCM-RT: "{{.*}}libclang_rt.profile.a"
+// NO-ROCM-RT-NOT: libclang_rt.profile_rocm.a
diff --git a/compiler-rt/lib/profile/InstrProfilingPlatformROCm.cpp b/compiler-rt/lib/profile/InstrProfilingPlatformROCm.cpp
index d0d9b1ea8f61d..b1db1d8a74041 100644
--- a/compiler-rt/lib/profile/InstrProfilingPlatformROCm.cpp
+++ b/compiler-rt/lib/profile/InstrProfilingPlatformROCm.cpp
@@ -66,6 +66,15 @@ struct OffloadSectionShadowGroup;
static int processDeviceOffloadPrf(void *DeviceOffloadPrf, const char *Target,
const OffloadSectionShadowGroup *Sections);
+#if defined(__linux__) && !defined(_WIN32)
+// Record a drained section-bounds tuple so the supplemental HSA-introspection
+// pass (Linux only) skips any code object the host-shadow path already
+// drained. Defined alongside the HSA drain below; forward-declared here so
+// processDeviceOffloadPrf can register every successful host-shadow drain.
+static void profRecordDrainedBounds(const void *Data, const void *Counters,
+ const void *Names);
+#endif
+
static int isVerboseMode() {
static int IsVerbose = -1;
if (IsVerbose == -1)
@@ -1119,8 +1128,14 @@ static int processDeviceOffloadPrf(void *DeviceOffloadPrf, const char *Target,
if (ret != 0) {
PROF_ERR("%s\n", "failed to write device profile using shared API");
- } else if (isVerboseMode()) {
- PROF_NOTE("%s\n", "Successfully wrote device profile using shared API");
+ } else {
+#if defined(__linux__) && !defined(_WIN32)
+ // Dedup against the supplemental HSA pass: this section is now drained, so
+ // the HSA walk must not drain the same device code object again.
+ profRecordDrainedBounds(DevDataBegin, DevCntsBegin, DevNamesBegin);
+#endif
+ if (isVerboseMode())
+ PROF_NOTE("%s\n", "Successfully wrote device profile using shared API");
}
return ret;
@@ -1148,72 +1163,635 @@ static int isHipAvailable(void) {
return pHipMemcpy != nullptr && pHipGetSymbolAddress != nullptr;
}
-/* -------------------------------------------------------------------------- */
-/* Collect device-side profile data */
-/* -------------------------------------------------------------------------- */
+/* ========================================================================== */
+/* Supplemental HSA-introspection drain (Linux only) */
+/* */
+/* The host-shadow drain above only sees device code objects registered */
+/* host-side (__hipRegisterVar shadows) or loaded through an intercepted */
+/* hipModuleLoad* call. Device code linked by the offload device linker with */
+/* no host-side shadow -- e.g. RCCL, whose many device functions are glued */
+/* into a single kernel with no source module -- is invisible to it. This */
+/* pass walks every GPU agent's loaded executables via HSA, finds each */
+/* __llvm_profile_sections table directly on the device, and drains the ones */
+/* the host-shadow pass did not already handle (deduped by the device */
+/* section-bounds tuple). It reuses processDeviceOffloadPrf() for the */
+/* copy/relocate/write so the on-disk profraw layout is identical. */
+/* ========================================================================== */
+#if defined(__linux__) && !defined(_WIN32)
-extern "C" int __llvm_profile_hip_collect_device_data(void) {
- if (NumShadowVariables == 0 && NumDynamicModules == 0)
+/* Minimal HSA type/enum stubs. compiler-rt cannot depend on ROCm headers at
+ * build time, so mirror just the handful of HSA declarations the drain needs.
+ * Values match hsa/hsa.h and hsa/hsa_ven_amd_loader.h. */
+typedef uint32_t prof_hsa_status_t;
+#define PROF_HSA_STATUS_SUCCESS ((prof_hsa_status_t)0x0)
+#define PROF_HSA_STATUS_INFO_BREAK ((prof_hsa_status_t)0x1)
+
+typedef struct {
+ uint64_t handle;
+} prof_hsa_agent_t;
+typedef struct {
+ uint64_t handle;
+} prof_hsa_executable_t;
+typedef struct {
+ uint64_t handle;
+} prof_hsa_executable_symbol_t;
+
+typedef uint32_t prof_hsa_agent_info_t;
+#define PROF_HSA_AGENT_INFO_NAME ((prof_hsa_agent_info_t)0)
+#define PROF_HSA_AGENT_INFO_DEVICE ((prof_hsa_agent_info_t)17)
+
+typedef uint32_t prof_hsa_device_type_t;
+#define PROF_HSA_DEVICE_TYPE_GPU ((prof_hsa_device_type_t)1)
+
+typedef uint32_t prof_hsa_symbol_kind_t;
+#define PROF_HSA_SYMBOL_KIND_VARIABLE ((prof_hsa_symbol_kind_t)0)
+
+typedef uint32_t prof_hsa_executable_symbol_info_t;
+#define PROF_HSA_EXECUTABLE_SYMBOL_INFO_TYPE \
+ ((prof_hsa_executable_symbol_info_t)0)
+#define PROF_HSA_EXECUTABLE_SYMBOL_INFO_NAME_LENGTH \
+ ((prof_hsa_executable_symbol_info_t)1)
+#define PROF_HSA_EXECUTABLE_SYMBOL_INFO_NAME \
+ ((prof_hsa_executable_symbol_info_t)2)
+#define PROF_HSA_EXECUTABLE_SYMBOL_INFO_VARIABLE_ADDRESS \
+ ((prof_hsa_executable_symbol_info_t)21)
+
+#define PROF_HSA_EXTENSION_AMD_LOADER ((uint16_t)0x201)
+
+typedef uint32_t prof_hsa_loader_storage_type_t;
+
+typedef struct {
+ prof_hsa_agent_t agent;
+ prof_hsa_executable_t executable;
+ prof_hsa_loader_storage_type_t code_object_storage_type;
+ const void *code_object_storage_base;
+ size_t code_object_storage_size;
+ size_t code_object_storage_offset;
+ const void *segment_base;
+ size_t segment_size;
+} prof_hsa_loader_segment_descriptor_t;
+
+typedef prof_hsa_status_t (*hsa_init_ty)(void);
+typedef prof_hsa_status_t (*hsa_iterate_agents_ty)(
+ prof_hsa_status_t (*)(prof_hsa_agent_t, void *), void *);
+typedef prof_hsa_status_t (*hsa_agent_get_info_ty)(prof_hsa_agent_t,
+ prof_hsa_agent_info_t,
+ void *);
+typedef prof_hsa_status_t (*hsa_executable_iterate_agent_symbols_ty)(
+ prof_hsa_executable_t, prof_hsa_agent_t,
+ prof_hsa_status_t (*)(prof_hsa_executable_t, prof_hsa_agent_t,
+ prof_hsa_executable_symbol_t, void *),
+ void *);
+typedef prof_hsa_status_t (*hsa_executable_symbol_get_info_ty)(
+ prof_hsa_executable_symbol_t, prof_hsa_executable_symbol_info_t, void *);
+typedef prof_hsa_status_t (*hsa_system_get_major_extension_table_ty)(uint16_t,
+ uint16_t,
+ size_t,
+ void *);
+typedef prof_hsa_status_t (*hsa_loader_query_segment_descriptors_ty)(
+ prof_hsa_loader_segment_descriptor_t *, size_t *);
+
+/* First two members of hsa_ven_amd_loader_1_00_pfn_t. Only
+ * query_segment_descriptors is used; query_host_address keeps the offset. */
+typedef struct {
+ void *query_host_address;
+ hsa_loader_query_segment_descriptors_ty query_segment_descriptors;
+} prof_hsa_loader_pfn_t;
+
+static hsa_iterate_agents_ty pHsaIterateAgents = nullptr;
+static hsa_agent_get_info_ty pHsaAgentGetInfo = nullptr;
+static hsa_executable_iterate_agent_symbols_ty pHsaExecIterAgentSyms = nullptr;
+static hsa_executable_symbol_get_info_ty pHsaSymGetInfo = nullptr;
+static hsa_loader_query_segment_descriptors_ty pQuerySegDescs = nullptr;
+
+/* 0 = not yet attempted, 1 = ready, -1 = unavailable. Accessed with acquire/
+ * release atomics: a thread observing HsaRuntimeState==1 (acquire) also sees
+ * the fully-written p* function pointers (published before the release store
+ * of HsaRuntimeState=1 below). */
+static int HsaRuntimeState = 0;
+
+static int setHsaRuntimeState(int S) {
+ __atomic_store_n(&HsaRuntimeState, S, __ATOMIC_RELEASE);
+ return S > 0 ? 0 : -1;
+}
+
+/* Resolve HSA entry points (and the AMD loader extension) once, and confirm
+ * HIP's hipMemcpy is reachable for the device-to-host copies. HIP itself is
+ * resolved by the shared ensureHipLoaded() above. */
+static int loadHsaRuntimePointers(void) {
+ int State = __atomic_load_n(&HsaRuntimeState, __ATOMIC_ACQUIRE);
+ if (State)
+ return State > 0 ? 0 : -1;
+
+ if (!__interception::DynamicLoaderAvailable()) {
+ if (isVerboseMode())
+ PROF_NOTE("%s", "Dynamic library loading not available - "
+ "HSA device profiling disabled\n");
+ return setHsaRuntimeState(-1);
+ }
+
+ void *Hsa = __interception::OpenLibrary("libhsa-runtime64.so");
+ if (!Hsa)
+ Hsa = __interception::OpenLibrary("libhsa-runtime64.so.1");
+ if (!Hsa) {
+ if (isVerboseMode())
+ PROF_NOTE("%s", "libhsa-runtime64.so not loadable - "
+ "HSA device profiling disabled\n");
+ return setHsaRuntimeState(-1);
+ }
+
+ hsa_init_ty pHsaInit =
+ (hsa_init_ty)__interception::LookupSymbol(Hsa, "hsa_init");
+ hsa_system_get_major_extension_table_ty pGetExtTable =
+ (hsa_system_get_major_extension_table_ty)__interception::LookupSymbol(
+ Hsa, "hsa_system_get_major_extension_table");
+ pHsaIterateAgents = (hsa_iterate_agents_ty)__interception::LookupSymbol(
+ Hsa, "hsa_iterate_agents");
+ pHsaAgentGetInfo = (hsa_agent_get_info_ty)__interception::LookupSymbol(
+ Hsa, "hsa_agent_get_info");
+ pHsaExecIterAgentSyms =
+ (hsa_executable_iterate_agent_symbols_ty)__interception::LookupSymbol(
+ Hsa, "hsa_executable_iterate_agent_symbols");
+ pHsaSymGetInfo =
+ (hsa_executable_symbol_get_info_ty)__interception::LookupSymbol(
+ Hsa, "hsa_executable_symbol_get_info");
+
+ if (!pHsaInit || !pGetExtTable || !pHsaIterateAgents || !pHsaAgentGetInfo ||
+ !pHsaExecIterAgentSyms || !pHsaSymGetInfo) {
+ PROF_WARN("%s",
+ "required HSA symbols missing - HSA device profiling disabled\n");
+ return setHsaRuntimeState(-1);
+ }
+
+ /* Bring HSA up (idempotent, refcounted). This runs lazily on the first drain
+ * rather than from the library constructor, so merely loading the
+ * instrumented library does not initialize HSA in the process -- which would
+ * break fork-based callers that deliberately keep HIP/HSA uninitialized in
+ * the parent (see the constructor note at the end of the HSA block). In the
+ * common case the drain runs from the profile write path while HSA is still
+ * alive; if it only runs after HSA's own atexit(hsa_shut_down) has executed,
+ * this simply re-initializes HSA (the process is exiting anyway). */
+ prof_hsa_status_t St = pHsaInit();
+ if (St != PROF_HSA_STATUS_SUCCESS && St != PROF_HSA_STATUS_INFO_BREAK) {
+ if (isVerboseMode())
+ PROF_NOTE("hsa_init failed (0x%x) - HSA device ...
[truncated]
|
Contributor
Author
|
@ronlieb @jhuber6 this adds a supplemental HSA-introspection |
ronlieb
requested review from
jhuber6 and
yxsamliu
and removed request for
yxsamliu
June 14, 2026 19:41
jhuber6
reviewed
Jun 14, 2026
lfmeadow
force-pushed
the
upstream-pgo-hsa-drain
branch
2 times, most recently
from
June 14, 2026 20:27
36178b0 to
89e1d81
Compare
yxsamliu
reviewed
Jun 15, 2026
| #!/usr/bin/env python3 | ||
| """Minimal lit-style runner for the HIP device-PGO tests. | ||
|
|
||
| The compiler-rt profile lit suite (and llvm-lit / FileCheck) is not part of the |
Contributor
There was a problem hiding this comment.
This should not block this PR, but could be a good future direction: is it possible to make these .hip tests part of the normal check-profile lit run, guarded by proper lit features? It seems they would need to be gated on the amdgcn profile runtime, the host ROCm profile runtime, and a non-empty amdgpu-arch result. That would let lit discover them directly while still skipping them on machines that do not have the needed GPU/HIP setup.
Contributor
There was a problem hiding this comment.
We have infra on the pure device side to run lit tests, you can do ninja check-compiler-rt-amdgcn-amd-amdhsa. But that would only really tell you some limited information. The problem is it's a little tough to weave HIP compilation through the test suite.
Contributor
Author
There was a problem hiding this comment.
FWIW once this lands in amd-staging I have another PR with theRock CI that does a much better job testing. I'll defer to @jhuber6 on the difficulty of doing this upstream.
Contributor
There was a problem hiding this comment.
We really need to improve our testing of HIP and friends in upstream LLVM. It would be easier if we had more facilities to actually consume HIP code upstream.
yxsamliu
reviewed
Jun 15, 2026
lfmeadow
force-pushed
the
upstream-pgo-hsa-drain
branch
from
June 15, 2026 21:01
89e1d81 to
2cde2fd
Compare
lfmeadow
added a commit
to lfmeadow/llvm-project
that referenced
this pull request
Jun 15, 2026
The (data, counters, names) dedup table used to compose the host-shadow and HSA-introspection device drains was a fixed 256-entry static array, and entries past the cap were silently dropped. In non-RDC mode the entry count scales like num_code_objects * num_agents, so the cap could be exceeded, dropping a tuple from the dedup set and risking a section being drained twice (double-counted). Replace it with a realloc-backed array that doubles on demand (initial cap 64), matching the existing growth idiom in this file (growPtrArray and the shadow arrays). On allocation failure the existing table is kept and recording is skipped; the worst case is one duplicate profraw record, never a crash. Addresses review feedback on PR llvm#203056. Co-authored-by: Cursor <cursoragent@cursor.com>
lfmeadow
force-pushed
the
upstream-pgo-hsa-drain
branch
from
June 17, 2026 17:03
6933ad0 to
8ee77ce
Compare
Contributor
Author
I did quite a bit of refactoring. I think it looks pretty good now. |
lfmeadow
force-pushed
the
upstream-pgo-hsa-drain
branch
from
June 17, 2026 18:47
8ee77ce to
fc0383b
Compare
lfmeadow
added a commit
to lfmeadow/llvm-project
that referenced
this pull request
Jun 17, 2026
The (data, counters, names) dedup table used to compose the host-shadow and HSA-introspection device drains was a fixed 256-entry static array, and entries past the cap were silently dropped. In non-RDC mode the entry count scales like num_code_objects * num_agents, so the cap could be exceeded, dropping a tuple from the dedup set and risking a section being drained twice (double-counted). Replace it with a realloc-backed array that doubles on demand (initial cap 64), matching the existing growth idiom in this file (growPtrArray and the shadow arrays). On allocation failure the existing table is kept and recording is skipped; the worst case is one duplicate profraw record, never a crash. Addresses review feedback on PR llvm#203056. Co-authored-by: Cursor <cursoragent@cursor.com>
lfmeadow
added a commit
to lfmeadow/llvm-project
that referenced
this pull request
Jun 17, 2026
Remove the device-profile-runtime forwarding added to AMDGCN::Linker::constructLldCommand (and its VirtualFileSystem.h include). HIP defaults to the new offload driver (UseNewOffloadingDriver is true whenever an offload kind is active), so the device link goes through clang-linker-wrapper / LinkerWrapper::ConstructJob, which forwards -fprofile* and links the device profile runtime via addProfileRTLibs. constructLldCommand is only reached under --no-offload-new-driver, the legacy path that is being deprecated. Per review on PR llvm#203056, drop this so the PR carries no driver changes; device PGO relies on the default (new-driver) link path. Co-authored-by: Cursor <cursoragent@cursor.com>
lfmeadow
added a commit
to lfmeadow/llvm-project
that referenced
this pull request
Jun 17, 2026
The (data, counters, names) dedup table used to compose the host-shadow and HSA-introspection device drains was a fixed 256-entry static array, and entries past the cap were silently dropped. In non-RDC mode the entry count scales like num_code_objects * num_agents, so the cap could be exceeded, dropping a tuple from the dedup set and risking a section being drained twice (double-counted). Replace it with a realloc-backed array that doubles on demand (initial cap 64), matching the existing growth idiom in this file (growPtrArray and the shadow arrays). On allocation failure the existing table is kept and recording is skipped; the worst case is one duplicate profraw record, never a crash. Addresses review feedback on PR llvm#203056. Co-authored-by: Cursor <cursoragent@cursor.com>
lfmeadow
added a commit
to lfmeadow/llvm-project
that referenced
this pull request
Jun 17, 2026
Remove the device-profile-runtime forwarding added to AMDGCN::Linker::constructLldCommand (and its VirtualFileSystem.h include). HIP defaults to the new offload driver (UseNewOffloadingDriver is true whenever an offload kind is active), so the device link goes through clang-linker-wrapper / LinkerWrapper::ConstructJob, which forwards -fprofile* and links the device profile runtime via addProfileRTLibs. constructLldCommand is only reached under --no-offload-new-driver, the legacy path that is being deprecated. Per review on PR llvm#203056, drop this so the PR carries no driver changes; device PGO relies on the default (new-driver) link path. Co-authored-by: Cursor <cursoragent@cursor.com>
lfmeadow
force-pushed
the
upstream-pgo-hsa-drain
branch
from
June 17, 2026 18:52
fc0383b to
f01d7b2
Compare
jhuber6
reviewed
Jun 17, 2026
jhuber6
left a comment
jhuber6
left a comment
Contributor
There was a problem hiding this comment.
Much better, thanks. Can you clean up some of the comments? Most of them are full paragraphs.
| #define PROF_HSA_STATUS_SUCCESS ((prof_hsa_status_t)0x0) | ||
| #define PROF_HSA_STATUS_INFO_BREAK ((prof_hsa_status_t)0x1) | ||
|
|
||
| typedef struct { |
Contributor
There was a problem hiding this comment.
We really need to think about unifying this stuff now that offload/, asan/ and profile/ will now all maintain their own versions of it.
Contributor
Author
There was a problem hiding this comment.
Perhaps a follow-on PR?
Contributor
Author
Done. |
…O tests
The host-shadow device-profile drain (InstrProfilingPlatformROCm.cpp) can only
collect device counters for kernels that registered a host-side shadow via
__hipRegisterVar. Device-linked programs (e.g. RCCL) link the instrumented code
object directly into the device image with no host shadow, so their counters are
never drained.
Add a supplemental, Linux-only drain that introspects the loaded code objects via
the HSA runtime: it walks each GPU agent, enumerates only the code objects
actually resident there, reads each one's __llvm_profile_sections, and routes
them through the existing processDeviceOffloadPrf() path so the emitted profraw
layout is identical. A small content-dedup set keyed on the
(data, counters, names) device-pointer triple ensures a section already drained
by the host-shadow pass is not drained again, so the two passes compose without
double-counting. HSA is brought up lazily from the drain (never from a library
constructor) to avoid poisoning fork-based callers.
Because the HSA walk only ever touches resident code objects, it also makes the
host-shadow pass's collect-all fallback unnecessary on Linux: when no kernel
launch was tracked (a program that never launches, collects before its first
launch, or launches only via an untracked API), the host-shadow pass is skipped
and the HSA drain covers it safely instead of faulting/hanging on a non-resident
device on a multi-GPU host.
Link the device profile runtime on both the new-offload-driver
(LinkerWrapper::ConstructJob) and traditional (HIPAMD constructLldCommand) link
paths so instrumented device images resolve the runtime symbols.
On the host side, link clang_rt.profile_rocm -- the self-contained runtime
variant that carries the device-counter drain and the hipModuleLoad interceptor
-- for any instrumented host link on a ROCm-equipped toolchain, not only for
links with an active HIP offload action (OFK_HIP). HIP host code is frequently
linked into a shared library or executable from pre-compiled objects (e.g.
RCCL's librccl.so is linked from .o inputs by a plain clang++ -shared); such a
link carries no OFK_HIP yet still needs the drain, and gating on it left those
.profraw files with host counters only. profile_rocm is emitted ahead of the
base clang_rt.profile (which stays inert), guarded by an existence check that
leaves lean toolchains unchanged, and both the interceptor and the drain
self-skip when the process has no resident device code.
Also add a GPU-executed test suite (compiler-rt/test/profile/{GPU,AMDGPU}/*.hip)
and a dependency-free "lit-lite" runner (run_gpu_tests.py) so the device drain
can be exercised on a real AMD GPU runner: basic/coverage/pgo-use, multi-kernel,
device-branching, multi-GPU and non-default-device drain + dedup, early-collect /
no-kernel edges, RDC vs non-RDC __llvm_profile_sections, fork-safety (the RCCL
parent-no-HIP / kernel-in-forked-child pattern), quantitative device-counter
correctness, multi-process offline accumulation, and explicit-collect
idempotency. A standalone device-pgo/ build helper reproduces the toolchain
locally. The object-only host-link path is exercised by
clang/test/Driver/hip-profile-rocm-runtime.hip.
Co-authored-by: Cursor <cursoragent@cursor.com>
The (data, counters, names) dedup table used to compose the host-shadow and HSA-introspection device drains was a fixed 256-entry static array, and entries past the cap were silently dropped. In non-RDC mode the entry count scales like num_code_objects * num_agents, so the cap could be exceeded, dropping a tuple from the dedup set and risking a section being drained twice (double-counted). Replace it with a realloc-backed array that doubles on demand (initial cap 64), matching the existing growth idiom in this file (growPtrArray and the shadow arrays). On allocation failure the existing table is kept and recording is skipped; the worst case is one duplicate profraw record, never a crash. Addresses review feedback on PR llvm#203056. Co-authored-by: Cursor <cursoragent@cursor.com>
Remove the device-profile-runtime forwarding added to AMDGCN::Linker::constructLldCommand (and its VirtualFileSystem.h include). HIP defaults to the new offload driver (UseNewOffloadingDriver is true whenever an offload kind is active), so the device link goes through clang-linker-wrapper / LinkerWrapper::ConstructJob, which forwards -fprofile* and links the device profile runtime via addProfileRTLibs. constructLldCommand is only reached under --no-offload-new-driver, the legacy path that is being deprecated. Per review on PR llvm#203056, drop this so the PR carries no driver changes; device PGO relies on the default (new-driver) link path. Co-authored-by: Cursor <cursoragent@cursor.com>
Move the Linux-only supplemental HSA-introspection drain out of InstrProfilingPlatformROCm.cpp into InstrProfilingPlatformROCmHSA.cpp, with a private InstrProfilingPlatformROCmInternal.h declaring the shared __prof_rocm interface (HIP helpers, processDeviceOffloadPrf, UniqueFree, and the drained-bounds dedup hooks). The new file is guarded by __linux__ && !_WIN32 and compiles to an empty TU elsewhere; there is no Windows HSA path (Windows uses only the host-shadow HIP drain). No behavior change; wired into CMake and the Bazel overlay. Co-authored-by: Cursor <cursoragent@cursor.com>
…ision Trim the verbose, AI-flavored comments in the supplemental HSA drain down to concise intent comments, and clarify that the shared dedup table relies on device collection being single-threaded (the acquire/release atomics only guard the drain flags, not SeenBounds). Fix a device-profraw filename collision: the host-shadow path and the HSA pass could both emit the bare `arch` target for two distinct code objects on the same arch, truncating each other's file. HSA-drained objects now use a separate `.hsaN` suffix space, which cannot collide with the host path's `arch`/`arch.<i>` names. HSA idempotency is already provided by the HsaDrainCompleted latch, so the host-shadow path keeps its stable per-shadow names (overwrite-on-repeat), which explicit-collect idempotency depends on. Also drop the unused InstrProfilingInternal.h include from both ROCm TUs and silence misc-use-internal-linkage on the INTERCEPTOR-generated real_* pointers (which must keep external linkage). Validated on 2x MI210 (gfx90a): run_gpu_tests.py GPU+AMDGPU = 15 passed. Co-authored-by: Cursor <cursoragent@cursor.com>
Factor the host-shadow drain body out of __llvm_profile_hip_collect_device_data into collectHostShadowData(). The collect entry point now just gates the host-shadow pass and runs the supplemental HSA drain, flattening the previous four-level nesting and removing the whole-block reindentation. No behavior change. Validated on 2x MI210 (gfx90a): run_gpu_tests.py GPU+AMDGPU = 15 passed. Co-authored-by: Cursor <cursoragent@cursor.com>
…aders HSA is dlopened, not linked, so the drain hand-mirrors the few HSA declarations it needs. Move that block out of InstrProfilingPlatformROCmHSA.cpp into a dedicated InstrProfilingPlatformROCmHSADefs.h to confine the maintenance surface, mirroring offload's dynamic_hsa/ approach. Add an opt-in build-time drift guard: when the real ROCm headers are available (find_package(hsa-runtime64) in CMakeLists.txt defines PROFILE_VERIFY_HSA_ABI and adds the include dir), static_asserts cross-check every mirrored enum value and the loader-segment-descriptor / pfn-table layout against hsa/hsa.h and hsa/hsa_ven_amd_loader.h. It is never a build requirement -- on hosts without ROCm the checks are skipped and the mirror stands alone. Linux only. Validated on 2x MI210 (gfx90a): the HSA TU built with -DPROFILE_VERIFY_HSA_ABI against ROCm 7.1 headers (asserts pass), run_gpu_tests.py GPU+AMDGPU = 15 passed. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the four hand-rolled realloc/doubling sites (TU list, dynamic modules, shadow variables, and the section-shadow groups/shadows) and the old void*-only growPtrArray with a single type-erased growArray helper. It doubles from an initial capacity, zeroes the newly grown tail, and on OOM leaves the existing array intact, preserving the no-crash semantics. Co-authored-by: Cursor <cursoragent@cursor.com>
…HSA drain Move the growable-array helper into InstrProfilingPlatformROCmInternal.h as a single inline definition in __prof_rocm so both drains share one copy, and route the HSA pass's SeenBounds dedup table through it, replacing the last hand-rolled realloc/doubling block. OOM semantics (keep the old table, skip recording) are unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
…atus shorthands - Replace the fixed kMaxPairs=512 (agent, executable) seen-list with a growArray-backed dynamic array, so dedup never silently caps out; SeenBounds remains the backstop if a grow fails under OOM. - Introduce PROF_HSA_NAME_MAX for the agent/symbol name buffers in place of the bare 64s. - Add hsaOk()/hsaOkOrBreak() shorthands (mirroring the thin HIP wrappers) and use them for the repeated prof_hsa_status_t checks. Co-authored-by: Cursor <cursoragent@cursor.com>
The GPU/ and AMDGPU/ device-profile tests were driven by a hand-rolled "lit-lite" Python runner (run_gpu_tests.py) plus a device-pgo/ build recipe, because upstream lit had no wiring for them. That capability already exists in lit, so this removes the custom runner and wires the tests in properly: - lit.cfg.py adds the .hip suffix and self-configuring detection: it runs amdgpu-arch (next to the test clang) and probes for libamdhip64, then exposes the hip/amdgpu/multi-device features and the %amdgpu_arch and %hip_lib_path substitutions the tests use. Both are overridable via --param amdgpu_arch=... / --param hip_lib_path=.... - GPU/ and AMDGPU/ lit.local.cfg.py gate on hip+amdgpu (so they report UNSUPPORTED rather than fail on a GPU-less host, e.g. upstream CI) and join a size-1 'gpu' parallelism group so they do not contend on the device. Removes run_gpu_tests.py and the device-pgo/ build recipe. No CMake or lit config referenced them. Co-authored-by: Cursor <cursoragent@cursor.com>
The device drain logic had no coverage that runs without a GPU, so the host-side bookkeeping (which is where the multi-GPU "drain each counter set once" correctness lives) was untested in upstream CI. Extract the section-bounds dedup table into a small, header-only ProfBoundsSet (contains/record over (data, counters, names) tuples, backed by growArray) in the internal header, and use it from the HSA drain in place of the file-static array and hand-rolled helpers. Add two pure host lit tests under test/profile/ (no REQUIRES, so they run anywhere the profile runtime is tested, including upstream CI): - instrprof-rocm-grow-array.cpp: growArray doubling, tail zeroing, content preservation, and element-size byte math. - instrprof-rocm-bounds-dedup.cpp: ProfBoundsSet idempotent record, full three-field key (guards against subset-key dedup dropping a counter set), growth past the initial capacity, and null-tuple keys. Co-authored-by: Cursor <cursoragent@cursor.com>
Address PR review feedback on comment verbosity: condense the explanatory comments in the HSA drain and its shared internal/ABI-mirror headers while keeping the essential intent. Also drop the redundant `!defined(_WIN32)` from the `__linux__` guards (never defined together on Linux). No functional change. Co-authored-by: Cursor <cursoragent@cursor.com>
lfmeadow
force-pushed
the
upstream-pgo-hsa-drain
branch
from
June 22, 2026 04:55
f01d7b2 to
6802b9d
Compare
jhuber6
approved these changes
Jun 22, 2026
This was referenced Jun 25, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #202095 (now landed). #202095's host-shadow device-profile drain can
only collect device counters for kernels that registered a host-side shadow via
__hipRegisterVar. Device-linked programs (e.g. RCCL), whose instrumented codeobjects are linked directly into the device image with no host shadow, are never
drained.
This adds a supplemental, Linux-only HSA-introspection drain that runs after
the host-shadow drain: it walks each GPU agent, enumerates only the code objects
actually resident there, reads each one's
__llvm_profile_sectionstable on thedevice, and routes them through the existing
processDeviceOffloadPrf()path sothe emitted
.profrawlayout is identical. A content-dedup set keyed on the(data, counters, names)device-pointer triple ensures a section already drainedby the host-shadow pass is not drained twice, so the two passes compose without
double-counting.
It is purely additive — it does not modify #202095's host-shadow drain or its
launch-tracking. Highlights:
compiler-rt/lib/profile/InstrProfilingPlatformROCmHSA.cpp: HSA agent/segment/symbol walk + dedup; record drained bounds after each host-shadow drain; lazy
HSA init (no library constructor, for fork-safety).
host-shadow drain's collect-all fallback on Linux. When no kernel launch was
tracked (program never launches, collects before its first launch, or launches
only via an untracked API), the host-shadow pass is skipped and the HSA drain
covers it safely — instead of faulting/hanging reading a non-resident device on
a multi-GPU host. This also closes the silent-data-loss gap for untracked launch
APIs (
hipExtLaunchKernel, cooperative/graph launches).clang/lib/Driver/ToolChains/Clang.cpp/HIPAMD.cpp: link the device profileruntime on both the new-offload-driver (
LinkerWrapper::ConstructJob) andtraditional (
lld) link paths, guarded byneedsProfileRT+ VFS existence.hip/amdgpufeatures(auto-detected from the toolchain + a visible GPU) so they report UNSUPPORTED
rather than fail when no GPU is present. Plus GPU-free host unit tests for the
device-profile host helpers that run everywhere, including upstream CI.
Test plan
gfx90a:sramecc+:xnack-), ROCm 7.1.llvm-lit -sv <build>/.../compiler-rt/test/profile/Profile-x86_64over theGPU/andAMDGPU/subdirectories. The profilelit.cfg.pyauto-detects avisible GPU (
amdgpu-arch) and the HIP runtime (libamdhip64) and exposes thehip/amdgpu/multi-devicefeatures and the%amdgpu_arch/%hip_lib_pathsubstitutions; both are overridable via
--param amdgpu_arch=… / hip_lib_path=….multiple-kernels, device-branching, multi-gpu and non-default-device drain,
early-collect / no-kernel edges, RDC vs non-RDC
__llvm_profile_sections,dedup (host-shadow drains the used device, HSA finds it and dedups), and
fork-safety (the RCCL parent-no-HIP / kernel-in-forked-child pattern).
UNSUPPORTED instead of failing, and the GPU subdirectories serialize via a
size-1
gpulit parallelism group when they do run.CI):
instrprof-rocm-grow-array.cpp(the dynamic-array helper) andinstrprof-rocm-bounds-dedup.cpp(the(data, counters, names)dedup tablethat backs the "drain each counter set once" guarantee).
clang-formatclean.