[PGO][HIP] Decouple device profile drain via HSA introspection

clang/docs/SourceBasedCodeCoverage.rst

@@ -426,6 +426,107 @@ To specify an alternate directory for raw profiles, use
 ``-DLLVM_PROFILE_DATA_DIR``. To change the size of the profile merge pool, use
 ``-DLLVM_PROFILE_MERGE_POOL_SIZE``.
 
+HIP / AMDGPU device code coverage
+=================================
+
+Source-based coverage works for HIP device code (GPU kernels) in addition to
+host code. Device counters live in GPU memory inside separately-compiled code
+objects, so an extra device-side profile runtime
+(``libclang_rt.profile-amdgcn.a``) is required, and the device counters are
+written out as their own ``.profraw`` files at process exit.
+
+Building the device profile runtime
+------------------------------------
+
+The device runtime is a ``compiler-rt`` build cross-compiled to
+``amdgcn-amd-amdhsa`` **using the just-built host clang** (the device runtime
+and the instrumentation the compiler emits share a versioned ABI, so they must
+come from the same toolchain). Configure ``compiler-rt`` standalone with the
+profile library enabled for the ROCm/AMDGPU target:
+
+.. code-block:: console
+
+    # $STAGE1 is the bin/ dir of a clang+lld build you have already produced
+    % cmake -G Ninja -S compiler-rt -B build-amdgcn-rt \
+        -DCMAKE_C_COMPILER="$STAGE1/clang" \
+        -DCMAKE_CXX_COMPILER="$STAGE1/clang++" \
+        -DCMAKE_C_COMPILER_TARGET=amdgcn-amd-amdhsa \
+        -DCMAKE_CXX_COMPILER_TARGET=amdgcn-amd-amdhsa \
+        -DLLVM_CONFIG_PATH="$STAGE1/llvm-config" \
+        -DCOMPILER_RT_DEFAULT_TARGET_ARCH=amdgcn \
+        -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
+        -DCOMPILER_RT_BUILD_PROFILE=ON \
+        -DCOMPILER_RT_BUILD_PROFILE_ROCM=ON \
+        -DCOMPILER_RT_BUILD_BUILTINS=OFF \
+        -DCOMPILER_RT_BUILD_SANITIZERS=OFF
+    % ninja -C build-amdgcn-rt
+
+Then install the resulting archive into the host clang's resource directory so
+the driver can find it automatically. It must go in the per-target runtime
+directory (``lib/<device-triple>/libclang_rt.profile.a``, without the
+``-amdgcn`` suffix) -- that is the path the driver resolves for the device
+toolchain when forwarding the profile runtime to the offload device link:
+
+.. code-block:: console
+
+    % RESDIR=$($STAGE1/clang -print-resource-dir)
+    % mkdir -p "$RESDIR/lib/amdgcn-amd-amdhsa"
+    % cp build-amdgcn-rt/lib/linux/libclang_rt.profile-amdgcn.a \
+         "$RESDIR/lib/amdgcn-amd-amdhsa/libclang_rt.profile.a"
+
+Compiling, running, and reporting
+---------------------------------
+
+Compile HIP code with the usual coverage flags. For a HIP translation unit the
+driver links both the host and device profile runtimes automatically:
+
+.. code-block:: console
+
+    % clang -x hip --offload-arch=gfx90a -fno-gpu-rdc \
+        -fprofile-instr-generate -fcoverage-mapping \
+        -o myapp my_kernel.hip -lamdhip64
+
+At process exit a library constructor in the host profile runtime installs an
+``atexit`` handler that walks every loaded HSA code object on every GPU agent,
+copies the device counters back to the host, and writes them out. A run
+produces:
+
+* ``<name>.<pid>.profraw`` — host counters (from ``LLVM_PROFILE_FILE``).
+* ``gfx<arch>.<name>.<pid>.profraw`` — device counters, arch-prefixed, one per
+  loaded code object.
+
+Merge and report as usual, but note that the **device** report must be rendered
+against the device ELF (which carries the device coverage map), not the host
+executable:
+
+.. code-block:: console
+
+    % llvm-profdata merge gfx*.profraw -o device.profdata
+    % llvm-cov report ./my_kernel.amdgcn.elf -instr-profile=device.profdata
+
+Hosts that are not compiled as HIP
+----------------------------------
+
+The driver only force-links the device drain for HIP host links (``-x hip``, or
+a ``.hip`` / ``.cu`` input). A plain C++ host that loads device code at runtime
+(``hipModuleLoad`` / ``hsa_executable_*``) must force-link the drain object out
+of the static archive manually:
+
+.. code-block:: console
+
+    % clang -fprofile-instr-generate -fcoverage-mapping \
+        -Wl,-u,__llvm_profile_hip_collect_device_data \
+        -o host_app host.cc -lamdhip64
+
+Device-side limitations
+-----------------------
+
+* The device drain runs only at ``atexit``; device counters are lost if the
+  process is terminated by a fatal signal (e.g. ``abort()``) or if a code
+  object is unloaded mid-run.
+* Coverage is collected per loaded code object across all GPU agents; an
+  uninstrumented host is fine, but counters are only flushed at a clean exit.
+
 Drawbacks and limitations
 =========================
 

clang/lib/CodeGen/CGCUDANV.cpp

@@ -74,11 +74,6 @@ class CGNVCUDARuntime : public CGCUDARuntime {
   /// ModuleCtorFunction() and used to create corresponding cleanup calls in
   /// ModuleDtorFunction()
   llvm::GlobalVariable *GpuBinaryHandle = nullptr;
-  /// Host-side shadow for the per-TU __llvm_profile_sections_<CUID> global,
-  /// emitted only for HIP host compiles when PGO is on. Registered via
-  /// __hipRegisterVar (non-RDC) or an offloading entry (RDC) so the runtime
-  /// can locate the device-side table by name.
-  llvm::GlobalVariable *OffloadProfShadow = nullptr;
   /// Whether we generate relocatable device code.
   bool RelocatableDeviceCode;
   /// Mangle context for device.
@@ -183,13 +178,6 @@ class CGNVCUDARuntime : public CGCUDARuntime {
   void transformManagedVars();
   /// Create offloading entries to register globals in RDC mode.
   void createOffloadingEntries();
-  /// For HIP+PGO, emit the per-TU __llvm_profile_sections_<CUID> global.
-  /// On the device side it is the populated 7-pointer section-bounds table.
-  /// On the host side it is a placeholder void* shadow stored in
-  /// OffloadProfShadow, registered later by makeRegisterGlobalsFn (non-RDC)
-  /// or createOffloadingEntries (RDC) so the runtime can locate the
-  /// device-side table by name.
-  void emitOffloadProfilingSections();
 
 public:
   CGNVCUDARuntime(CodeGenModule &CGM);
@@ -749,32 +737,6 @@ llvm::Function *CGNVCUDARuntime::makeRegisterGlobalsFn() {
     }
   }
 
-  // Register the per-TU offload-profiling shadow so the host runtime can
-  // locate the matching device-side __llvm_profile_sections_<CUID>. We
-  // emit both __hipRegisterVar (so the HIP runtime can map the host
-  // shadow to the device symbol) and
-  // __llvm_profile_offload_register_shadow_variable (so the profile
-  // runtime adds the shadow to its drain list).
-  if (OffloadProfShadow) {
-    llvm::Constant *Name =
-        makeConstantString(std::string(OffloadProfShadow->getName()));
-    llvm::Value *RegisterVarArgs[] = {
-        &GpuBinaryHandlePtr,
-        OffloadProfShadow,
-        Name,
-        Name,
-        llvm::ConstantInt::get(IntTy, /*Extern=*/0),
-        llvm::ConstantInt::get(VarSizeTy, CGM.getDataLayout().getPointerSize()),
-        llvm::ConstantInt::get(IntTy, /*Constant=*/0),
-        llvm::ConstantInt::get(IntTy, 0)};
-    Builder.CreateCall(RegisterVar, RegisterVarArgs);
-
-    llvm::FunctionCallee RegisterShadow = CGM.CreateRuntimeFunction(
-        llvm::FunctionType::get(VoidTy, {PtrTy}, false),
-        "__llvm_profile_offload_register_shadow_variable");
-    Builder.CreateCall(RegisterShadow, {OffloadProfShadow});
-  }
-
   Builder.CreateRetVoid();
   return RegisterKernelsFunc;
 }
@@ -1299,124 +1261,11 @@ void CGNVCUDARuntime::createOffloadingEntries() {
           I.Flags.getSurfTexType());
     }
   }
-
-  // Register the per-TU offload-profiling shadow. The offloading entry
-  // makes the linker-wrapper emit the host __hipRegisterVar call in the
-  // combined ctor. Separately emit a per-TU ctor that registers the
-  // shadow with the profile runtime's drain list.
-  if (OffloadProfShadow) {
-    llvm::offloading::emitOffloadingEntry(
-        M, Kind, OffloadProfShadow, OffloadProfShadow->getName(),
-        CGM.getDataLayout().getPointerSize(),
-        llvm::offloading::OffloadGlobalEntry, /*Data=*/0);
-
-    llvm::LLVMContext &Ctx = M.getContext();
-    auto *PtrTy = llvm::PointerType::getUnqual(Ctx);
-    llvm::FunctionCallee RegisterShadow = CGM.CreateRuntimeFunction(
-        llvm::FunctionType::get(VoidTy, {PtrTy}, false),
-        "__llvm_profile_offload_register_shadow_variable");
-    auto *CtorFn = llvm::Function::Create(
-        llvm::FunctionType::get(VoidTy, false),
-        llvm::GlobalValue::InternalLinkage,
-        "__llvm_profile_register_shadow." + CGM.getContext().getCUIDHash(), &M);
-    auto *Entry = llvm::BasicBlock::Create(Ctx, "entry", CtorFn);
-    llvm::IRBuilder<> B(Entry);
-    B.CreateCall(RegisterShadow, {OffloadProfShadow});
-    B.CreateRetVoid();
-    llvm::appendToGlobalCtors(M, CtorFn, /*Priority=*/65535);
-  }
-}
-
-// For HIP host+device compiles with PGO enabled, emit the per-TU global
-// __llvm_profile_sections_<CUID>. Device side: a 7-pointer struct holding
-// section start/stop bounds for the names/counters/data sections plus the
-// raw-version variable. Host side: an opaque void* shadow whose only
-// purpose is to give the host-runtime a registered symbol name to look up
-// via hipGetSymbolAddress; the actual device-side data lives in the
-// matching device-side global.
-void CGNVCUDARuntime::emitOffloadProfilingSections() {
-  if (!CGM.getLangOpts().HIP)
-    return;
-  if (!CGM.getCodeGenOpts().hasProfileInstr())
-    return;
-
-  StringRef CUIDHash = CGM.getContext().getCUIDHash();
-  if (CUIDHash.empty())
-    return;
-
-  llvm::Module &M = CGM.getModule();
-  llvm::LLVMContext &Ctx = M.getContext();
-  std::string Name = ("__llvm_profile_sections_" + CUIDHash).str();
-
-  // If the global already exists (e.g. another TU was merged in), don't
-  // duplicate it.
-  if (M.getNamedValue(Name))
-    return;
-
-  if (CGM.getLangOpts().CUDAIsDevice) {
-    // Device side: emit the populated struct. Section start/stop symbols
-    // are linker-defined (ELF auto-generates __start_/__stop_ for any
-    // section whose name is a valid C identifier; AMDGPU is ELF).
-    unsigned GlobalAS = M.getDataLayout().getDefaultGlobalsAddressSpace();
-    auto *PtrTy = llvm::PointerType::get(Ctx, GlobalAS);
-    auto getOrDeclare = [&](StringRef SymName) {
-      if (auto *GV = M.getNamedGlobal(SymName))
-        return GV;
-      auto *GV = new llvm::GlobalVariable(
-          M, llvm::Type::getInt8Ty(Ctx), /*isConstant=*/false,
-          llvm::GlobalValue::ExternalLinkage, /*Initializer=*/nullptr, SymName,
-          /*InsertBefore=*/nullptr, llvm::GlobalValue::NotThreadLocal,
-          GlobalAS);
-      GV->setVisibility(llvm::GlobalValue::HiddenVisibility);
-      return GV;
-    };
-    auto *VersionGV = M.getNamedGlobal("__llvm_profile_raw_version");
-    if (!VersionGV) {
-      VersionGV = new llvm::GlobalVariable(
-          M, llvm::Type::getInt64Ty(Ctx), /*isConstant=*/true,
-          llvm::GlobalValue::ExternalLinkage, /*Initializer=*/nullptr,
-          "__llvm_profile_raw_version",
-          /*InsertBefore=*/nullptr, llvm::GlobalValue::NotThreadLocal,
-          GlobalAS);
-    }
-
-    auto *StructTy = llvm::StructType::get(
-        Ctx, {PtrTy, PtrTy, PtrTy, PtrTy, PtrTy, PtrTy, PtrTy});
-    llvm::Constant *Fields[] = {
-        getOrDeclare("__start___llvm_prf_names"),
-        getOrDeclare("__stop___llvm_prf_names"),
-        getOrDeclare("__start___llvm_prf_cnts"),
-        getOrDeclare("__stop___llvm_prf_cnts"),
-        getOrDeclare("__start___llvm_prf_data"),
-        getOrDeclare("__stop___llvm_prf_data"),
-        VersionGV,
-    };
-    auto *Init = llvm::ConstantStruct::get(StructTy, Fields);
-    auto *GV = new llvm::GlobalVariable(
-        M, StructTy, /*isConstant=*/true, llvm::GlobalValue::ExternalLinkage,
-        Init, Name, /*InsertBefore=*/nullptr, llvm::GlobalValue::NotThreadLocal,
-        GlobalAS);
-    GV->setVisibility(llvm::GlobalValue::ProtectedVisibility);
-    CGM.addCompilerUsedGlobal(GV);
-    return;
-  }
-
-  // Host side: emit an opaque void* shadow. Layout doesn't matter — the
-  // runtime locates it by name via hipGetSymbolAddress and treats it as
-  // the address of the device-side struct. Registration with the HIP
-  // runtime is added by makeRegisterGlobalsFn (non-RDC) or
-  // createOffloadingEntries (RDC).
-  auto *PtrTy = llvm::PointerType::getUnqual(Ctx);
-  OffloadProfShadow = new llvm::GlobalVariable(
-      M, PtrTy, /*isConstant=*/false, llvm::GlobalValue::ExternalLinkage,
-      llvm::ConstantPointerNull::get(PtrTy), Name);
-  CGM.addCompilerUsedGlobal(OffloadProfShadow);
 }
 
 // Returns module constructor to be added.
 llvm::Function *CGNVCUDARuntime::finalizeModule() {
   transformManagedVars();
-  emitOffloadProfilingSections();
   if (CGM.getLangOpts().CUDAIsDevice) {
     // Mark ODR-used device variables as compiler used to prevent it from being
     // eliminated by optimization. This is necessary for device variables

clang/lib/Driver/ToolChains/Clang.cpp

@@ -9871,6 +9871,21 @@ void LinkerWrapper::ConstructJob(Compilation &C, const JobAction &JA,
           (TC->getTriple().isAMDGPU() || TC->getTriple().isNVPTX()))
         LinkerArgs.emplace_back("-lompdevice");
 
+      // With PGO/coverage instrumentation, GPU device code references the
+      // device profile runtime (__llvm_profile_instrument_gpu and the
+      // __llvm_profile_sections bounds table emitted by
+      // InstrProfilingPlatformGPU). The offload device link does not otherwise
+      // pull it in, so forward the static device profile runtime to the GPU
+      // device linker. The archive is arch-suffixed, so pass its full path
+      // rather than a -l name.
+      if (ToolChain::needsProfileRT(Args) &&
+          (TC->getTriple().isAMDGPU() || TC->getTriple().isNVPTX())) {
+        std::string ProfileRT =
+            TC->getCompilerRT(Args, "profile", ToolChain::FT_Static);
+        if (TC->getVFS().exists(ProfileRT))
+          LinkerArgs.emplace_back(Args.MakeArgString(ProfileRT));
+      }
+
       // For SPIR-V, pass some extra flags to `spirv-link`, the out-of-tree
       // SPIR-V linker. `spirv-link` isn't called in LTO mode so restrict these
       // flags to normal compilation.

clang/lib/Driver/ToolChains/Gnu.cpp

@@ -463,6 +463,16 @@ void tools::gnutools::Linker::ConstructJob(Compilation &C, const JobAction &JA,
   // The profile runtime also needs access to system libraries.
   getToolChain().addProfileRTLibs(Args, CmdArgs);
 
+  // For HIP host links built with PGO, force the device-side profile drain
+  // object (InstrProfilingPlatformROCm.o, defining
+  // __llvm_profile_hip_collect_device_data) into the link. Its atexit handler
+  // collects device counters via HSA introspection; it is otherwise
+  // unreferenced because the host no longer emits any per-TU offload-profiling
+  // shadow.
+  if ((C.getActiveOffloadKinds() & Action::OFK_HIP) &&
+      ToolChain::needsProfileRT(Args))
+    CmdArgs.push_back("-u__llvm_profile_hip_collect_device_data");
+
   if (D.CCCIsCXX() &&
       !Args.hasArg(options::OPT_nostdlib, options::OPT_nodefaultlibs,
                    options::OPT_r)) {