Repository navigation
rocmPackages_10: add profiling and debugging tools, RCCL - #579517
Draft
randomizedcoder wants to merge 33 commits into
Draft
randomizedcoder wants to merge 33 commits into
randomizedcoder wants to merge 33 commits into
Conversation
Prepare the update script for a rocmPackages_10 set built from the TheRock monorepos, whose sources are tagged therock-X.Y[.Z]: - maxMajor (default 7) replaces the hard-coded major cap, so a second set can call it with its own cap. - tagPrefixes (default [ "rocm" ]) selects which tag families count as releases. The current set keeps rocm- only, so the therock-7.x preview tags are not picked up. - setName (default "rocmPackages") is the attribute path passed to update-source-version. - Pick the highest matching version instead of the first release listed: releases are not ordered by version (a rocm-7.2.x maintenance release can be published after a newer series). - Follow GitHub's pagination (100 per page) so the highest version is found even when it is not on the first page, and fail on fetch errors instead of reporting "no valid version". No change to any derivation; passthru.updateScript only. Assisted-by: Claude Code (claude-opus-5-5)
(cherry picked from commit ab348e2)
Add an empty rocmPackages_10 scope next to rocmPackages, sharing its structure (rocm-modules/10/default.nix is derived from rocm-modules/default.nix). The following PRs add the ROCm 10.1 packages to it tier by tier; a final PR on staging switches rocmPackages over. Following the rocmPackages_5/_6 layout. It is deliberately not wrapped in recurseIntoAttrs, so Hydra and the CI rebuild count are unaffected until the switch. The update script runs with maxMajor = 10, the therock- tag prefix, and setName = "rocmPackages_10". Assisted-by: Claude Code (claude-opus-5-5)
Verbatim copies of the rocmPackages (ROCm 7.2.4) expressions for llvm, rocm-device-libs, not yet part of the rocmPackages_10 scope. The following commits update each one to ROCm 10.1, so their diffs read against 7.2.4. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.llvm copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.llvm: 7.2.4 -> 10.1.0, build LLVM 24 on the llvmPackages_23 base": ROCm 10.1's llvm-project fork is LLVM 24 (LLVMVersion.cmake: LLVM_VERSION_MAJOR 24), sourced from the therock-10.1 tag. nixpkgs has no released llvmPackages_24, and the 24.0.0-git snapshot (llvmPackages_git) is an alias reserved for testing whose package-set splice resolves through that alias, so it cannot be evaluated with allowAliases = false (as CI does). Keep the released llvmPackages_23 as the base: it only supplies the shared compilers/llvm/common packaging and the bootstrap clang/bintools, while rocmLlvmVersion = "24.0.0-rocm" selects the LLVM 24 patch set for ROCm's own source. LTO is off, so the base's bitcode version is moot. Toolchain changes for the LLVM 22 -> 24 jump: - Drop the LLVM 22 backports (select-ptr-combine, arm64e_x1 bridge, v64i8 VBMI shuffle, scan-build-py option): all upstream in LLVM 24. - Drop the CompressedOffloadBundle::decompress postPatch and the EmissaryIds postUnpack symlink: ROCm 10.1 uses the raw_ostream* decompress form and removed CGEmitEmissaryExec.cpp/EmissaryIds.h (llvm#175265). - Replace the GZGavinZhao GetBundleIDsInFile fetchpatch (no longer applies) with a locally rebased clang-add-getbundleidsinfile.diff; it adds the set-returning OffloadBundler::GetBundleIDsInFile the comgr ISA-compat patch calls. - Drop clang-log-jobs.diff: LLVM 24 rewrote Compilation::ExecuteJobs around parallel offload job groups, so the patched loop no longer exists; it was cosmetic CI-progress logging. - Put zstd's lib dir on the toolchain default link path: LLVM 24's clang-linker-wrapper links libzstd when combining compressed offload images (--offload-compress defaults on). - Rebase rocm-device-libs' cmake.patch and guard rocm-comgr's EnvLLVMPath substitution against a null default. Revised during the split review: Set LIBOMP_COPY_EXPORTS to FALSE for openmp: ROCm's llvm-project defaults it to TRUE (upstream LLVM: FALSE), which copies libomp and its generated headers into the source tree. The generic LLVM >= 23 runtimes build of openmp (upstream PR 575569, carried at the bottom of this stack) then needs no writable copy of the source. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocm-device-libs copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.llvm: 7.2.4 -> 10.1.0, build LLVM 24 on the llvmPackages_23 base": ROCm 10.1's llvm-project fork is LLVM 24 (LLVMVersion.cmake: LLVM_VERSION_MAJOR 24), sourced from the therock-10.1 tag. nixpkgs has no released llvmPackages_24, and the 24.0.0-git snapshot (llvmPackages_git) is an alias reserved for testing whose package-set splice resolves through that alias, so it cannot be evaluated with allowAliases = false (as CI does). Keep the released llvmPackages_23 as the base: it only supplies the shared compilers/llvm/common packaging and the bootstrap clang/bintools, while rocmLlvmVersion = "24.0.0-rocm" selects the LLVM 24 patch set for ROCm's own source. LTO is off, so the base's bitcode version is moot. Toolchain changes for the LLVM 22 -> 24 jump: - Drop the LLVM 22 backports (select-ptr-combine, arm64e_x1 bridge, v64i8 VBMI shuffle, scan-build-py option): all upstream in LLVM 24. - Drop the CompressedOffloadBundle::decompress postPatch and the EmissaryIds postUnpack symlink: ROCm 10.1 uses the raw_ostream* decompress form and removed CGEmitEmissaryExec.cpp/EmissaryIds.h (llvm#175265). - Replace the GZGavinZhao GetBundleIDsInFile fetchpatch (no longer applies) with a locally rebased clang-add-getbundleidsinfile.diff; it adds the set-returning OffloadBundler::GetBundleIDsInFile the comgr ISA-compat patch calls. - Drop clang-log-jobs.diff: LLVM 24 rewrote Compilation::ExecuteJobs around parallel offload job groups, so the patched loop no longer exists; it was cosmetic CI-progress logging. - Put zstd's lib dir on the toolchain default link path: LLVM 24's clang-linker-wrapper links libzstd when combining compressed offload images (--offload-compress defaults on). - Rebase rocm-device-libs' cmake.patch and guard rocm-comgr's EnvLLVMPath substitution against a null default. Assisted-by: Claude Code (claude-opus-5-5)
Verbatim copies of the rocmPackages (ROCm 7.2.4) expressions for rocm-core, rocm-cmake, rocprofiler-register, rocm-runtime, rocm-comgr, rocminfo, amdsmi, rocm-smi, hip-common, roctracer, clr, hipify, half, not yet part of the rocmPackages_10 scope. The following commits update each one to ROCm 10.1, so their diffs read against 7.2.4. Assisted-by: Claude Code (claude-opus-5-5)
hipcc, rocm-docs-core are unchanged between ROCm 7.2.4 and 10.1 (no 10.x upstream release, or versioned independently of ROCm), so the rocmPackages_10 scope calls the existing expression instead of copying it. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocm-core copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Revised during the split review: Report the padded version (10.1.0) in the setup hook as well, the three-part form 7.2.4 exported; ROCM_VERSION is only read from the environment as a CPack fallback in comgr and hipcc. Keep the version helper in the outer let. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocm-cmake copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocprofiler-register copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From the mass bump: - rocprofiler-register: drop 3 fetchpatches now upstream. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocm-runtime copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.rocm-runtime: drop patches upstreamed in ROCr 1.21": ROCm 10 ships ROCr 1.21, which carries the queue-failure fix (PR NixOS#2850), the kfd_ioctl 1<<31 UB fix, and refactors aqlprofile to a runtime dlopen (removing the build-time circular dependency). Drop the three now-obsolete patches and add python3 for the new trap_handler find_package(Python3). Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocm-comgr copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.rocm-comgr: rebase ISA-compat CCOB patch for LLVM 24": The "[Comgr] Extend ISA compatibility for CCOB" change (GZGavinZhao 2c1e44fc) no longer applies as a fetchpatch: LLVM 24's comgr inserted comgr-unpackage-command.h into comgr-compiler.cpp's include block (shifting the comgr-metadata.h include past patch's default fuzz) and moved the unbundle() body ~477 lines down. Rebase it locally as comgr-isa-compat-ccob.patch against therock-10.1. The OffloadBundler::GetBundleIDsInFile it calls is provided clang-side by clang-add-getbundleidsinfile.diff in the llvm module. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocminfo copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.amdsmi copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.amdsmi: adapt to 27.0.0 ABI": amdsmi 27.0.0 (SONAME libamd_smi.so.27) requires libnl-3/libnl-genl-3 and libmnl; add them. py-interface/CMakeLists.txt now FATAL_ERRORs at configure time if it cannot locate a system python3 (it probes /usr/bin/python3 and /usr/libexec/platform-python, neither in the sandbox) unless AMDSMI_SYSTEM_PYTHON_SITELIB is set. The install it guards is DESTDIR-gated (nix uses CMAKE_INSTALL_PREFIX) and thus skipped anyway, so pass nix's sitePackages to satisfy the guard. Revised during the split review: Drop the drm_color_ctm_3x4 patch: amdsmi 10.1 builds without it against libdrm 2.4.134, whose drm_mode.h shares the bundled copy's include guard. Pin esmi_ib_library by the commit amdsmi pins. Link libamd_smi.so with -Bsymbolic-functions: with librocm_smi64 loaded (as after import torch) its exported rsmi_init was interposed and amdsmi enumerated no GPUs, so torch.cuda.device_count() was 0. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocm-smi copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.hip-common copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.roctracer copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.clr copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.hipify copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.half copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Verbatim copies of the rocmPackages (ROCm 7.2.4) expressions for aqlprofile, rocprof-trace-decoder, rocdbgapi, rocgdb, rocr-debug-agent, rocprofiler, rdc, rccl, rocprofiler-sdk, rocshmem, not yet part of the rocmPackages_10 scope. The following commits update each one to ROCm 10.1, so their diffs read against 7.2.4. Assisted-by: Claude Code (claude-opus-5-5)
rocprof-compute-viewer, mscclpp are unchanged between ROCm 7.2.4 and 10.1 (no 10.x upstream release, or versioned independently of ROCm), so the rocmPackages_10 scope calls the existing expression instead of copying it. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.aqlprofile copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocprof-trace-decoder copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.rocprof-trace-decoder: 0.1.7 -> 0.2.2": Track the rocm-systems monorepo at the therock-10.1 tag (the project keeps its own 0.x version, so the tag cannot derive from finalAttrs.version). Drop both local patches: use-system-dependencies.patch is obsolete (upstream removed the json FetchContent and prefers system gtest) and fix-test-dependency.patch (PR NixOS#3800) is merged upstream. Disable the ATT integration tests, which need pyelftools and bundled fixtures. From "rocmPackages.rocprof-trace-decoder: take comgr and runtime from the scope": It took the top-level rocmPackages and used rocmPackages.rocm-comgr and rocmPackages.rocm-runtime, so the per-arch scopes (rocmPackages.gfx906, ...) and any other instance of the scope built it against the default set's libraries. Take them as scope arguments like every other package. The default rocmPackages.rocprof-trace-decoder derivation is unchanged. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocdbgapi copy from the previous commit to ROCm
10.1 and add it to the rocmPackages_10 scope.
From "rocmPackages.{rocdbgapi,rocr-debug-agent}: move to rocm-systems
monorepo":
ROCdbgapi and rocr_debug_agent moved into the rocm-systems monorepo in
ROCm 10. Switch both to the sparse-checkout source (projects/<name> +
shared) at the therock-10.1 tag.
Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocgdb copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.rocgdb: 7.2.4 -> 10.1.0". Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocr-debug-agent copy from the previous commit
to ROCm 10.1 and add it to the rocmPackages_10 scope.
From "rocmPackages.{rocdbgapi,rocr-debug-agent}: move to rocm-systems
monorepo":
ROCdbgapi and rocr_debug_agent moved into the rocm-systems monorepo in
ROCm 10. Switch both to the sparse-checkout source (projects/<name> +
shared) at the therock-10.1 tag.
Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocprofiler copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From the mass bump: - rocprofiler-register: drop 3 fetchpatches now upstream. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rdc copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.rdc: drop CXXFLAGS patch fixed upstream in therock-10.1": ROCm/rocm-systems#2423 was closed without merging, but therock-10.1 fixes the same problem: projects/rdc/CMakeLists.txt now appends to CMAKE_CXX_FLAGS instead of overwriting it. fix-cmake-cxxflags.patch no longer applies. The libcap pkg-config patch (NixOS#2424) is still needed and applies with offsets. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rccl copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.rccl: monorepo source, rocm_smi -> amd_smi, drop mscclpp": rccl moved into the rocm-systems monorepo in ROCm 10. Switch to the sparse-checkout source and adapt: - rocm_smi -> amd_smi (CMakeLists find_package(amd_smi) gated on ROCM_VERSION >= 71100; SMI is dlopen'd at runtime). - Drop fix_hw_reg_hw_id_gt_gfx10.patch: its __trace_hwreg() target was removed when collTrace was restructured; the other hunk only touches tools/JitterBench (not built by default). - Drop the MSCCLPP integration (ENABLE_MSCCLPP / MSCCLPP_ROOT): upstream removed it from CMakeLists, so the separate mscclpp input is gone. Revised during the split review: Drop -DENABLE_MSCCL_KERNEL; 10.x no longer has the option. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocprofiler-sdk copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.rocprofiler-sdk: drop patches upstreamed in therock-10.1": ROCm 10.1 merged the whole nixpkgs patch stack, so remove it: - 86ee7b2e / 36d9d33d / 94a4595 fetchpatches (system-libraries build, fmt memory header, amd_comgr linkage): present upstream. - 0001 (rename-consumer-test) and 0008 (stop-setting-warning-flags): only touched test CMakeLists; tests are off by default. - 0002 (missing STL headers), 0003 (amd-comgr dep), 0004 (find_package GLOBAL scoping): upstream. - 0005/0006/0007 (system elfio/otf2/json): the ROCPROFILER_BUILD_ELFIO/ OTF2/JSON options and their system-dependency branches are upstream (PR NixOS#2319 / NixOS#1243). - 0010 (fmt 12.2 format.h): upstream. The -DROCPROFILER_BUILD_{ELFIO,OTF2,JSON}=false cmakeFlags still drive those now-upstream options to use the system elfio/otf2/nlohmann_json. rocprofiler-sdk-tool now find_package(rocprof-trace-decoder REQUIRED)s at configure time, so add it as a build input. Revised during the split review: Drop the 0009 glog guard, the glog input and ROCPROFILER_BUILD_GLOG: the 10.1 SDK logs through Abseil (rocm-systems d80117889b3f is in therock-10.1), and the guard's glog symbol only resolved through rocprofiler-register. Add ompt-explicit-tool.patch: ompt_start_tool now declines unless a tool was requested through ROCP_TOOL_LIBRARIES (as rocprofv3 does) or rocprofiler_force_configure. Registering clients from libomp's initialisation dlopens every library defining rocprofiler_configure; with torch, libtorch_cpu (Kineto) is still loading, glibc reruns its pending constructors, and OpenBLAS reenters OpenMP initialisation, livelocking import. Clients found by symbol still register when the HIP runtime starts and only miss OMPT callbacks. This replaces the OMP_TOOL edit torch carried in PR 573885. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocshmem copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.rocshmem: move to rocm-systems monorepo, fix device bitcode": rocSHMEM moved into the rocm-systems monorepo in ROCm 10. Switch to the sparse-checkout source and fix its device-bitcode build: cmake/DeviceBitcode.cmake compiles device-side .bc with a manual clang command whose -I list omits the HSA headers (the host librocshmem.a gets them via the hsa-runtime64 link interface), so the bitcode build failed with "'hsa/hsa.h' file not found". Add rocm-runtime's include dir to the bitcode flags. Assisted-by: Claude Code (claude-opus-5-5)
This was referenced Oct 27, 2026
nixpkgs-ci
Bot
added
10.rebuild-linux: 1-10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 4/14. Part of splitting #573885 (
rocmPackages: 7.2.4 -> 10.1) into smaller PRs, as asked in its review.rocmPackages_10:aqlprofile,rocprof-trace-decoder,rocdbgapi,rocgdb,rocr-debug-agent,rocprofiler,rdc,rccl,rocprofiler-sdk,rocshmem; shared unchanged:rocprof-compute-viewer,mscclppmaster. No existing derivation changes:rocmPackages_10is not recursed into00b0bf3ccf63) are new hereHow to review
rocm-modules/10/Notes
rocprofiler-sdkdepends on it; it dropsENABLE_MSCCL_KERNEL(removed in 10.x)ompt-explicit-tool.patch, which fixes animport torchhang:ompt_start_toolnow declines unless a tool was asked for (ROCP_TOOL_LIBRARIES, whichrocprofv3sets, orrocprofiler_force_configure)libtorch_cpu.sowhile it is still loading → OpenBLAS re-enters OpenMP init and spins on its lockOMP_TOOLworkaround in rocmPackages: 7.2.3 -> 10.1.0 #573885ompt_start_toolintorocprofv3); this patch can go once it landsTesting
rocmPackages_10evaluates: 36 derivations, 0 failuresSeries
Preparatory PRs, each against
master: #579509, #579510, #579511, #579512, #579513.master🤖 Generated with Claude Code