Repository navigation
rocmPackages_10: add Tensile and rocBLAS - #579520
Draft
randomizedcoder wants to merge 47 commits into
Draft
randomizedcoder wants to merge 47 commits into
randomizedcoder wants to merge 47 commits into
Conversation
Prepare the update script for a rocmPackages_10 set built from the TheRock monorepos, whose sources are tagged therock-X.Y[.Z]: - maxMajor (default 7) replaces the hard-coded major cap, so a second set can call it with its own cap. - tagPrefixes (default [ "rocm" ]) selects which tag families count as releases. The current set keeps rocm- only, so the therock-7.x preview tags are not picked up. - setName (default "rocmPackages") is the attribute path passed to update-source-version. - Pick the highest matching version instead of the first release listed: releases are not ordered by version (a rocm-7.2.x maintenance release can be published after a newer series). - Follow GitHub's pagination (100 per page) so the highest version is found even when it is not on the first page, and fail on fetch errors instead of reporting "no valid version". No change to any derivation; passthru.updateScript only. Assisted-by: Claude Code (claude-opus-5-5)
(cherry picked from commit ab348e2)
Add an empty rocmPackages_10 scope next to rocmPackages, sharing its structure (rocm-modules/10/default.nix is derived from rocm-modules/default.nix). The following PRs add the ROCm 10.1 packages to it tier by tier; a final PR on staging switches rocmPackages over. Following the rocmPackages_5/_6 layout. It is deliberately not wrapped in recurseIntoAttrs, so Hydra and the CI rebuild count are unaffected until the switch. The update script runs with maxMajor = 10, the therock- tag prefix, and setName = "rocmPackages_10". Assisted-by: Claude Code (claude-opus-5-5)
Verbatim copies of the rocmPackages (ROCm 7.2.4) expressions for llvm, rocm-device-libs, not yet part of the rocmPackages_10 scope. The following commits update each one to ROCm 10.1, so their diffs read against 7.2.4. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.llvm copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.llvm: 7.2.4 -> 10.1.0, build LLVM 24 on the llvmPackages_23 base": ROCm 10.1's llvm-project fork is LLVM 24 (LLVMVersion.cmake: LLVM_VERSION_MAJOR 24), sourced from the therock-10.1 tag. nixpkgs has no released llvmPackages_24, and the 24.0.0-git snapshot (llvmPackages_git) is an alias reserved for testing whose package-set splice resolves through that alias, so it cannot be evaluated with allowAliases = false (as CI does). Keep the released llvmPackages_23 as the base: it only supplies the shared compilers/llvm/common packaging and the bootstrap clang/bintools, while rocmLlvmVersion = "24.0.0-rocm" selects the LLVM 24 patch set for ROCm's own source. LTO is off, so the base's bitcode version is moot. Toolchain changes for the LLVM 22 -> 24 jump: - Drop the LLVM 22 backports (select-ptr-combine, arm64e_x1 bridge, v64i8 VBMI shuffle, scan-build-py option): all upstream in LLVM 24. - Drop the CompressedOffloadBundle::decompress postPatch and the EmissaryIds postUnpack symlink: ROCm 10.1 uses the raw_ostream* decompress form and removed CGEmitEmissaryExec.cpp/EmissaryIds.h (llvm#175265). - Replace the GZGavinZhao GetBundleIDsInFile fetchpatch (no longer applies) with a locally rebased clang-add-getbundleidsinfile.diff; it adds the set-returning OffloadBundler::GetBundleIDsInFile the comgr ISA-compat patch calls. - Drop clang-log-jobs.diff: LLVM 24 rewrote Compilation::ExecuteJobs around parallel offload job groups, so the patched loop no longer exists; it was cosmetic CI-progress logging. - Put zstd's lib dir on the toolchain default link path: LLVM 24's clang-linker-wrapper links libzstd when combining compressed offload images (--offload-compress defaults on). - Rebase rocm-device-libs' cmake.patch and guard rocm-comgr's EnvLLVMPath substitution against a null default. Revised during the split review: Set LIBOMP_COPY_EXPORTS to FALSE for openmp: ROCm's llvm-project defaults it to TRUE (upstream LLVM: FALSE), which copies libomp and its generated headers into the source tree. The generic LLVM >= 23 runtimes build of openmp (upstream PR 575569, carried at the bottom of this stack) then needs no writable copy of the source. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocm-device-libs copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.llvm: 7.2.4 -> 10.1.0, build LLVM 24 on the llvmPackages_23 base": ROCm 10.1's llvm-project fork is LLVM 24 (LLVMVersion.cmake: LLVM_VERSION_MAJOR 24), sourced from the therock-10.1 tag. nixpkgs has no released llvmPackages_24, and the 24.0.0-git snapshot (llvmPackages_git) is an alias reserved for testing whose package-set splice resolves through that alias, so it cannot be evaluated with allowAliases = false (as CI does). Keep the released llvmPackages_23 as the base: it only supplies the shared compilers/llvm/common packaging and the bootstrap clang/bintools, while rocmLlvmVersion = "24.0.0-rocm" selects the LLVM 24 patch set for ROCm's own source. LTO is off, so the base's bitcode version is moot. Toolchain changes for the LLVM 22 -> 24 jump: - Drop the LLVM 22 backports (select-ptr-combine, arm64e_x1 bridge, v64i8 VBMI shuffle, scan-build-py option): all upstream in LLVM 24. - Drop the CompressedOffloadBundle::decompress postPatch and the EmissaryIds postUnpack symlink: ROCm 10.1 uses the raw_ostream* decompress form and removed CGEmitEmissaryExec.cpp/EmissaryIds.h (llvm#175265). - Replace the GZGavinZhao GetBundleIDsInFile fetchpatch (no longer applies) with a locally rebased clang-add-getbundleidsinfile.diff; it adds the set-returning OffloadBundler::GetBundleIDsInFile the comgr ISA-compat patch calls. - Drop clang-log-jobs.diff: LLVM 24 rewrote Compilation::ExecuteJobs around parallel offload job groups, so the patched loop no longer exists; it was cosmetic CI-progress logging. - Put zstd's lib dir on the toolchain default link path: LLVM 24's clang-linker-wrapper links libzstd when combining compressed offload images (--offload-compress defaults on). - Rebase rocm-device-libs' cmake.patch and guard rocm-comgr's EnvLLVMPath substitution against a null default. Assisted-by: Claude Code (claude-opus-5-5)
Verbatim copies of the rocmPackages (ROCm 7.2.4) expressions for rocm-core, rocm-cmake, rocprofiler-register, rocm-runtime, rocm-comgr, rocminfo, amdsmi, rocm-smi, hip-common, roctracer, clr, hipify, half, not yet part of the rocmPackages_10 scope. The following commits update each one to ROCm 10.1, so their diffs read against 7.2.4. Assisted-by: Claude Code (claude-opus-5-5)
hipcc, rocm-docs-core are unchanged between ROCm 7.2.4 and 10.1 (no 10.x upstream release, or versioned independently of ROCm), so the rocmPackages_10 scope calls the existing expression instead of copying it. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocm-core copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Revised during the split review: Report the padded version (10.1.0) in the setup hook as well, the three-part form 7.2.4 exported; ROCM_VERSION is only read from the environment as a CPack fallback in comgr and hipcc. Keep the version helper in the outer let. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocm-cmake copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocprofiler-register copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From the mass bump: - rocprofiler-register: drop 3 fetchpatches now upstream. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocm-runtime copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.rocm-runtime: drop patches upstreamed in ROCr 1.21": ROCm 10 ships ROCr 1.21, which carries the queue-failure fix (PR NixOS#2850), the kfd_ioctl 1<<31 UB fix, and refactors aqlprofile to a runtime dlopen (removing the build-time circular dependency). Drop the three now-obsolete patches and add python3 for the new trap_handler find_package(Python3). Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocm-comgr copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.rocm-comgr: rebase ISA-compat CCOB patch for LLVM 24": The "[Comgr] Extend ISA compatibility for CCOB" change (GZGavinZhao 2c1e44fc) no longer applies as a fetchpatch: LLVM 24's comgr inserted comgr-unpackage-command.h into comgr-compiler.cpp's include block (shifting the comgr-metadata.h include past patch's default fuzz) and moved the unbundle() body ~477 lines down. Rebase it locally as comgr-isa-compat-ccob.patch against therock-10.1. The OffloadBundler::GetBundleIDsInFile it calls is provided clang-side by clang-add-getbundleidsinfile.diff in the llvm module. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocminfo copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.amdsmi copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.amdsmi: adapt to 27.0.0 ABI": amdsmi 27.0.0 (SONAME libamd_smi.so.27) requires libnl-3/libnl-genl-3 and libmnl; add them. py-interface/CMakeLists.txt now FATAL_ERRORs at configure time if it cannot locate a system python3 (it probes /usr/bin/python3 and /usr/libexec/platform-python, neither in the sandbox) unless AMDSMI_SYSTEM_PYTHON_SITELIB is set. The install it guards is DESTDIR-gated (nix uses CMAKE_INSTALL_PREFIX) and thus skipped anyway, so pass nix's sitePackages to satisfy the guard. Revised during the split review: Drop the drm_color_ctm_3x4 patch: amdsmi 10.1 builds without it against libdrm 2.4.134, whose drm_mode.h shares the bundled copy's include guard. Pin esmi_ib_library by the commit amdsmi pins. Link libamd_smi.so with -Bsymbolic-functions: with librocm_smi64 loaded (as after import torch) its exported rsmi_init was interposed and amdsmi enumerated no GPUs, so torch.cuda.device_count() was 0. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocm-smi copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.hip-common copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.roctracer copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.clr copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.hipify copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.half copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Verbatim copies of the rocmPackages (ROCm 7.2.4) expressions for aqlprofile, rocprof-trace-decoder, rocdbgapi, rocgdb, rocr-debug-agent, rocprofiler, rdc, rccl, rocprofiler-sdk, rocshmem, not yet part of the rocmPackages_10 scope. The following commits update each one to ROCm 10.1, so their diffs read against 7.2.4. Assisted-by: Claude Code (claude-opus-5-5)
rocprof-compute-viewer, mscclpp are unchanged between ROCm 7.2.4 and 10.1 (no 10.x upstream release, or versioned independently of ROCm), so the rocmPackages_10 scope calls the existing expression instead of copying it. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.aqlprofile copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocprof-trace-decoder copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.rocprof-trace-decoder: 0.1.7 -> 0.2.2": Track the rocm-systems monorepo at the therock-10.1 tag (the project keeps its own 0.x version, so the tag cannot derive from finalAttrs.version). Drop both local patches: use-system-dependencies.patch is obsolete (upstream removed the json FetchContent and prefers system gtest) and fix-test-dependency.patch (PR NixOS#3800) is merged upstream. Disable the ATT integration tests, which need pyelftools and bundled fixtures. From "rocmPackages.rocprof-trace-decoder: take comgr and runtime from the scope": It took the top-level rocmPackages and used rocmPackages.rocm-comgr and rocmPackages.rocm-runtime, so the per-arch scopes (rocmPackages.gfx906, ...) and any other instance of the scope built it against the default set's libraries. Take them as scope arguments like every other package. The default rocmPackages.rocprof-trace-decoder derivation is unchanged. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocdbgapi copy from the previous commit to ROCm
10.1 and add it to the rocmPackages_10 scope.
From "rocmPackages.{rocdbgapi,rocr-debug-agent}: move to rocm-systems
monorepo":
ROCdbgapi and rocr_debug_agent moved into the rocm-systems monorepo in
ROCm 10. Switch both to the sparse-checkout source (projects/<name> +
shared) at the therock-10.1 tag.
Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocgdb copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.rocgdb: 7.2.4 -> 10.1.0". Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocr-debug-agent copy from the previous commit
to ROCm 10.1 and add it to the rocmPackages_10 scope.
From "rocmPackages.{rocdbgapi,rocr-debug-agent}: move to rocm-systems
monorepo":
ROCdbgapi and rocr_debug_agent moved into the rocm-systems monorepo in
ROCm 10. Switch both to the sparse-checkout source (projects/<name> +
shared) at the therock-10.1 tag.
Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocprofiler copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From the mass bump: - rocprofiler-register: drop 3 fetchpatches now upstream. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rdc copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.rdc: drop CXXFLAGS patch fixed upstream in therock-10.1": ROCm/rocm-systems#2423 was closed without merging, but therock-10.1 fixes the same problem: projects/rdc/CMakeLists.txt now appends to CMAKE_CXX_FLAGS instead of overwriting it. fix-cmake-cxxflags.patch no longer applies. The libcap pkg-config patch (NixOS#2424) is still needed and applies with offsets. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocfft copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From the mass bump: - rocfft: rebase log-every-n-aot-jobs.patch onto the 10.1 source. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.hipfft copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Verbatim copies of the rocmPackages (ROCm 7.2.4) expressions for hipblas-common, hipblaslt, not yet part of the rocmPackages_10 scope. The following commits update each one to ROCm 10.1, so their diffs read against 7.2.4. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.hipblas-common copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.hipblaslt copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.hipblaslt: 1.4.1 (stinkytofu, amd_smi, nanobind)": hipBLASLt 1.4.1 needs shared/stinkytofu sources (rocisa compiles its .cpp unconditionally), switches rocm_smi -> amd_smi, renames the MSGPACK/LLVM enables to HIPBLASLT_ENABLE_YAML, and FetchContents nanobind when absent. Add the sources/inputs and rebase messagepack-compression-support.patch. Drop the Tensile-interning and TensileCreateLibrary-refactor tensilelite patches: they no longer apply to the reorganised tree; joblib is supplied via the python env instead. Rebase parallel-buildSourceCodeObjectFile (ROCm/rocm-libraries#1242 is still open) onto the new Toolchain/Component.py, adding -parallel-jobs=8 once to the HIP compiler's default args rather than appending it on every buildSourceCodeObjectFiles call. tensilelite's new HelperKernelCache defaults to $HOME/.tensile, so disable it. Set CMAKE_SUPPRESS_REGENERATION so cmake >= 4.4 does not re-run configure mid-build and lose the FetchContent nanobind_add_module. Revised during the split review: Add tensile-memory.patch, adapting the memory protections of the ROCm 7 Tensile patches to the new pipeline: worker count bounded by NIX_BUILD_CORES/MAKEFLAGS, interned YAML strings, compact post-assembly results. Refresh the MessagePack compression patch. Drop the stinkytofu clang-tidy workaround (upstream now guards it with ENABLE_CLANG_TIDY) and the redundant HOME override. Assisted-by: Claude Code (claude-opus-5-5)
Verbatim copies of the rocmPackages (ROCm 7.2.4) expressions for tensile, rocblas, not yet part of the rocmPackages_10 scope. The following commits update each one to ROCm 10.1, so their diffs read against 7.2.4. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.tensile copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. Source/version bump only. Assisted-by: Claude Code (claude-opus-5-5)
Update the rocmPackages.rocblas copy from the previous commit to ROCm 10.1 and add it to the rocmPackages_10 scope. From "rocmPackages.rocblas: split Tensile kernels into per-arch shards": Introduce a per-architecture Tensile "shard" split for rocBLAS: base.nix is the raw single-/multi-arch builder (the former default.nix), and default.nix now assembles one joined output from per-arch shards. For a multi-arch build the host librocblas.so + headers + cmake are built once for the head arch, and every other arch's self-contained lib/rocblas/library/<gfx>/ Tensile directory is built as a separate minimal shard and merged in postInstall. rocBLAS builds Tensile with SEPARATE_ARCHITECTURES + LAZY_LIBRARY_LOADING, so in a shared build the kernels are not linked into librocblas.so; each per-arch dir is loaded by path at runtime, making the join a plain per-arch-dir merge with no relink. Single-arch scopes (e.g. rocmPackages.gfx906.*) have no tail archs and return the base build verbatim, so their derivation stays byte-identical to before the split. The merge strips the shard store path that Tensile bakes into a per-arch code object's ELF metadata (remove-references-to). That reference is not load-bearing -- the unsplit multi-arch build self-references $out in the same spot and runs fine -- and disallowedReferences guards against a regression. Validated on a gfx1102 (Radeon Pro W7500): a [gfx906-head, gfx1102-shard] join runs a correct rocBLAS SGEMM on the gfx1102 hardware, loading the merged gfx1102 Tensile directory from the joined output. From "rocmPackages.rocblas: 7.2.4 -> 10.1.0": - Clients switched rocm_smi -> amd_smi (find_package(amd_smi CONFIG REQUIRED)); add amdsmi for the test/benchmark builds. - Drop the GZGavinZhao "Extend rocBLAS HIP ISA compatibility" patch: library/src/handle.cpp now has explicit Processor::gfx906/gfx90c cases, so with gfx906 in gpuTargets Tensile builds real gfx906 kernels. - Give cblas an explicit -L. rocBLAS compiles with the unwrapped rocm-toolchain clang, so buildInput lib dirs are not injected into the clang-linker-wrapper host-link path. With LLVM 24 the rocm-cmake rocm_check_target_ids probe (run before find_package(BLAS)) does a full --hip-link, and CMake seeds CMAKE_EXE_LINKER_FLAGS from $LDFLAGS, so a bare `-lcblas` makes that probe fail, emptying the supported-target list and aborting with "Unsupported target". Add amd-blis's libdir. - Drop the testing_logging.hpp /usr/bin/diff substitution; the source no longer shells out to it. - ROCm 10 installs a bin/rocblas/ CTest scaffolding dir alongside the client yaml/txt, so remove the whole bin/ tree from the library output. Revised during the split review: Forward package overrides to base.nix and read the effective settings from its passthru. Shard only builds whose Tensile libraries are separate-architecture and lazily loaded; use the base build otherwise. Omit -lzstd and the Tensile reference removal when Tensile is disabled. Assisted-by: Claude Code (claude-opus-5-5)
This was referenced Oct 27, 2026
nixpkgs-ci
Bot
added
10.rebuild-linux: 1-10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 7/14. Part of splitting #573885 (
rocmPackages: 7.2.4 -> 10.1) into smaller PRs, as asked in its review.rocmPackages_10:tensile,rocblasmaster. No existing derivation changes:rocmPackages_10is not recursed into6c84a785a3fa) are new hereHow to review
rocm-modules/10/Notes
rocblasrocblas_baseis marked broken (likecomposable_kernel_base), so Hydra never builds the all-arch base directlyrocblas.override { buildTests = false; }are passed on tobase.nixTesting
rocmPackages_10evaluates: 47 derivations, 0 failuresSeries
Preparatory PRs, each against
master: #579509, #579510, #579511, #579512, #579513.master🤖 Generated with Claude Code