Skip to content

rocmPackages_10: add profiling and debugging tools, RCCL - #21

Closed
randomizedcoder wants to merge 12 commits into
rocm10/03-runtimefrom
rocm10/04-tools-rccl
Closed

rocmPackages_10: add profiling and debugging tools, RCCL#21
randomizedcoder wants to merge 12 commits into
rocm10/03-runtimefrom
rocm10/04-tools-rccl

Conversation

@randomizedcoder

@randomizedcoder randomizedcoder commented Oct 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Part 4/14. Part of splitting upstream PR 573885 (rocmPackages: 7.2.4 -> 10.1) into smaller PRs, as asked in its review.

  • Adds to rocmPackages_10: aqlprofile, rocprof-trace-decoder, rocdbgapi, rocgdb, rocr-debug-agent, rocprofiler, rdc, rccl, rocprofiler-sdk, rocshmem; shared unchanged: rocprof-compute-viewer, mscclpp
  • Target: master. No existing derivation changes: rocmPackages_10 is not recursed into

How to review

  • First commit: copies the 7.2.4 expressions unchanged into rocm-modules/10/
  • Then one commit per package: the 10.1 changes, with the reasons in the commit message

Notes

  • RCCL is here because rocprofiler-sdk depends on it; it drops ENABLE_MSCCL_KERNEL (removed in 10.x)
  • rocprofiler-sdk drops the 0009 glog patch (10.1 logs through Abseil)
  • rocprofiler-sdk adds ompt-explicit-tool.patch, which fixes an import torch hang:
    • ompt_start_tool now declines unless a tool was asked for (ROCP_TOOL_LIBRARIES, which rocprofv3 sets, or rocprofiler_force_configure)
    • without it: libomp init → the SDK dlopens libtorch_cpu.so while it is still loading → OpenBLAS re-enters OpenMP init and spins on its lock
    • tools found by symbol still register when HIP starts; they only miss OMPT callbacks
    • replaces the torch OMP_TOOL workaround in upstream PR 573885
    • AMD's own fix for the same deadlock, rocm-systems PR 7789, is still open (it moves ompt_start_tool into rocprofv3); this patch can go once it lands

Testing

  • rocmPackages_10 evaluates: 36 derivations, 0 failures
  • Built on x86_64-linux with the default GPU targets
  • nixpkgs-review: no rebuilds in the default package set
  • Hardware smoke tests: see rocmPackages: 7.2.4 -> 10.1 #31

Series

Preparatory PRs, each against master: #10, #11, #15, #16, #17.

Part PR Contents Base
1 #18 rocmPackages_10: init master
2 #19 llvm (LLVM 24) #18
3 #20 runtime #19
▶ 4 #21 profiling and debugging tools, RCCL (this PR) #20
5 #22 rocPRIM, rocRAND, rocFFT #21
6 #23 hipBLASLt #22
7 #24 Tensile and rocBLAS #23
8 #25 sparse, solvers, rocALUTION, rocWMMA, hipfort #24
9 #26 composable_kernel #25
10 #27 rocMLIR #26
11 #28 MIOpen #27
12 #29 MIGraphX, RPP, MIVisionX #28
13 #30 remaining packages #29
14 #31 rocmPackages: 7.2.4 -> 10.1 (the switch) #30

🤖 Generated with Claude Code

@randomizedcoder
randomizedcoder force-pushed the rocm10/03-runtime branch 2 times, most recently from 02bd7c2 to 9380f83 Compare October 17, 2026 02:50
@randomizedcoder
randomizedcoder force-pushed the rocm10/04-tools-rccl branch 3 times, most recently from 2617d3f to 9a5b6b6 Compare October 21, 2026 07:15
@randomizedcoder
rocmPackages_10: copy profiling and debugging tools, RCCL
d3030b3 
Verbatim copies of the rocmPackages (ROCm 7.2.4) expressions for
aqlprofile, rocprof-trace-decoder, rocdbgapi, rocgdb, rocr-debug-agent,
rocprofiler, rdc, rccl, rocprofiler-sdk, rocshmem, not yet part of the
rocmPackages_10 scope. The following commits update each one to ROCm
10.1, so their diffs read against 7.2.4.

Assisted-by: Claude Code (claude-opus-5-5)
@randomizedcoder
rocmPackages_10: share rocprof-compute-viewer, mscclpp with rocmPackages
f7a6fe3 
rocprof-compute-viewer, mscclpp are unchanged between ROCm 7.2.4 and
10.1 (no 10.x upstream release, or versioned independently of ROCm), so
the rocmPackages_10 scope calls the existing expression instead of
copying it.

Assisted-by: Claude Code (claude-opus-5-5)
@randomizedcoder
rocmPackages_10.aqlprofile: init at 10.1
714d302 
Update the rocmPackages.aqlprofile copy from the previous commit to ROCm
10.1 and add it to the rocmPackages_10 scope. Source/version bump only.

Assisted-by: Claude Code (claude-opus-5-5)
@randomizedcoder
rocmPackages_10.rocprof-trace-decoder: init at 0.2.2
ccf1430 
Update the rocmPackages.rocprof-trace-decoder copy from the previous
commit to ROCm 10.1 and add it to the rocmPackages_10 scope.

From "rocmPackages.rocprof-trace-decoder: 0.1.7 -> 0.2.2":

Track the rocm-systems monorepo at the therock-10.1 tag (the project
keeps its own 0.x version, so the tag cannot derive from
finalAttrs.version). Drop both local patches:
use-system-dependencies.patch is obsolete (upstream removed the json
FetchContent and prefers system gtest) and fix-test-dependency.patch (PR
NixOS#3800) is merged upstream. Disable the ATT integration tests, which need
pyelftools and bundled fixtures.

From "rocmPackages.rocprof-trace-decoder: take comgr and runtime from
the scope":

It took the top-level rocmPackages and used rocmPackages.rocm-comgr and
rocmPackages.rocm-runtime, so the per-arch scopes (rocmPackages.gfx906,
...) and any other instance of the scope built it against the default
set's libraries. Take them as scope arguments like every other package.
The default rocmPackages.rocprof-trace-decoder derivation is unchanged.

Assisted-by: Claude Code (claude-opus-5-5)
@randomizedcoder
rocmPackages_10.rocdbgapi: init at 10.1
d692999 
Update the rocmPackages.rocdbgapi copy from the previous commit to ROCm
10.1 and add it to the rocmPackages_10 scope.

From "rocmPackages.{rocdbgapi,rocr-debug-agent}: move to rocm-systems
monorepo":

ROCdbgapi and rocr_debug_agent moved into the rocm-systems monorepo in
ROCm 10. Switch both to the sparse-checkout source (projects/<name> +
shared) at the therock-10.1 tag.

Assisted-by: Claude Code (claude-opus-5-5)
@randomizedcoder
rocmPackages_10.rocgdb: init at 10.1
90c358a 
Update the rocmPackages.rocgdb copy from the previous commit to ROCm
10.1 and add it to the rocmPackages_10 scope.

From "rocmPackages.rocgdb: 7.2.4 -> 10.1.0".

Assisted-by: Claude Code (claude-opus-5-5)
@randomizedcoder
rocmPackages_10.rocr-debug-agent: init at 10.1
80a94c1 
Update the rocmPackages.rocr-debug-agent copy from the previous commit
to ROCm 10.1 and add it to the rocmPackages_10 scope.

From "rocmPackages.{rocdbgapi,rocr-debug-agent}: move to rocm-systems
monorepo":

ROCdbgapi and rocr_debug_agent moved into the rocm-systems monorepo in
ROCm 10. Switch both to the sparse-checkout source (projects/<name> +
shared) at the therock-10.1 tag.

Assisted-by: Claude Code (claude-opus-5-5)
@randomizedcoder
rocmPackages_10.rocprofiler: init at 10.1
6b5e088 
Update the rocmPackages.rocprofiler copy from the previous commit to
ROCm 10.1 and add it to the rocmPackages_10 scope.

From the mass bump:
- rocprofiler-register: drop 3 fetchpatches now upstream.

Assisted-by: Claude Code (claude-opus-5-5)
@randomizedcoder
rocmPackages_10.rdc: init at 10.1
29472b7 
Update the rocmPackages.rdc copy from the previous commit to ROCm 10.1
and add it to the rocmPackages_10 scope.

From "rocmPackages.rdc: drop CXXFLAGS patch fixed upstream in
therock-10.1":

ROCm/rocm-systems#2423 was closed without merging, but therock-10.1
fixes the same problem: projects/rdc/CMakeLists.txt now appends to
CMAKE_CXX_FLAGS instead of overwriting it. fix-cmake-cxxflags.patch no
longer applies. The libcap pkg-config patch (NixOS#2424) is still needed and
applies with offsets.

Assisted-by: Claude Code (claude-opus-5-5)
@randomizedcoder
rocmPackages_10.rccl: init at 10.1
6ea18ed 
Update the rocmPackages.rccl copy from the previous commit to ROCm 10.1
and add it to the rocmPackages_10 scope.

From "rocmPackages.rccl: monorepo source, rocm_smi -> amd_smi, drop
mscclpp":

rccl moved into the rocm-systems monorepo in ROCm 10. Switch to the
sparse-checkout source and adapt:
- rocm_smi -> amd_smi (CMakeLists find_package(amd_smi) gated on
  ROCM_VERSION >= 71100; SMI is dlopen'd at runtime).
- Drop fix_hw_reg_hw_id_gt_gfx10.patch: its __trace_hwreg() target was
  removed when collTrace was restructured; the other hunk only touches
  tools/JitterBench (not built by default).
- Drop the MSCCLPP integration (ENABLE_MSCCLPP / MSCCLPP_ROOT): upstream
  removed it from CMakeLists, so the separate mscclpp input is gone.

Revised during the split review:

Drop -DENABLE_MSCCL_KERNEL; 10.x no longer has the option.

Assisted-by: Claude Code (claude-opus-5-5)
@randomizedcoder
rocmPackages_10.rocprofiler-sdk: init at 10.1
e89558d 
Update the rocmPackages.rocprofiler-sdk copy from the previous commit to
ROCm 10.1 and add it to the rocmPackages_10 scope.

From "rocmPackages.rocprofiler-sdk: drop patches upstreamed in
therock-10.1":

ROCm 10.1 merged the whole nixpkgs patch stack, so remove it:
- 86ee7b2e / 36d9d33d / 94a4595 fetchpatches (system-libraries build,
  fmt memory header, amd_comgr linkage): present upstream.
- 0001 (rename-consumer-test) and 0008 (stop-setting-warning-flags):
  only touched test CMakeLists; tests are off by default.
- 0002 (missing STL headers), 0003 (amd-comgr dep), 0004 (find_package
  GLOBAL scoping): upstream.
- 0005/0006/0007 (system elfio/otf2/json): the ROCPROFILER_BUILD_ELFIO/
  OTF2/JSON options and their system-dependency branches are upstream
  (PR NixOS#2319 / NixOS#1243).
- 0010 (fmt 12.2 format.h): upstream.

The -DROCPROFILER_BUILD_{ELFIO,OTF2,JSON}=false cmakeFlags still drive
those now-upstream options to use the system elfio/otf2/nlohmann_json.
rocprofiler-sdk-tool now find_package(rocprof-trace-decoder REQUIRED)s
at configure time, so add it as a build input.

Revised during the split review:

Drop the 0009 glog guard, the glog input and ROCPROFILER_BUILD_GLOG: the
10.1 SDK logs through Abseil (rocm-systems d80117889b3f is in
therock-10.1), and the guard's glog symbol only resolved through
rocprofiler-register.

Add ompt-explicit-tool.patch: ompt_start_tool now declines unless a tool
was requested through ROCP_TOOL_LIBRARIES (as rocprofv3 does) or
rocprofiler_force_configure. Registering clients from libomp's
initialisation dlopens every library defining rocprofiler_configure;
with torch, libtorch_cpu (Kineto) is still loading, glibc reruns its
pending constructors, and OpenBLAS reenters OpenMP initialisation,
livelocking import. Clients found by symbol still register when the HIP
runtime starts and only miss OMPT callbacks. This replaces the OMP_TOOL
edit torch carried in PR 573885.

Assisted-by: Claude Code (claude-opus-5-5)
@randomizedcoder
rocmPackages_10.rocshmem: init at 10.1
e01bc52 
Update the rocmPackages.rocshmem copy from the previous commit to ROCm
10.1 and add it to the rocmPackages_10 scope.

From "rocmPackages.rocshmem: move to rocm-systems monorepo, fix device
bitcode":

rocSHMEM moved into the rocm-systems monorepo in ROCm 10. Switch to the
sparse-checkout source and fix its device-bitcode build:
cmake/DeviceBitcode.cmake compiles device-side .bc with a manual clang
command whose -I list omits the HSA headers (the host librocshmem.a gets
them via the hsa-runtime64 link interface), so the bitcode build failed
with "'hsa/hsa.h' file not found". Add rocm-runtime's include dir to the
bitcode flags.

Assisted-by: Claude Code (claude-opus-5-5)
This was referenced Oct 27, 2026
Closed
Closed
Closed
Closed
Closed
Closed
This was referenced Oct 27, 2026
Closed
Closed
Closed
Closed
Closed
Closed
Closed
@randomizedcoder

randomizedcoder commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

Opened upstream as NixOS#579517.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@randomizedcoder