rocmPackages.hipblaslt: massively reduce peak disk space usage by LunNova · Pull Request #451188 · NixOS/nixpkgs

LunNova · 2025-10-12T00:48:05Z

Fix zstd compression not being applied
Add regression test that .dat files are reasonably sized
Apply part of [hipblaslt] Refactor Parallel.py to drop joblib, decimate resource usage ROCm/rocm-libraries#2073 to reduce peak build dir size from 150GB to 25GB
- Only applying Parallel -> multiprocessing and simple memory fixes, full PR is WIP.
Remove workaround from rocmPackages.hipblaslt: stop inlining war and peace in asm comments #449985
- back to generating huge whitespace/comment sections in assembly files; no longer matters as they don't stick around for long

Things done

Add a 👍 reaction to pull requests you find important.

without this zstd compression of msgpack .dats silently failed

let's avoid regressing compression in future, oops

LunNova · 2025-10-12T02:18:10Z

Marking draft while investigating ROCm/rocm-libraries#2073 (comment)

Unsure if it's a pre-existing issue I just discovered or I broke something with this patch.

LunNova · 2025-10-13T00:05:25Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 451188
Commit: 4374426f0ee8eed7d41c68423d511d68d97bfb86

`x86_64-linux`

❌ 1 package failed to build:

rocmPackages.mivisionx (rocmPackages.mivisionx-hip) (will be fixed by rocmPackages.mivisionx{-hip,-cpu}: remove libjpeg_turbo input #448970)

✅ 24 packages built:

llama-cpp-rocm
magma (magma-hip)
magma.test (magma-hip.test)
ollama-rocm
python312Packages.torchWithRocm
python312Packages.torchWithRocm.cxxdev
python312Packages.torchWithRocm.dev
python312Packages.torchWithRocm.dist
python312Packages.torchWithRocm.lib
python313Packages.torchWithRocm
python313Packages.torchWithRocm.cxxdev
python313Packages.torchWithRocm.dev
python313Packages.torchWithRocm.dist
python313Packages.torchWithRocm.lib
rocmPackages.hipblas
rocmPackages.hipblaslt
rocmPackages.hipblaslt.benchmark
rocmPackages.hipsolver
rocmPackages.migraphx
rocmPackages.miopen (rocmPackages.miopen-hip)
rocmPackages.rocalution
rocmPackages.rocblas
rocmPackages.rocsolver
zluda

Flakebi

I didn’t look at the patch in detail, but it mostly seemed to make sense, so LGTM.
Thanks for reducing this even more!

Kitt3120

I'm not a maintainer, but this looks good to me, too. Well done!

LunNova · 2025-10-14T06:18:20Z

I've been working on much more significant rework in ROCm/rocm-libraries#2073 Some of it is not yet ready, but I've added the items that are low risk to this patch set for further resources reduction.

See the table in ROCm/rocm-libraries#2073 for commit by commit resource usage improvements.

We should probably land this soon since the previous attempt was extremely marginal for whether it can succeed on hydra.

pkgs/development/rocm-modules/6/hipblaslt/default.nix

…usage Peak build dir usage is now 25GB Partially applies [hipblaslt] Refactor Parallel.py to drop joblib, decimate resource usage

no longer needed now we unlink .s / .o files as soon as possible

06kellyjac · 2025-10-15T18:16:45Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 451188
Commit: bfd0d13dc984b8f6fcedccd2c26f1a2c7b5e3261

`x86_64-linux`

❌ 6 packages failed to build:

python312Packages.torchWithRocm
python312Packages.torchWithRocm.cxxdev
python312Packages.torchWithRocm.dev
python312Packages.torchWithRocm.dist
python312Packages.torchWithRocm.lib
rocmPackages.mivisionx (rocmPackages.mivisionx-hip)

✅ 19 packages built:

llama-cpp-rocm
magma (magma-hip)
magma.test (magma-hip.test)
ollama-rocm
python313Packages.torchWithRocm
python313Packages.torchWithRocm.cxxdev
python313Packages.torchWithRocm.dev
python313Packages.torchWithRocm.dist
python313Packages.torchWithRocm.lib
rocmPackages.hipblas
rocmPackages.hipblaslt
rocmPackages.hipblaslt.benchmark
rocmPackages.hipsolver
rocmPackages.migraphx
rocmPackages.miopen (rocmPackages.miopen-hip)
rocmPackages.rocalution
rocmPackages.rocblas
rocmPackages.rocsolver
zluda

Error logs: `x86_64-linux`

python312Packages.torchWithRocm

In file included from /build/pytorch/c10/core/TensorOptions.h:4:
In file included from /build/pytorch/c10/core/DefaultDtype.h:3:
In file included from /build/pytorch/c10/core/ScalarType.h:13:
In file included from /build/pytorch/c10/util/complex.h:9:
In file included from /nix/store/ivbqhyb8rrv1kxvh3shqsvphjl8y9y0h-rocthrust-6.4.3/include/thrust/complex.h:26:
In file included from /nix/store/ivbqhyb8rrv1kxvh3shqsvphjl8y9y0h-rocthrust-6.4.3/include/thrust/detail/type_traits.h:29:
In file included from /nix/store/zyzxcqvh5bnra2ya10xzc7bhwnl86s2f-rocprim-6.4.3/include/rocprim/type_traits.hpp:27:
In file included from /nix/store/zyzxcqvh5bnra2ya10xzc7bhwnl86s2f-rocprim-6.4.3/include/rocprim/type_traits_interface.hpp:24:
/nix/store/zyzxcqvh5bnra2ya10xzc7bhwnl86s2f-rocprim-6.4.3/include/rocprim/types.hpp:82:7: warning: macro '__AMDGCN_WAVEFRONT_SIZE' has been marked as deprecated: compile-time-constant access to the wavefront size will be removed in a future release [-Wdeprecated-pragma]
   82 | #elif ROCPRIM_WAVEFRONT_SIZE == 64
      |       ^
/nix/store/zyzxcqvh5bnra2ya10xzc7bhwnl86s2f-rocprim-6.4.3/include/rocprim/config.hpp:244:36: note: expanded from macro 'ROCPRIM_WAVEFRONT_SIZE'
  244 |     #define ROCPRIM_WAVEFRONT_SIZE __AMDGCN_WAVEFRONT_SIZE
      |                                    ^
<built-in>:429:139: note: macro marked 'deprecated' here
  429 | #pragma clang deprecated(__AMDGCN_WAVEFRONT_SIZE, "compile-time-constant access to the wavefront size will be removed in a future release")
      |                                                                                                                                           ^
5 warnings generated when compiling for host.
Job completed: /build/pytorch/build/caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/./torch_hip_generated_ReduceSumProdKernel.hip.o
[7124/7530] Building HIPCC object caffe2/CMakeFiles/torch_hip.dir/__/aten/src/ATen/native/hip/bgemm_kernels/torch_hip_generated_bgemm_kernel_bf16bf16bf16_256_16x256x64_16x16_1x4_8x16x1_8x16x1_1x16x1x16_4_Intrawave_v2.hip.o

I killed python312Packages.torchWithRocm because I didn't see much value building it and python313Packages.torchWithRocm and by this point I was tired of the review running :P

JohnRTitor · 2025-10-16T14:50:33Z

pkgs/development/rocm-modules/6/hipblaslt/default.nix

@@ -86,6 +86,8 @@ stdenv.mkDerivation (finalAttrs: {
  env.ROCM_PATH = "${clr}";
  env.TENSILE_ROCM_ASSEMBLER_PATH = lib.getExe' clr "amdclang++";
  env.TENSILE_GEN_ASSEMBLY_TOOLCHAIN = lib.getExe' clr "amdclang++";
+  env.LD_PRELOAD = "${jemalloc}/lib/libjemalloc.so";


Can we patchElf this?

Well it's just for the duration of the build AFAICT. If it's to affect the existing tooling in the store we'd either need to apply it always, build a second copy that has it patchelf-ed, or a wrapper which sets LD_PRELOAD.

IMO a comment explaining the benefits and maybe using lib.getLib would be sufficient tweaks

Suggested change

env.LD_PRELOAD = "${jemalloc}/lib/libjemalloc.so";

# has around x improvement (or y benefit) when running z

env.LD_PRELOAD = "${lib.getLib jemalloc}/lib/libjemalloc.so";

I'd say stdenv.hostPlatform.extensions.sharedLibrary would be unnecessary being linux only but could also be included

yeah this is for the allocation hungry build time TensileCreateLibrary processes, not relevant at runtime.

Without jemalloc:

Peak memory usage (MB): 31,011.9 Current memory usage (MB): 28,199.3

With jemalloc:

Peak memory usage (MB): 30,893.7 Current memory usage (MB): 22,695.9

…#451188)

LunNova added 2 commits October 11, 2025 18:03

rocmPackages.hipblaslt: add zstd to nativeBuildInputs

987e5a2

without this zstd compression of msgpack .dats silently failed

rocmPackages.hipblaslt: verify .dat files aren't huge in preInstallCheck

7baf5a9

let's avoid regressing compression in future, oops

LunNova force-pushed the push-nmzswnymunon branch from 1e9b995 to 0c86325 Compare October 12, 2025 01:03

LunNova marked this pull request as ready for review October 12, 2025 01:29

LunNova requested a review from 06kellyjac October 12, 2025 01:29

nix-owners bot requested review from Flakebi, GZGavinZhao and mschwaig October 12, 2025 01:30

LunNova marked this pull request as draft October 12, 2025 02:17

Exatio mentioned this pull request Oct 12, 2025

Build failure: rocmPackages.hipblaslt #449880

Closed

3 tasks

LunNova force-pushed the push-nmzswnymunon branch 3 times, most recently from 539440c to 4374426 Compare October 12, 2025 04:17

LunNova requested a review from JohnRTitor October 12, 2025 04:23

tv42 mentioned this pull request Oct 12, 2025

Build fails with 8GB space on /tmp - No space left on device #54707

Closed

LunNova marked this pull request as ready for review October 12, 2025 23:51

LunNova force-pushed the push-nmzswnymunon branch from 4374426 to a436d1c Compare October 13, 2025 00:14

Flakebi approved these changes Oct 13, 2025

View reviewed changes

nixpkgs-ci bot added 12.approvals: 1 This PR was reviewed and approved by one person. 12.approved-by: package-maintainer This PR was reviewed and approved by a maintainer listed in any of the changed packages. labels Oct 13, 2025

Kitt3120 approved these changes Oct 13, 2025

View reviewed changes

nixpkgs-ci bot added 12.approvals: 2 This PR was reviewed and approved by two persons. and removed 12.approvals: 1 This PR was reviewed and approved by one person. labels Oct 13, 2025

LunNova force-pushed the push-nmzswnymunon branch from a436d1c to a19680a Compare October 14, 2025 06:02

LunNova force-pushed the push-nmzswnymunon branch from a19680a to ff8c845 Compare October 14, 2025 06:12

Flakebi reviewed Oct 14, 2025

View reviewed changes

pkgs/development/rocm-modules/6/hipblaslt/default.nix Outdated Show resolved Hide resolved

LunNova added 2 commits October 14, 2025 08:20

rocmPackages.hipblaslt: apply PR to massively reduce peak disk space …

4cea628

…usage Peak build dir usage is now 25GB Partially applies [hipblaslt] Refactor Parallel.py to drop joblib, decimate resource usage

rocmPackages.hipblaslt: remove comment patch

bfd0d13

no longer needed now we unlink .s / .o files as soon as possible

LunNova force-pushed the push-nmzswnymunon branch from ff8c845 to bfd0d13 Compare October 14, 2025 15:20

Kitt3120 approved these changes Oct 15, 2025

View reviewed changes

06kellyjac approved these changes Oct 15, 2025

View reviewed changes

nixpkgs-ci bot added 12.approvals: 3+ This PR was reviewed and approved by three or more persons. and removed 12.approvals: 2 This PR was reviewed and approved by two persons. labels Oct 15, 2025

JohnRTitor reviewed Oct 16, 2025

View reviewed changes

JohnRTitor approved these changes Oct 16, 2025

View reviewed changes

JohnRTitor added this pull request to the merge queue Oct 16, 2025

Kitt3120 approved these changes Oct 16, 2025

View reviewed changes

Merged via the queue into NixOS:master with commit 1b3f5cf Oct 16, 2025
27 checks passed

LunNova deleted the push-nmzswnymunon branch October 16, 2025 17:16

PedroHLC pushed a commit to PedroHLC/nixpkgs that referenced this pull request Oct 17, 2025

rocmPackages.hipblaslt: massively reduce peak disk space usage (NixOS…

95d8e5f

…#451188)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

rocmPackages.hipblaslt: massively reduce peak disk space usage#451188

rocmPackages.hipblaslt: massively reduce peak disk space usage#451188
JohnRTitor merged 4 commits intoNixOS:masterfrom
LunNova:push-nmzswnymunon

LunNova commented Oct 12, 2025 •

edited

Loading

Uh oh!

LunNova commented Oct 12, 2025

Uh oh!

LunNova commented Oct 13, 2025

Uh oh!

Flakebi left a comment

Uh oh!

Kitt3120 left a comment

Uh oh!

LunNova commented Oct 14, 2025

Uh oh!

Uh oh!

06kellyjac commented Oct 15, 2025 •

edited

Loading

Uh oh!

JohnRTitor Oct 16, 2025

Uh oh!

06kellyjac Oct 16, 2025

Uh oh!

LunNova Oct 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	env.LD_PRELOAD = "${jemalloc}/lib/libjemalloc.so";
	# has around x improvement (or y benefit) when running z
	env.LD_PRELOAD = "${lib.getLib jemalloc}/lib/libjemalloc.so";

Uh oh!

Conversation

LunNova commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things done

Uh oh!

LunNova commented Oct 12, 2025

Uh oh!

LunNova commented Oct 13, 2025

nixpkgs-review result

x86_64-linux

Uh oh!

Flakebi left a comment

Choose a reason for hiding this comment

Uh oh!

Kitt3120 left a comment

Choose a reason for hiding this comment

Uh oh!

LunNova commented Oct 14, 2025

Uh oh!

Uh oh!

06kellyjac commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nixpkgs-review result

x86_64-linux

Uh oh!

JohnRTitor Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

06kellyjac Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

LunNova Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

LunNova commented Oct 12, 2025 •

edited

Loading

`nixpkgs-review` result

`x86_64-linux`

06kellyjac commented Oct 15, 2025 •

edited

Loading

`nixpkgs-review` result

`x86_64-linux`