[fedeizzo]

What this is

A side-by-side ROCm 10.1.0 build of Gufo, so the toolchain can be measured and qualified without moving the 7.2.3 pin. The default package, release output, checks and development shell are unchanged; the default ROCr derivation is identical to main.

This supersedes the packaging in #228 (a vendored ROCm 10.0 module tree). That PR's investigation notes and negative results still apply; its prefill_quant_gemm.hip SLP hunk is a no-op on current main (#459 removed those spills).

nix build .#rocm10 --out-link result-rocm10
./result-rocm10/bin/gufo diagnose      # ROCm Version: 10.1.0
nix develop .#rocm10                   # same build plus rocprofiler-sdk, ccache, gdb

Changes

• build(nix)
  • New flake input nixpkgs-rocm10, pinned to the head of rocmPackages: 7.2.3 -> 10.1.0 NixOS/nixpkgs#573885 (925eadb6, ROCm 7.2.3 → 10.1.0, HIP Clang 24). The whole package set comes from that commit, so glibc, stdenv and the ROCm compiler stay consistent.
  • packages.rocm10 uses rocmPackages.gfx1151, which builds the kernel libraries for gfx1151 only.
  • devShells.rocm10 leaves out the ROCm torch reference tools, which would mean rebuilding torch.
  • rocr-fixed-cwsr-rocm10.patch rebases the queue-memory patch onto ROCr 1.21:
    • The queues.c hunk follows the move to ctx->hsakmt_is_svm_api_supported.
    • MemoryRegion::Lock now clears NonPaged for caller-registered memory. ROCr 10.1 passes the region flags to RegisterMemory/MakeMemoryResident, and libhsakmt rejects NonPaged there. Without this, every model load fails with mapped registration: invalid argument.
  • README documents the outputs as an unqualified evaluation build.
• fix(diagnose): gufo diagnose reported the hard-coded 7.2.3 regardless of the toolchain. It now reports the rocm-core version Gufo was built against, when CMake finds it. On the Nix 7.2.3 package the output is unchanged. Builds without Nix now show the installed ROCm, and because the machine fingerprint includes that version, its ID changes when that ROCm is not 7.2.3.
• build(cmake): /opt/rocm was appended to CMAKE_PREFIX_PATH, which CMake searches before the environment's prefixes. In a development shell on a host with its own /opt/rocm, find_package(hip) therefore resolved the host's HIP: the rocm10 shell mixed ROCm 7.2.3 headers into a clang 24 build and failed. The default shell had the same problem, unnoticed because this host's /opt/rocm is also 7.2.3. /opt/rocm is now a system prefix, searched after the shell's ROCm. A system with only /opt/rocm still finds it. nix build is unaffected, since its sandbox has no /opt/rocm.

Effect on the ROCm 7.2.3 build

• The default ROCr derivation, including the queue-memory patch, is identical to main (8vh4vz32…-rocm-runtime-7.2.3.drv). The patch is selected by ROCr version, so 7.2.3 keeps the original patch.
• No kernel or inference code changes. The default binary differs only in gufo diagnose, which still reports 7.2.3 on the Nix package.
• nix build, .#release and checks.pr never evaluate the new input. Commands that evaluate every output (nix flake check, nix flake show) fetch the fork's nixpkgs source, but build nothing from it.
• build/rocm7 in nix develop now resolves HIP from Nix's clr 7.2.3 instead of the host's /opt/rocm/hip, and builds.

Evidence so far

Strix Halo gfx1151, kernel 7.2.9, same source revision for both arms.

• ROCm 10.1 build time: about 35 minutes for 71 derivations on 32 cores. There is no binary cache.

• Queue-memory patch: with HSAKMT_DEBUG_LEVEL=6, the patched ROCr 10.1 never prints Allocating GTT for CWSR; unpatched 10.1 prints it for every queue.

• The patch also matters for speed. Qwen3.8-27B UD-Q4_K_XL, one repetition each:

  ROCr 10.1	pp64	pp2048	tg128
  unpatched	269.1	516.7	10.92
  patched	336.6	645.3	12.08

• Qwen3.8 Flash-Next UD-Q4_K_XL, 7.2.3 vs 10.1. Two alternating runs per arm, 3 repetitions each, no codegen changes, tokens/s:

  Test	7.2.3	10.1	Δ
  pp64	374.5	359.8	-3.9% (10.1 stdev 20–46, inconclusive)
  pp2048	1789.8	1774.0	-0.9%
  tg128	29.32	29.45	+0.4%
  pp64 @ d4096	362.9	362.4	-0.2%
  pp2048 @ d4096	1733.7	1717.3	-0.9%
  tg128 @ d4096	28.72	28.81	+0.3%

  Three unrelated embedding servers stayed loaded on the GPU during these runs; I did not verify that they were idle. No quality check has been run yet.

Risks before switching the default

From NixOS/nixpkgs#573885

• Unmerged and moving. Reviewers asked to split it, starting with a separate rocmPackages_10 set. The pinned commit may be rewritten, or the fork branch removed; flake.lock keeps the hash, but we would need to mirror the commit.
• The whole nixpkgs comes from the fork's base (master as of 2026-10-14): glibc, gcc 16, openssl, curl, ffmpeg. That base gets no nixos-unstable updates until the PR is rebased.
• Unfinished by the PR's own account:
  • The full-set rebuild after the latest clr fix is not done.
  • LLVM 24 is built on the llvmPackages_23 base, because nixpkgs has no llvmPackages_24.
  • Hardware testing covered gfx1102, gfx908 and gfx906, not gfx1151.
• No binary cache. Every flake user and every devlab job rebuilds LLVM and the BLAS libraries, and devlab runners start empty each job.

In Gufo

• Numerics. Clang 23+ changes Flash-Next greedy text (that unit builds with -ffast-math, see ROCm 10 evaluation for gfx1151, and a clang 23 prefill fix #228), and 27B Q4 --validate-prefill shifts slightly. Different output is expected; equal quality is not yet shown.
• Performance beyond Flash-Next. In ROCm 10 evaluation for gfx1151, and a clang 23 prefill fix #228, @sdwilsh measured 27B Q4/Q8 on stock 10.1 at -1.5% to -13%. Most of that is recoverable with per-file codegen changes (no real-true16 on prefill_fp16.hip; no SLP on prefill_gemm.hip and small_batch_wave64.hip; a WmmaCausalAttention<32,16,false> spill fix). Q4 decode stays about 3% slower. None of those changes are in this PR.
• Queue-memory patch. It is ported and keeps CWSR off movable pages, but survival of a real compaction is untested.
• hipBLASLt plan database. Plans are keyed by the HIP and hipBLASLt versions, so 10.1 misses every stored plan and uses hipBLASLt's own heuristics until they are re-tuned.
• Builds without Nix. README qualifies 7.2.3. AMD's prebuilt ROCm 10.0 could not create a HIP stream on Strix Halo (ROCm 10 evaluation for gfx1151, and a clang 23 prefill fix #228); 10.1's prebuilt packages are untested.

Qualification plan

Each stage compares the same revision on 7.2.3 and 10.1. A stage gates the next.

1. Compile and operator checks.
   • tools/ci/check-kernel-resources.py --compare across the two toolchains, to list every kernel whose spills or VGPRs change.
   • ctest --preset gpu-fast in nix develop .#rocm10: the Flash-Next, Qwen and DS4 operator oracles.
2. Numerical quality.
   • Matched-token KL/NLL against the reference (method). 10.1 must stay within the model's declared limits and no worse than 7.2.3.
   • --validate-prefill on Flash-Next, 27B Q4/Q8 and DS4.
   • DS4's ds4.target and ds4.dspark.
   • Flash-Next session, snapshot and continuation-adapter tests.
   • MTP and DFlash2 acceptance rates, and greedy faithfulness.
3. Serving. tests/functional/run.py on Flash-Next across all four profiles: long-context, cache, tools, sampling, structured output and concurrency suites, plus a pi_agent.py replay with inspected tool results. Correctness is strict; timings use the per-request 5%/3 ms gate.
4. Memory pressure and soak.
   • Repeated /proc/sys/vm/compact_memory during long Flash-Next generations, with no stalls, evictions or amdgpu/KFD errors in dmesg.
   • over_ram_qwen.py and device_loss.py.
   • An 8–24 h concurrent agent soak on both toolchains, tracking RSS, GTT, latency drift and dmesg.
5. Performance.
   • Re-tune hipBLASLt plans for 10.1 first.
   • Then the full BENCHMARKS.md set: Flash-Next, 27B Q4/Q8, DS4, and image/audio if 10.1 becomes the only toolchain.
   • Fix regressions beyond noise.
6. Distribution.
   • The ROCm 10 set lands in nixpkgs, or its pinned commit is mirrored.
   • A gfx1151 binary cache.
   • Qualified-toolchain docs (README, THIRD_PARTY_NOTICES.md) move to 10.1.
   • The non-Nix ROCm 10.1 install is checked.

The pin moves only after stages 1–5 pass and stage 6 has a source; the rocm10 output is then removed.

[github-actions]

⚠️ Devlab functional checks · PR tier: Cluster job cancelled

_{Workflow run · PR e998567 vs merge base cc5f405 · model flash-next-q4 · 4 deferred, 9 for the release run, 352 out of scope
devlab/functional is advisory on PRs; only the automatic release merge waits for it. Candidate correctness only; other-model, numerical, agent replay and timing qualification remain separate.}