Add AMD ROCm/HIP build support for nixlbench (follow-up to #1642)

[andyluo7]

Summary

Adds AMD ROCm/HIP build support for nixlbench (the canonical NIXL benchmark tool), gated behind a default-off use_rocm=true Meson option. Stacks on top of #1642 (which provides the same option for the NIXL library proper).

When use_rocm=true, nixlbench builds against ROCm/HIP using the existing CUDA-shaped source code unchanged — translation happens transparently at preprocess time via header-only symbol aliases.

Approach: header-only symbol translation

benchmark/nixlbench/src/utils/utils.h adds an #ifdef __HIP_PLATFORM_AMD__ block that #defines every CUDA runtime + driver API symbol used by nixlbench to its HIP equivalent:

// Status types
#define cudaError_t                          hipError_t
#define cudaSuccess                          hipSuccess
#define CUresult                             hipError_t
#define CUDA_SUCCESS                         hipSuccess

// Runtime API
#define cudaMalloc                           hipMalloc
#define cudaMemcpy                           hipMemcpy
#define cudaSetDevice                        hipSetDevice
// ... etc

// Driver API: virtual memory
#define CUdeviceptr                          hipDeviceptr_t
#define CUmemAllocationProp                  hipMemAllocationProp
#define cuMemCreate                          hipMemCreate
#define cuMemMap                             hipMemMap
// ... etc

.cpp source files stay unchanged with cu*/cuda* names. utils.cpp and nixl_worker.cpp had direct #include <cuda.h> / #include <cuda_runtime.h> lines that bypassed utils.h's translations — those are removed; utils.h now provides the right header for both paths.

Two narrow code patches needed beyond pure aliasing:

1. uintptr_t → void* cast at HIP virtual-memory call sites. CUDA's cuMemUnmap(CUdeviceptr=ulong, ...) accepts integer iov.addr implicitly; HIP's hipMemUnmap(void*, ...) rejects it. New NIXLB_DEV_PTR() macro (identity on CUDA, reinterpret_cast<void*> on HIP) wraps the call sites.
2. [[nodiscard]] on hipDrvGetErrorString — original macro discards return; (void) cast added.

What's enabled / disabled on ROCm builds

Component	ROCm path
UCX backend (NIXL library)	✅ via PR #1642
nixlbench binary	✅ via this PR
VRAM (GPU memory) transfers via UCX	✅ runtime-validated end-to-end
cuMem* virtual-memory allocator path (HAVE_CUDA_FABRIC)	⏭️ stays disabled — cpp.has_header_symbol('cuda.h', 'CU_MEM_HANDLE_TYPE_FABRIC') naturally fails on HIP, so the build falls through to the cudaMalloc/hipMalloc path
nvshmem worker	⏭️ unchanged (NVIDIA-only); future rocSHMEM work tracked separately

Runtime validation: nixlbench on AMD MI355X

Built and ran on AAC1 MI355X (gfx950, ROCm 7.2.0, container rocm/sgl-dev:v0.5.10.post1-rocm720-mi35x-20260503) with UCX 1.18.x --with-rocm. Two-process VRAM transfer via ASIO runtime, 50 iterations per block size:

Block Size (B)  Batch Size  B/W (GB/Sec)  Avg Lat (us)  Avg Tx (us)
4096            1           0.134640       30.4          14.1
8192            1           0.223101       36.7          11.5
16384           1           0.327987       50.0          10.0
65536           1           0.480172       136.5         11.5
1048576         1           0.621937       1686.0        4.0
16777216        1           0.637596       26313.2       0.0
67108864        1           0.637027       105347.0      0.0

Bandwidth saturates around 0.64 GB/s. This is not a peak-bandwidth number — UCX warned Driver ionic does not support the kernel ABI of 1 (supports 4 to 4) for the AAC1 RDMA HCAs, so UCX fell back to a non-RDMA transport. Same ionic driver issue we saw with the simpler vram_test in PR #1642's runtime evidence. The load-bearing claims here are correctness + scaling: payload transfers cleanly, latency scales linearly, two-process coordination works.

Stacks on PR #1642

This branch is built on top of #1642's commits (use_rocm option, hipcc toolchain, hw_info AMD detection, plugin disables). Both PRs need to be built with the same -Duse_rocm=true toolchain.

Once PR1 merges, this PR will be rebased onto main with no diff change.

Out of scope (intentional)

• Multi-node UCX RDMA — needs RDMA fabric setup; not blocking for nixlbench correctness.
• cuMem* virtual-memory path on HIP — would unlock the --enable_vmm mode. The cuMemCreate/cuMemAddressReserve/cuMemSetAccess block in nixl_worker.cpp:432-471 is gated on HAVE_CUDA_FABRIC which fails on HIP. If desired, follow-up could extend the translations + add CU_MEM_HANDLE_TYPE_FABRIC aliasing.
• rocSHMEM worker — equivalent of the existing nvshmem worker but for ROCm. Separate plugin if NIXL maintainers want it.

Test plan

• Build nixlbench on AAC1 MI355X with use_rocm=true (gfx950, ROCm 7.2.0, container)
• Confirm HIP linkage: ldd build/nixlbench | grep amdhip64 → libamdhip64.so.7
• Two-process VRAM transfer with --runtime_type ASIO, 50 iter, 4 KiB → 64 MiB block sweep — passes cleanly
• Plugin loading: NIXL UCX + POSIX plugins picked up by nixlbench's NIXL_PLUGIN_DIR

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added ROCm/HIP accelerator support alongside CUDA for benchmark runs, including memory allocation, copy, and error reporting on ROCm devices.
• Chores
  • Build/configuration now exposes an option to enable ROCm and will prefer ROCm linking when enabled; build flags and generated config reflect ROCm availability and warn if no accelerator is found.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi andyluo7! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[andyluo7]

Cross-node RDMA bandwidth on AMD MI355X + AINIC

Following up on the build-only evidence in the PR description: I now have
end-to-end cross-node RDMA bandwidth working on AMD hardware with no Mellanox
involvement.

Setup

• 2× AMD Instinct MI355X (gfx950) nodes on AAC1 (smci355-ccs-aus-g12-26 ↔ g12-30)
• AMD AINIC 400 Gb/s NDR ports (Pensando-derived ionic_0), RoCE v2
• AMD libionic1 54.0-184 userspace + ionic-dkms 26.01.6.002 (the
  AMD-shipped Pensando stack, not stock Ubuntu rdma-core 39)
• ROCm 7.2.2, UCX 1.18.1 with --with-rocm --with-verbs
• NIXL with use_rocm=true (Add AMD ROCm/HIP build support for AMD Instinct GPUs (gfx942, gfx950) #1642) + nixlbench with use_rocm=true (this PR)

Bandwidth — cross-node, RC verbs over RoCE v2

Tool	Block	Bandwidth
ib_write_bw (raw rdma-core)	64 KB	327 Gb/s ← ground-truth
ucx_perftest -t tag_bw	4 MB	299 Gb/s (37.4 GB/s)
nixlbench UCX backend (DRAM)	16 MB	341 Gb/s = 42.6 GB/s

nixlbench exceeds ucx_perftest at 16 MB blocks because larger blocks
amortize per-transfer overhead better than perftest's default config. The
NIXL → UCX → AINIC path delivers 85% of NIC line rate (341/400 Gb/s) on
AMD hardware.

Critical config detail worth surfacing

The cross-node UCX wireup needs an AM-capable auxiliary transport.
UCX_TLS=rc_verbs,self alone fails with no auxiliary transport ... rc_verbs/ionic_0:1 - no connect to iface even though the RC iface is
created and QPs allocated successfully (visible in UCX_LOG_LEVEL=debug).
Adding tcp to UCX_TLS resolves it:

export UCX_TLS=rc_verbs,tcp,self
export UCX_NET_DEVICES=ionic_0:1,<mgmt-iface>
export UCX_IB_GID_INDEX=1   # RoCE v2 IPv4 (not default link-local v1)

Where earlier sessions hit walls

Earlier I'd reported failures on Tensorwave's amd-aim partition with
the same Pensando hardware. Root cause was purely that Tensorwave ships
the stock Ubuntu rdma-core 39.0 (libionic-rdmav34.so), which only
exposes UD QPs to userspace — no RC. AMD's own libionic1 54.0-184
exposes RC + CM properly, and the entire NIXL stack works end-to-end
with no source changes.

This dependency is purely environmental (driver stack on the deployment
node), not anything the PR can or should address.

VRAM caveat

VRAM→VRAM cross-node hit a UCX runtime memory-type detection issue
(VRAM memory is detected as host by UCX) despite UCX 1.18 being built
with --with-rocm and reporting rocm_copy/rocm_ipc transports plus
memory types: host (access,reg,cache), rocm (reg,cache) for rc_verbs.
This looks like UCX-rocm UCM hook configuration tuning — the same kind of
runtime config required on NVIDIA paths too, separate from the NIXL/nixlbench
PR scope. DRAM cross-node and intra-node VRAM (validated in earlier sessions)
already cover the code path; VRAM cross-node is a deployment/UCX-config item.

[andyluo7]

Follow-up: VRAM cross-node bandwidth (root-caused + ceiling identified)

Closing the VRAM caveat from the previous comment. Quick differential
diagnosis revealed it was a UCX behavior gotcha, not a NIXL bug:

When UCX_TLS excludes rocm_copy / rocm_ipc, UCX closes the rocm
Memory Domain — but the rocm MD owns the hsa_amd_pointer_info query
path used to classify memory type. With it closed, ucp_mem_query returns
HOST for any address, including verified HIP device pointers (verified
in-process: HIP says hipMemoryTypeDevice, UCX says HOST on the same
address). Including rocm_copy,rocm_ipc in UCX_TLS keeps the MD alive
for detection.

Required env for AMD VRAM workloads:

export UCX_TLS=rc_verbs,tcp,self,rocm_copy,rocm_ipc

VRAM cross-node bandwidth (MI355X ↔ MI355X, AINIC, GPUDirect)

Block size	nixlbench VRAM BW
1 MB	20.8 GB/s
4 MB	26.9 GB/s
16 MB	28.9 GB/s = 231 Gb/s

NIXL overhead vs raw UCX = ~0%

ucx_perftest -m rocm cross-node, 16 MB, same env, same NICs: 230 Gb/s.
Identical to nixlbench's 231 Gb/s — confirms NIXL adds no measurable
overhead on top of UCX for the VRAM data path.

Updated bandwidth table

Tool	Memory	BW	% of 400 Gb/s line rate
ib_write_bw (raw RC)	DRAM	327 Gb/s	82%
ucx_perftest	DRAM	299 Gb/s	75%
nixlbench UCX backend	DRAM	341 Gb/s	85%
ucx_perftest -m rocm	VRAM (GPUDirect)	230 Gb/s	57%
nixlbench UCX backend	VRAM (GPUDirect)	231 Gb/s	58%

The 231 Gb/s VRAM number reflects the actual GPUDirect ceiling of the
MI355X + AINIC + libionic 54 + UCX 1.18 stack today, not anything in
NIXL's path. Future improvements would come from UCX 1.19+ / AMD's
rocm/ucx 1.22 fork / libionic dmabuf-path tuning — all outside the
scope of these two PRs.

[andyluo7]

@tvegas1 — yes, the gist's approach is what I'll adopt here. After we finalize the restructure on #1642 (dropping all ROCm-specific code from top-level nixl — discussion thread there), this PR will follow the gist exactly:

• Add use_rocm + rocm_path options to nixlbench's meson_options.txt (not top-level)
• Declare a separate rocm_dep (link -lamdhip64 -lhiprtc, include rocm/include), don't conflate with cuda_dep
• Add -D__HIP_PLATFORM_AMD__ only as a nixlbench project arg
• Conditionally pick cuda_dep OR rocm_dep per-target in utils/meson.build and worker/meson.build (your gist's exact pattern)
• Source-side: #if HAVE_CUDA / HAVE_ROCM guards around cuda_runtime.h / hip_runtime.h, VRAM allocation path, VMM disablement on ROCm

The cross-node bandwidth numbers in my earlier comments (341 Gb/s DRAM, 231 Gb/s VRAM cross-node on MI355X + AINIC) were captured with the current branch, so I'll re-run after the restructure to confirm the gist-style refactor doesn't regress anything.

Will push the restructure on #1642 first, then rebase this PR on top and apply the gist pattern. Should I post the refactored diff here for review before pushing, or just push and let you review the new commit directly?

[aranadive]

@andyluo7 can you update the PR / resolve conflicts?

[andyluo7]

@aranadive thanks for the ping. Heads up: the rebase is more than a textual conflict resolution — this PR's design is being changed per @tvegas1's review on the stacked PR #1642.

Where we are:

1. Add AMD ROCm/HIP build support for AMD Instinct GPUs (gfx942, gfx950) #1642 (the base PR) is at green-light state after @tvegas1's review pushed us to drop the use_rocm Meson option from top-level nixl entirely. Latest commit on Add AMD ROCm/HIP build support for AMD Instinct GPUs (gfx942, gfx950) #1642 (f72aad2c) removes the HIP project args, drops the use_rocm option, and adds a generic wheel_variant override — net -47 lines. CI green. Waiting on tvegas1's final ack before he merges.

2. Add AMD ROCm/HIP build support for nixlbench (follow-up to #1642) #1647 (this PR) was designed when ROCm support was expected to live partially in top-level nixl. With Add AMD ROCm/HIP build support for AMD Instinct GPUs (gfx942, gfx950) #1642's restructure, ALL HIP build support now needs to land here, but using @tvegas1's gist pattern, not the current header-only #define CUDA→HIP translation approach. The gist's pattern:

   • use_rocm / rocm_path options in benchmark/nixlbench/meson_options.txt (not top-level)
   • Separate rocm_dep (link -lamdhip64 -lhiprtc, include rocm/include), don't conflate with cuda_dep
   • -D__HIP_PLATFORM_AMD__ only as a nixlbench project arg
   • Conditional cuda_dep OR rocm_dep per-target in utils/meson.build and worker/meson.build
   • Source-side: #if HAVE_CUDA / HAVE_ROCM guards around cuda_runtime.h / hip_runtime.h, VRAM allocation, VMM disablement

A pure textual rebase here would actually be churn — the current commits all touch meson.build / meson_options.txt / utils.h in ways the gist redesign will undo.

Two options — happy with either, your call:

• (a) Wait for Add AMD ROCm/HIP build support for AMD Instinct GPUs (gfx942, gfx950) #1642 to merge (tvegas1's ack pending), then I push a full gist-style rewrite on top — single force-push, clean diff, no preserved history. This is the cleanest end-state.
• (b) I push a minimal textual rebase now (preserving the current #define approach) just to clear conflicts and keep CI green, then do the gist rewrite as a second commit once Add AMD ROCm/HIP build support for AMD Instinct GPUs (gfx942, gfx950) #1642 lands.

Option (a) is what tvegas1 implicitly asked for. Option (b) gets CI green faster but commits to throwaway intermediate state. Let me know which you prefer.

Cross-node bandwidth numbers I posted earlier (341 Gb/s DRAM, 231 Gb/s VRAM on MI355X + AINIC) were validated with the current branch's approach, so I'll re-run after the rewrite to confirm no regression.

[andyluo7]

@tvegas1 @edgargabriel @aranadive — thanks for the input. With #1642 now ✅ approved by @tvegas1 + @iyastreb, this PR's path forward is clear. Plan:

1. Full gist-style restructure (in progress)

Now that #1642 is locked in (drops use_rocm from top-level nixl), I'll rewrite this PR per @tvegas1's gist pattern:

• Move use_rocm / rocm_path options into benchmark/nixlbench/meson_options.txt (not top-level)
• Separate rocm_dep (link -lamdhip64 -lhiprtc, include rocm/include)
• -D__HIP_PLATFORM_AMD__ only as a nixlbench project arg
• Conditional cuda_dep OR rocm_dep per-target in utils/meson.build + worker/meson.build
• Source-side: #if HAVE_CUDA / HAVE_ROCM guards around runtime headers, VRAM allocation, VMM disablement
• Drops the current header-only #define CUDA→HIP translation approach

This is a force-push (single clean diff vs preserving the current commits). Will re-validate the cross-node bandwidth numbers (341 Gb/s DRAM / 231 Gb/s VRAM on MI355X + AINIC) after the rewrite.

2. ROCm Dockerfile (re: @tvegas1 on README.md:166)

Proposing to land it here in this PR (so the gist-style nixlbench build is reproducible end-to-end). Will use rocm/dev-ubuntu-24.04:7.1.1-complete as base, UCX 1.21.x with --with-rocm --with-verbs, and document the dmabuf workaround (-v /boot:/boot:ro + UCX_ROCM_{COPY,IPC}_DMABUF=yes + UCX_ROCM_IPC_MIN_ZCOPY=0) we found during the AINIC validation work.

3. nixl_rocm in release (re: @tvegas1) — agreed. #1642 already adds the wheel_variant Meson option; the release workflow can pass -Dwheel_variant=rocm to emit nixl_rocm wheels. I'll wire that into the release config either here or in a small follow-up — your call which.

4. ROCm CI (re: @edgargabriel) — your short-term compile-only proposal sounds great. Happy to:

• Provide a compile recipe via the Dockerfile from (2) above for the runner to consume
• Test the runner from my side once it lands (I have AAC1 MI355X + Hotaisle MI300X access for ad-hoc validation)
• Coordinate with @kahalon on the medium-term private-runner plan when you're ready

@aranadive — to your earlier ping: the rebase-vs-redesign decision is settled (option a from my last reply). I'll push the restructure + Dockerfile in the next day or so once I've finished the gist-style rewrite locally. CI will be green on the new commit, no incremental textual rebase needed.

[andyluo7]

Pushed fd9ffb8 — full gist-style rewrite + ROCm Dockerfile. Force-push; single commit; +263/-9 lines across 9 files.

 benchmark/nixlbench/contrib/Dockerfile.rocm        | 158 +++++++++++++++++++
 benchmark/nixlbench/meson.build                    |  43 ++++-
 benchmark/nixlbench/meson_options.txt              |   2 +
 benchmark/nixlbench/src/utils/meson.build          |   7 +-
 benchmark/nixlbench/src/utils/utils.cpp            |  14 +-
 benchmark/nixlbench/src/utils/utils.h              |  12 ++
 benchmark/nixlbench/src/worker/meson.build         |   7 +-
 benchmark/nixlbench/src/worker/nixl/meson.build    |   2 +
 benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp|  27 ++-
 9 files changed, 263 insertions(+), 9 deletions(-)

Mapping to the gist (@tvegas1):

• ✅ use_rocm + rocm_path options in benchmark/nixlbench/meson_options.txt (NOT top-level — Add AMD ROCm/HIP build support for AMD Instinct GPUs (gfx942, gfx950) #1642 already drops them from top-level nixl)
• ✅ Separate rocm_dep declared with HIP link/include (not a cuda_dep shim)
• ✅ -D__HIP_PLATFORM_AMD__ as a nixlbench-scoped project arg only
• ✅ Conditional cuda_dep OR rocm_dep per target in utils/, worker/, worker/nixl/ meson.build files
• ✅ Source-side #if HAVE_ROCM guards include hip/hip_runtime.h, map CHECK_CUDA_ERROR onto hipSuccess / hipGetErrorString, route VRAM hipMalloc/hipMemset/hipMemcpy/hipFree, and disable VMM on ROCm
• ✅ The earlier header-only #define cuda→hip translation approach is gone

Dockerfile (@tvegas1's req from this PR + #1642):

• Base: rocm/dev-ubuntu-24.04:7.1.1-complete per your sketch
• UCX 1.21.x built from source with --with-rocm --with-verbs (Ubuntu's rdma-core UCX lacks ROCm support)
• Documents the two-part dmabuf workaround we validated on AINIC: -v /boot:/boot:ro + UCX_ROCM_COPY_DMABUF=yes + UCX_ROCM_IPC_DMABUF=yes + UCX_ROCM_IPC_MIN_ZCOPY=0
• Builds nixl with -Dwheel_variant=rocm (uses the option Add AMD ROCm/HIP build support for AMD Instinct GPUs (gfx942, gfx950) #1642 introduces) → nixl_rocm wheel naming
• Builds nixlbench with -Duse_rocm=true

Outstanding:

• @edgargabriel — Dockerfile is ready to consume for the compile-only CI you mentioned. Happy to coordinate.
• @aranadive — to your conflict-resolution ping: this force-push subsumes the previous textual conflicts; CI will rebuild from a clean base on top of main.
• I'll re-run the cross-node bandwidth validation (was 341/231 Gb/s DRAM/VRAM on MI355X+AINIC with the old approach) on this rewrite before declaring the PR ready for final review.

Depends on #1642 landing (the wheel_variant option). #1642 has both approvals — once it merges, this PR is structurally complete pending the BW re-validation.

[dpressle]

👀 Investigating Copyright Checks…

[dpressle]

🤖 CI Triage Agent — Copyright Checks · commit fd9ffb8b

I have everything needed for a complete diagnosis. Here is the full breakdown:

---

Summary: Copyright check fails because benchmark/nixlbench/meson_options.txt has a stale SPDX copyright year (2025) while the file was last modified in 2026.

Root cause: The copyright-check.sh script compares the year declared in the SPDX-FileCopyrightText header against the file's last-modified year. Line 1 of benchmark/nixlbench/meson_options.txt reads:

# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This PR (nixlbench-rocm, commit fd9ffb8b) touched the file in 2026 without bumping the copyright year in the header, triggering the exact error:

❌ SPDX header check failed:
  - benchmark/nixlbench/meson_options.txt (copyright year 2025 < last modified 2026)

Implicated commit: [REDACTED:Hex High Entropy String] (author unknown from logs; most recent prior recorded commits are 2001a31d by ovidiusm and 6e06af14 by Subhadeep Bhattacharya)

File: benchmark/nixlbench/meson_options.txt:1

Suggested fix: Update the copyright year on line 1 of benchmark/nixlbench/meson_options.txt from 2025 to 2025-2026 (or just 2026):

-# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Using the range form (2025-2026) is the conventional NVIDIA style and acknowledges both the original creation year and the current modification year.

Related: none

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 0e839a8f-9890-401e-9259-74d1dcd47537 in the triage console for the audit trail.

[andyluo7]

@tvegas1 great question on the Dockerfile. Quick analysis before I act:

Sharing with contrib/Dockerfile is impractical — the existing top-level Dockerfile is layered on nvcr.io/nvidia/cuda-dl-base:25.10-cuda13.0-devel-ubuntu24.04 with CUDA-specific install steps and conditional UCX stages keyed off the CUDA base. ROCm needs a different base (rocm/dev-ubuntu-24.04:7.1.1-complete), a different verbs stack (rdma-core + libibverbs from Ubuntu, not the cuda-dl-base's), and UCX configured --with-rocm instead of --with-cuda. Trying to fold them into one Dockerfile would require either a HW_VENDOR ARG with two giant conditional branches, or factoring shared steps into a base layer — both add reviewer cost without much code reuse.

Splitting into a separate PR is the right call — your option (b). Going to:

1. Drop the Dockerfile from this PR in the next push, so Add AMD ROCm/HIP build support for nixlbench (follow-up to #1642) #1647 becomes pure source + meson changes (single small reviewable scope: ROCm build support for nixlbench).
2. Open a follow-up PR (call it PR3) with the Dockerfile, ROCm CI build stub for @edgargabriel, and any contrib/*.sh modifications needed. That PR can iterate independently on the Docker base / UCX build matrix / dmabuf env-var documentation without re-blocking the actual code changes here.
3. Wheel generation — the -Dwheel_variant=rocm option from Add AMD ROCm/HIP build support for AMD Instinct GPUs (gfx942, gfx950) #1642 already handles nixl_rocm naming. The release workflow plumbing (.github/workflows/ whatever-emits-wheels) likely belongs in PR3 as well so we keep the ROCm-packaging surface in one place.

Will push the Dockerfile drop shortly. CI (copyright-checks was failing on stale SPDX year in meson_options.txt) is fixed in 9330c15 — bumped to 2025-2026 per @dpressle's triage.

[andyluo7]

Dockerfile dropped in 608b1a2 — PR is now source + meson only:

 benchmark/nixlbench/meson.build                    | 43 +++++-
 benchmark/nixlbench/meson_options.txt              |  4 +-
 benchmark/nixlbench/src/utils/meson.build          |  7 ++-
 benchmark/nixlbench/src/utils/utils.cpp            | 14 ++-
 benchmark/nixlbench/src/utils/utils.h              | 12 ++
 benchmark/nixlbench/src/worker/meson.build         |  7 ++-
 benchmark/nixlbench/src/worker/nixl/meson.build    |  2 +
 benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp| 27 ++-
 8 files changed, 106 insertions(+), 10 deletions(-)

Will open PR3 with the Dockerfile + CI compile job + any contrib/*.sh modifications + wheel-generation workflow plumbing once #1642 and this PR land, so the ROCm-packaging surface stays in one place.

[andyluo7]

🎉 #1642 just merged (313121be on main) — this PR is now rebased on top in 241c6e4. Force-pushed; single commit; zero conflicts.

Diff against main is unchanged in shape (8 files / +106 -10):

 benchmark/nixlbench/meson.build                    | 43 ++++++++++++++++++--
 benchmark/nixlbench/meson_options.txt              |  4 +-
 benchmark/nixlbench/src/utils/meson.build          |  7 ++-
 benchmark/nixlbench/src/utils/utils.cpp            | 14 +++++--
 benchmark/nixlbench/src/utils/utils.h              | 12 ++++++
 benchmark/nixlbench/src/worker/meson.build         |  7 ++-
 benchmark/nixlbench/src/worker/nixl/meson.build    |  2 +
 benchmark/nixlbench/src/worker/nixl/nixl_worker.cpp| 27 +++++++++++-

CI should re-run on the rebased HEAD shortly.

---

Review-ask resolution table

Asker	Comment	Plan	Status
@tvegas1	check the gist, see if the approach works (utils.h)	Adopted exactly: separate rocm_dep, per-target conditional, source-side HAVE_ROCM guards	✅ done
@tvegas1	"We would need to add the ROCm build to CI somehow" (meson.build)	@edgargabriel volunteered compile-only CI mirroring UCC/UCX, short-term	🟡 owned by @edgargabriel
@tvegas1	"Dockerfile with ROCm libraries + nixl_rocm to release" (README.md)	Deferred to follow-up PR3 — keeps ROCm-packaging surface (Dockerfile + CI compile job + nixl_rocm release wheel workflow) in one place	🟡 planned for PR3
@edgargabriel	"can you ping me separately... compile-only CI first, private runner medium-term"	Will coordinate with you; AAC1 MI355X + Hotaisle MI300X available for validation runs	🟡 active

Pending on my side

• BW re-validation on this rewrite — previously measured 341 Gb/s DRAM / 231 Gb/s VRAM cross-node on MI355X+AINIC with the old header-only #define approach. Will re-run against the gist-style code path as soon as MI355X access frees up and post results here as evidence of no regression.

Ready for a fresh look from any of @tvegas1 @aranadive @iyastreb @brminich @edgargabriel — your approvals on #1642 carried this through, would love a re-review on this slimmer follow-up.