Conversation
|
👋 Hi andyluo7! Thank you for contributing to ai-dynamo/nixl. Your PR reviewers will review your contribution then trigger the CI to test your changes. 🚀 |
|
👀 Investigating |
|
🤖 CI Triage Agent — The PR branch files aren't in the default branch yet. The log evidence is sufficient for a definitive diagnosis. Here is the full analysis: Summary: The Root cause: The workflow file ( Implicated commit: File: Suggested fix: There are two complementary actions needed:
Related: PR #1735 — contrib/rocm: add ROCm Dockerfile + compile-only CI + wheel workflow
|
|
@dpressle thanks for the precise diagnosis — all three recommendations are valid and addressed in 1. Image not published (the bootstrap problem) — Confirmed. The image build is in flight on AAC1 (separate workstream); image namespace is moving to the org per #2 below before the push, so the URI in this PR is the final target. 2. Move from personal namespace to org namespace — ✅ done. Retargeted all three references from 3. Gate the build-check on image availability — ✅ done in a slightly different shape. Added an Where this leaves the PR:
Visual review of the Dockerfile + workflow YAML welcomed in the meantime. |
| # --cap-add=IPC_LOCK --ipc=host \ | ||
| # -v /boot:/boot:ro \ | ||
| # -e UCX_ROCM_COPY_DMABUF=yes \ | ||
| # -e UCX_ROCM_IPC_DMABUF=yes \ |
There was a problem hiding this comment.
UCX_ROCM_IPC_DMABUF does not exist as a parameter in UCX, only UCX_ROCM_COPY_DMABUF exists.
|
@edgargabriel thanks for the correction — verified against upstream UCX v1.21.x source and you're right: // src/uct/rocm/copy/rocm_copy_md.c — has DMABUF config field ✅
{"DMABUF", ...}
// src/uct/rocm/ipc/rocm_ipc_md.c — config table is empty (only inherits md super) ❌
static ucs_config_field_t uct_rocm_ipc_md_config_table[] = {
{"", "", NULL, ucs_offsetof(uct_rocm_ipc_md_config_t, super),
UCS_CONFIG_TYPE_TABLE(uct_md_config_table)},
{NULL}
};So
CI on the amended HEAD should be unchanged (still all public checks ✅, ROCm compile-only correctly skipped on Draft). |
da8b66b to
397a14b
Compare
| # faster than rebuilding the whole stack each time, and it isolates | ||
| # regression risk to nixl/nixlbench code rather than infra. | ||
| # | ||
| # Future (medium-term): private GitHub runner with real MI300X/MI355X |
There was a problem hiding this comment.
Future -> TODO: to be able to grep. also separate from header.
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # | ||
| # Short-term compile-only ROCm CI per @edgargabriel's ask on #1647: |
There was a problem hiding this comment.
maybe don't add discussion in the file just short descriptive comment?
| WORKDIR /workspace | ||
| COPY . /workspace/nixl | ||
|
|
||
| # NIXL — emits a `nixl_${WHEEL_VARIANT}` wheel via #1642's wheel_variant option. |
There was a problem hiding this comment.
don't reference commits in comment
| @@ -0,0 +1,110 @@ | |||
| # NIXL + nixlbench on AMD ROCm/HIP | |||
|
|
|||
| Validated on AMD Instinct MI300X (`gfx942`) and MI355X (`gfx950`) with | |||
There was a problem hiding this comment.
Supported instead of validated
| ghcr.io/ai-dynamo/nixl-rocm:base-latest | ||
| ``` | ||
|
|
||
| This image has UCX 1.21.x built with `--with-rocm --with-verbs`, all |
There was a problem hiding this comment.
do not reference version here?
|
|
||
| ## Stacks on | ||
|
|
||
| - ai-dynamo/nixl#1642 — adds `wheel_variant` Meson option (merged) |
There was a problem hiding this comment.
don't reference dev/commit details at all
|
|
||
| ## CI integration | ||
|
|
||
| - **`rocm-build-check.yml`** — compile-only CI on every PR touching |
There was a problem hiding this comment.
need to remove all references to dev workflow, and users comments.
| @@ -0,0 +1,196 @@ | |||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
There was a problem hiding this comment.
WDYT with approach described in gist:
FROM <img> as common-base
...
FROM common-base as rocm-build
...
FROM common-base as cuda-build
...Also maybe add similar change to .py as listed in the GIST.
Restructure contrib/Dockerfile into three stages: - common-base: shared apt deps, Abseil/gRPC/etcd-cpp source builds, python venv + uv, gtest-parallel, pybind11 - rocm-build: UCX configured --with-rocm, PyTorch from rocm wheels, NIXL meson with disable_plugins=LIBFABRIC,GDS,GDS_MT,GPUNETIO and build_nixl_ep=false; wheel name = nixl_rocm - cuda-build: DOCA install, libfabric from source, UCX --with-cuda, PyTorch CUDA wheels, optional NIXL EP build contrib/build-container.sh: add --rocm flag toggling base image to rocm/dev-ubuntu-24.04:7.2.4-complete, target rocm-build, wheel name nixl_rocm, PyTorch ROCm index, BUILD_NIXL_EP=false. contrib/build-wheel.sh: add --wheel-name and --backend-dir args, skip nvcc CUDA detection when WHEEL_NAME provided, pass new python_backend_dir/python_backend_package meson opts, extend auditwheel excludes to cover ROCm runtime libs. meson.build + meson_options.txt: add python_backend_dir and python_backend_package options that override cuda_wheel_dir / package name (used by contrib/build-wheel.sh). Compatible with the existing wheel_variant option from ai-dynamo#1642 - python_backend_dir wins when set. src/bindings/python/nixl-meta/__init__.py: add _get_torch_rocm_version, rename _load_cuda_backend -> _load_backend, dispatch to nixl_rocm before nixl_cu* when torch reports a HIP version. src/bindings/python/nixl-meta/meson.build: use python_backend_package for the WHEEL_DEPS template substitution so the meta wheel correctly depends on nixl-rocm when built for ROCm. Closes architectural ask on PR ai-dynamo#1735. Signed-off-by: andyluo7 <andy.luo@amd.com>
andyluo7
changed the title
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds ROCm backend detection and dynamic backend loading, Meson options for backend naming, multi-stage Docker builds and scripts for ROCm/CUDA wheel creation, CI workflows to validate and publish ROCm wheels, and ROCm contributor documentation. ChangesROCm Backend and Build Infrastructure
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related issues
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
Merged latest The only conflict was in
The rest of #1728 (the @aranadive — branch is rebased and conflict-free, tvegas1 has approved. Could you trigger CI approval / |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@contrib/Dockerfile`:
- Around line 334-336: Remove the `apt-get upgrade -y` line from the Dockerfile
to ensure reproducible builds. The apt-get update and apt-get install commands
should remain, but delete the middle line containing only the upgrade command.
This prevents non-deterministic package version drift between builds while still
allowing the doca-sdk packages and dependencies to be installed correctly.
- Around line 332-334: The Dockerfile downloads the doca-host.deb package
without verifying its integrity, creating a supply-chain security risk. After
the wget command downloads
https://www.mellanox.com/downloads/DOCA/DOCA_v3.3.0/host/doca-host_3.3.0-088000-26.01-${MELLANOX_OS}_${ARCH_SUFFIX}.deb
to doca-host.deb, add a step to download and verify a checksum file (or GPG
signature if available) from the Mellanox repository. Verify the downloaded
doca-host.deb against this checksum before proceeding with the dpkg -i command,
and fail the build if verification fails.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: d7c33bba-c8c9-4b1d-bb48-beeb782e8cc4
📒 Files selected for processing (3)
contrib/Dockerfilemeson.buildmeson_options.txt
| wget --tries=3 --waitretry=5 --no-verbose https://www.mellanox.com/downloads/DOCA/DOCA_v3.3.0/host/doca-host_3.3.0-088000-26.01-${MELLANOX_OS}_${ARCH_SUFFIX}.deb -O doca-host.deb && \ | ||
| dpkg -i doca-host.deb && \ | ||
| apt-get update && \ |
There was a problem hiding this comment.
Verify downloaded DOCA artifact integrity before dpkg -i.
This installs a remotely downloaded .deb as root without any checksum/signature verification, which is a supply-chain gap.
🔒 Suggested hardening patch
+ARG DOCA_HOST_SHA256
RUN ARCH_SUFFIX=$(if [ "${ARCH}" = "aarch64" ]; then echo "arm64"; else echo "amd64"; fi) && \
MELLANOX_OS="$(. /etc/lsb-release; echo ${DISTRIB_ID}${DISTRIB_RELEASE} | tr A-Z a-z | tr -d .)" && \
wget --tries=3 --waitretry=5 --no-verbose https://www.mellanox.com/downloads/DOCA/DOCA_v3.3.0/host/doca-host_3.3.0-088000-26.01-${MELLANOX_OS}_${ARCH_SUFFIX}.deb -O doca-host.deb && \
+ echo "${DOCA_HOST_SHA256} doca-host.deb" | sha256sum -c - && \
dpkg -i doca-host.deb && \🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@contrib/Dockerfile` around lines 332 - 334, The Dockerfile downloads the
doca-host.deb package without verifying its integrity, creating a supply-chain
security risk. After the wget command downloads
https://www.mellanox.com/downloads/DOCA/DOCA_v3.3.0/host/doca-host_3.3.0-088000-26.01-${MELLANOX_OS}_${ARCH_SUFFIX}.deb
to doca-host.deb, add a step to download and verify a checksum file (or GPG
signature if available) from the Mellanox repository. Verify the downloaded
doca-host.deb against this checksum before proceeding with the dpkg -i command,
and fail the build if verification fails.
|
Follow-up after rebasing onto
Real fix (follow-up): switch rocm-build-check to consume the pre-built All other checks are green. @tvegas1 — the rebase dismissed your approval; would you re-approve when you get a chance? @aranadive — still need a vetter |
|
Update: Root cause of the earlier failures wasn't the build — the full graph (compile, UCX, nixl, wheel, auditwheel, pip-install) succeeded every time. The runner was being OOM-killed/recycled at the final Fix (
This still complements the durable plan (consume a pre-built @tvegas1 the rebase dismissed your approval — would you re-approve? @aranadive still need a vetter |
| runs-on: ubuntu-latest | ||
| # Non-blocking for now: a from-scratch ROCm build exceeds the hosted-runner | ||
| # budget (the "-complete" image plus building UCX/gRPC/etc. from source runs | ||
| # well past the timeout). Flip to blocking once the build consumes the |
There was a problem hiding this comment.
can we fix the timeout issue instead?
|
|
||
| - name: Smoke - import nixl, dispatch to nixl_rocm backend | ||
| # The ROCm "-complete" base image is ~30GB extracted and does not fit on a | ||
| # stock ubuntu-latest runner (~14GB free); reclaim space before building. |
There was a problem hiding this comment.
maybe we should fix the runner to use bigger disk?
| # dispatches to the ROCm backend. Running these here lets CI validate the build | ||
| # with --output=cacheonly, avoiding the multi-GB image export that OOMs the | ||
| # hosted runner during layer compression. | ||
| RUN ls /workspace/nixl/dist/nixl_rocm-*.whl /workspace/nixl/dist/nixl-*-none-any.whl && \ |
There was a problem hiding this comment.
can we rather use the same scheme as done for nixl_cu*wheels?
| docker build --platform linux/$ARCH -f $DOCKER_FILE $DOCKER_TARGET $BUILD_ARGS $TAG $NO_CACHE $BUILD_CONTEXT | ||
| if [ -n "$CHECK_ONLY" ]; then | ||
| # Build the full graph (incl. in-Dockerfile smoke RUNs) but do not export an | ||
| # image. Skipping the multi-GB image export avoids OOM-ing hosted CI runners. |
There was a problem hiding this comment.
we should probably rather fix CI runners?
|
@tvegas1 thanks — agreed on all four points. The hosted-runner workarounds (free-disk, Target scheme (mirrors
This is also exactly how ROCm/TheRock ships ROCm + PyTorch ROCm wheels (pypa Two things to sort out:
Given the runner dependency, I'd suggest landing the |
2321f5f to
aa968c7
Compare
Restructure contrib/Dockerfile into three stages: - common-base: shared apt deps, Abseil/gRPC/etcd-cpp source builds, python venv + uv, gtest-parallel, pybind11 - rocm-build: UCX configured --with-rocm, PyTorch from rocm wheels, NIXL meson with disable_plugins=LIBFABRIC,GDS,GDS_MT,GPUNETIO and build_nixl_ep=false; wheel name = nixl_rocm - cuda-build: DOCA install, libfabric from source, UCX --with-cuda, PyTorch CUDA wheels, optional NIXL EP build contrib/build-container.sh: add --rocm flag toggling base image to rocm/dev-ubuntu-24.04:7.2.4-complete, target rocm-build, wheel name nixl_rocm, PyTorch ROCm index, BUILD_NIXL_EP=false. contrib/build-wheel.sh: add --wheel-name and --backend-dir args, skip nvcc CUDA detection when WHEEL_NAME provided, pass new python_backend_dir/python_backend_package meson opts, extend auditwheel excludes to cover ROCm runtime libs. meson.build + meson_options.txt: add python_backend_dir and python_backend_package options that override cuda_wheel_dir / package name (used by contrib/build-wheel.sh). Compatible with the existing wheel_variant option from ai-dynamo#1642 - python_backend_dir wins when set. src/bindings/python/nixl-meta/__init__.py: add _get_torch_rocm_version, rename _load_cuda_backend -> _load_backend, dispatch to nixl_rocm before nixl_cu* when torch reports a HIP version. src/bindings/python/nixl-meta/meson.build: use python_backend_package for the WHEEL_DEPS template substitution so the meta wheel correctly depends on nixl-rocm when built for ROCm. Closes architectural ask on PR ai-dynamo#1735. Signed-off-by: andyluo7 <andy.luo@amd.com>
|
Rebased onto latest All CI is green on the rebased head (
@tvegas1 — the rebase dismissed your approval; would you re-approve when you get a chance? |
|
@andyluo7 after speaking with the NVIDIA team on the topic of building and CI we are going to start with a plan to have separate |
Restructure contrib/Dockerfile into three stages: - common-base: shared apt deps, Abseil/gRPC/etcd-cpp source builds, python venv + uv, gtest-parallel, pybind11 - rocm-build: UCX configured --with-rocm, PyTorch from rocm wheels, NIXL meson with disable_plugins=LIBFABRIC,GDS,GDS_MT,GPUNETIO and build_nixl_ep=false; wheel name = nixl_rocm - cuda-build: DOCA install, libfabric from source, UCX --with-cuda, PyTorch CUDA wheels, optional NIXL EP build contrib/build-container.sh: add --rocm flag toggling base image to rocm/dev-ubuntu-24.04:7.2.4-complete, target rocm-build, wheel name nixl_rocm, PyTorch ROCm index, BUILD_NIXL_EP=false. contrib/build-wheel.sh: add --wheel-name and --backend-dir args, skip nvcc CUDA detection when WHEEL_NAME provided, pass new python_backend_dir/python_backend_package meson opts, extend auditwheel excludes to cover ROCm runtime libs. meson.build + meson_options.txt: add python_backend_dir and python_backend_package options that override cuda_wheel_dir / package name (used by contrib/build-wheel.sh). Compatible with the existing wheel_variant option from ai-dynamo#1642 - python_backend_dir wins when set. src/bindings/python/nixl-meta/__init__.py: add _get_torch_rocm_version, rename _load_cuda_backend -> _load_backend, dispatch to nixl_rocm before nixl_cu* when torch reports a HIP version. src/bindings/python/nixl-meta/meson.build: use python_backend_package for the WHEEL_DEPS template substitution so the meta wheel correctly depends on nixl-rocm when built for ROCm. Closes architectural ask on PR ai-dynamo#1735. Signed-off-by: andyluo7 <andy.luo@amd.com>
rocm-build-check.yml: builds the ROCm container via ./contrib/build-container.sh --rocm and smoke-tests that `import nixl` dispatches to the nixl_rocm backend and that both nixl-* and nixl_rocm-* wheels are present under /workspace/nixl/dist. Builds end-to-end in CI rather than consuming a pre-built image, so the workflow is self-bootstrapping. wheel-rocm.yml: on tagged releases, builds the same container and extracts dist/nixl_rocm-*.whl for upload as a release asset. contrib/README.rocm.md: short ROCm-only runtime guide covering the build command, the AMD Pensando AINIC dmabuf knobs (/boot mount + UCX_ROCM_COPY_DMABUF + UCX_ROCM_IPC_MIN_ZCOPY), and the nixlbench build step. No dev workflow content. Signed-off-by: andyluo7 <andy.luo@amd.com>
- Dockerfile: collapse 4-line gcc-13 comment to 2-line; drop UCX bug reference - wheel-rocm.yml: shorten header comment to one line Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
- Pin all action refs to immutable commit SHAs to prevent tag-based supply-chain attacks (checkout@v4, upload-artifact@v4, action-gh-release@v2) - Add persist-credentials: false to both checkout steps - Add concurrency block to wheel-rocm.yml to prevent parallel builds of the same ref Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
Both env vars are set by the container; add a note for users building outside Docker to export them or substitute the defaults directly. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
Cold --no-cache build on ubuntu-latest exceeds the previous 60-min limit. 90 min matches wheel-rocm.yml and our observed build time. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
…blocks Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
The ROCm '-complete' base image is ~30GB extracted and exhausts the stock ubuntu-latest runner during base-image extraction. Reclaim ~25-30GB by removing unused toolchains/SDKs and pruning docker images before the build.
A from-scratch ROCm container build does not fit the hosted-runner time budget (reaches only gRPC before the 90 min timeout). Mark the job continue-on-error so it no longer blocks PRs; it becomes a real gate once it consumes the pre-built base image and compiles only nixl.
Reclaim build-cache layers in addition to images so the from-scratch ROCm build has maximum headroom on the hosted runner.
The build itself (compile, wheel, auditwheel, pip-install) succeeds on the hosted runner; only the final docker image export OOMs/recycles the runner while compressing the multi-GB ROCm image. Move the wheel-presence and import-dispatch smoke checks into the rocm-build stage as RUN steps and build with buildx --output=cacheonly so the graph (and smokes) run without materializing the image.
| WHL_PLATFORM=${WHL_BASE}_${ARCH} | ||
| WHL_PYTHON_VERSIONS="3.12" | ||
| UCX_REF=${UCX_REF:-v1.21.x} | ||
| UCX_REF=${UCX_REF:-940c1c1d9} |
There was a problem hiding this comment.
Please remove this. The default must not be changed here, but in the caller that drives the script through --ucx-ref.
|
|
||
|
|
||
| def _load_cuda_backend() -> str: | ||
| def _get_torch_rocm_version() -> str | None: |
There was a problem hiding this comment.
This will need import latency optimizations similar to #1895
| @@ -15,26 +15,20 @@ | |||
|
|
|||
| ARG BASE_IMAGE="nvcr.io/nvidia/cuda-dl-base" | |||
| ARG BASE_IMAGE_TAG="25.10-cuda13.0-devel-ubuntu24.04" | |||
There was a problem hiding this comment.
This can technically work but I find the changes in this Dockerfile hard to understand. We also use it in many of our testing workflows so there is a risk of introducing breakage. Wondering if it might make sense to consider other options, maybe 2 Dockerfiles
WDYT @Bohatchuk @aranadive
|
Thanks @sbates130272 — happy to align with #1936 rather than run two parallel ROCm efforts. Looking at both PRs, I think they overlap on one layer but are actually complementary overall, so I'd propose splitting responsibilities instead of a straight supersede: #1936 owns the ROCm container/build environment. It's built on the merged #1900, lives in the canonical #1735 contributes the packaging + runtime layer that #1936 doesn't cover. As noted in #1936 ("NIXL is not automatically built into this image"), it provides a build environment but not shippable artifacts or runtime enablement. #1735 adds, on top of a build env:
Proposal: once #1936 lands, I'll rebase #1735 on top of it — dropping my One small thing worth carrying over from #1735: I pin UCX to the SHA Does that division work for you and @riley-dixon? If so, I'll start the rebase as soon as #1936 is merged (or against its branch now if that's easier to review). Also happy to move any of my pieces into (One small note for reviewers: the earlier |
|
Hey @andyluo7 , I have a few comments but am happy to work with you. I think one of the more significant things to be aware of is that we are planning on keeping the CUDA & ROCm build systems separate, and we do not have plans to unify them. This may change your mind how you approach I will also advise that for building & packaging NIXL that those steps occur in a CI workflow file rather than as part of the Dockerfile. UCX - Pinning it to a specific SHA or static release works for me. It will make caching a lot easier in the long term. Please let me know your thoughts. |
|
@ovidiusm as I recall the plan is for AMD to publish our own wheels. Due to concerns of AMd wheels on NVIDIA PyPi. Do I recall that correctly? |
7 participants
Summary
Restructures
contrib/Dockerfileinto a unified three-stage build (common-base → rocm-build / cuda-build) and adds ROCm-specific CI and release infrastructure. Splits the packaging surface out of #1647 per @tvegas1's request.What this adds
contrib/Dockerfilecommon-base(shared apt deps, Abseil/gRPC, Python tooling) →rocm-build(UCX--with-rocm, PyTorch ROCm wheels,nixl_rocmwheel) /cuda-build(DOCA, libfabric, UCX--with-cuda, NIXL EP,nixlCUDA wheel). UCX pinned to940c1c1d9(last clean commit before aput_offload.c-Werrorregression on v1.21.x).contrib/build-container.sh--rocmflag to select therocm-buildtarget, ROCm base image, andnixl_rocmwheel name.contrib/build-wheel.sh--wheel-name/--backend-dirargs for ROCm wheel naming.contrib/README.rocm.md/bootbind-mount +UCX_ROCM_*env vars), nixlbench launch command..github/workflows/rocm-build-check.ymlubuntu-latestvia./contrib/build-container.sh --rocm. Smoke-tests thatimport nixldispatches to thenixl_rocmbackend and that both wheel files are present. Skips draft PRs..github/workflows/wheel-rocm.ymlnixl_rocm-*.whlas a release asset.src/bindings/python/nixl-meta/nixl/__init__.py_load_cuda_backend→_load_backend; dispatch tonixl_rocmfirst whentorchreports a HIP version.meson.build/meson_options.txtpython_backend_dir/python_backend_packageoptions used bybuild-wheel.shto emit the correct wheel name for ROCm.Validated on
gbt350-odcdh2-b05-1— full container build,nixl_rocmwheel extracted, smoke test passes.smcb300-ccs-aus-j11-03— full container build,nixlwheel extracted, smoke test passes.Key design notes
--target rocm-buildor--target cuda-build.940c1c1d9is the last commit on v1.21.x before a-Werror=int-conversionregression input_offload.cthat breaks both gcc 13 and gcc 14 builds.common-basedo not propagate to child stages. All shared ARGs are re-declared with defaults in bothrocm-buildandcuda-build.rocm-build-check.ymlbuilds the container from source on each PR rather than consuming a pre-built image, so no external image publishing is required.Stacks on
wheel_variantMeson optionuse_rocmMeson option + nixlbench ROCm/HIP code pathsTest plan
./contrib/build-container.sh --rocmbuilds end-to-end on ROCm host (MI350X)import nixl; nixl._pkg.__name__ == "nixl_rocm"smoke test passesnixl_rocm-*.whlandnixl-*-none-any.whlpresent under/workspace/nixl/dist./contrib/build-container.sh(CUDA default) builds end-to-end on CUDA host (B300)nixl-*.whlpresent and imports cleanlySummary by CodeRabbit