NIXL EP wheels: Add support for multiple PyTorch versions - #1775
Merged
Merged
Conversation
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
…he code Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Co-authored-by: Raul Akhmetshin <74596089+rakhmets@users.noreply.github.com> Signed-off-by: ovidiusm <ovidium@nvidia.com>
Co-authored-by: Raul Akhmetshin <74596089+rakhmets@users.noreply.github.com> Signed-off-by: ovidiusm <ovidium@nvidia.com>
Remove comments about return values for copyfile. Signed-off-by: ovidiusm <ovidium@nvidia.com>
Removed comments regarding the return value of the copyfile function. Signed-off-by: ovidiusm <ovidium@nvidia.com>
Signed-off-by: ovidiusm <ovidium@nvidia.com>
…comments Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
Signed-off-by: Ovidiu Mara <ovidium@nvidia.com>
nv-nmailhot
pushed a commit
that referenced
this pull request
Jul 1, 2026
) ## What? Cherry-pick of #1775 ## Why? Pack multiple torch versions into nixl wheel See #1646 Signed-off-by: Ovidiu Mara <ovidium@nvidia.com> Signed-off-by: ovidiusm <ovidium@nvidia.com> Co-authored-by: ofarjon <34404710+ofirfarjun7@users.noreply.github.com> Co-authored-by: Raul Akhmetshin <74596089+rakhmets@users.noreply.github.com>
Alexey-Rivkin
added a commit
to Alexey-Rivkin/nixl
that referenced
this pull request
Jul 1, 2026
ai-dynamo#1775 made --torch-versions mandatory with --build-nixl-ep and updated Dockerfile.manylinux, but left contrib/Dockerfile calling --build-nixl-ep alone. So every EP container build has since errored at the wheel step with "--build-nixl-ep requires --torch-versions". Mirror the manylinux default (WHL_TORCH_VERSIONS=2.11,2.12,2.13) so the EP wheel step completes.
Alexey-Rivkin
added a commit
to Alexey-Rivkin/nixl
that referenced
this pull request
Jul 1, 2026
The container build (contrib/Dockerfile -> build-wheel.sh --build-nixl-ep, plus the nixlbench image) was only exercised by the nightly/manual nixl-ci-build-container job, never on PRs. That is how ai-dynamo#1775 - which made --torch-versions mandatory with --build-nixl-ep - broke every EP container build on main without any PR going red. Add a build-only leaf job (nixl-ci-build-container-pr) to the dispatcher fan-out. It runs the same build-container.sh (nixl, EP + debug) and nixlbench build the nightly job runs, but pushes nothing and touches no Artifactory - it only proves the images still build. To keep per-PR cost down it is x86_64-only and skips the heavy build when a PR touches no build-affecting files (contrib/, benchmark/, .ci/, meson*, pyproject/setup). The job still runs and reports green fast in that case, so it can be a required check without ever hanging a PR that only edits unrelated code.
Alexey-Rivkin
added a commit
to Alexey-Rivkin/nixl
that referenced
this pull request
Jul 1, 2026
The container build (contrib/Dockerfile -> build-wheel.sh --build-nixl-ep, plus the nixlbench image) was only exercised by the nightly/manual nixl-ci-build-container job, never on PRs. That is how ai-dynamo#1775 - which made --torch-versions mandatory with --build-nixl-ep - broke every EP container build on main without any PR going red. Add a build-only leaf job (nixl-ci-build-container-pr) to the dispatcher fan-out. It runs the same build-container.sh (nixl, EP + debug) and nixlbench build the nightly job runs, but pushes nothing and touches no Artifactory - it only proves the images still build. To keep per-PR cost down it is x86_64-only and skips the heavy build when a PR touches no build-affecting files (contrib/, benchmark/, .ci/, meson*, pyproject/setup). The job still runs and reports green fast in that case, so it can be a required check without ever hanging a PR that only edits unrelated code.
Alexey-Rivkin
added a commit
to Alexey-Rivkin/nixl
that referenced
this pull request
Jul 1, 2026
The container build (contrib/Dockerfile -> build-wheel.sh --build-nixl-ep, plus the nixlbench image) was only exercised by the nightly/manual nixl-ci-build-container job, never on PRs. That is how ai-dynamo#1775 - which made --torch-versions mandatory with --build-nixl-ep - broke every EP container build on main without any PR going red. Add a build-only leaf job (nixl-ci-build-container-pr) to the dispatcher fan-out. It runs the same build-container.sh (nixl, EP + debug) and nixlbench build the nightly job runs, but pushes nothing and touches no Artifactory - it only proves the images still build. To keep per-PR cost down it is x86_64-only and skips the heavy build when a PR touches no build-affecting files (contrib/, benchmark/, .ci/, meson*, pyproject/setup). The job still runs and reports green fast in that case, so it can be a required check without ever hanging a PR that only edits unrelated code.
Alexey-Rivkin
added a commit
to Alexey-Rivkin/nixl
that referenced
this pull request
Jul 1, 2026
The container build (contrib/Dockerfile -> build-wheel.sh --build-nixl-ep, plus the nixlbench image) was only exercised by the nightly/manual nixl-ci-build-container job, never on PRs. That is how ai-dynamo#1775 - which made --torch-versions mandatory with --build-nixl-ep - broke every EP container build on main without any PR going red. Add a build-only leaf job (nixl-ci-build-container-pr) to the dispatcher fan-out. It runs the same build-container.sh (nixl, EP + debug) and nixlbench build the nightly job runs, but pushes nothing and touches no Artifactory - it only proves the images still build. To keep per-PR cost down it is x86_64-only and skips the heavy build when a PR touches no build-affecting files (contrib/, benchmark/, .ci/, meson*, pyproject/setup). The job still runs and reports green fast in that case, so it can be a required check without ever hanging a PR that only edits unrelated code.
Alexey-Rivkin
added a commit
to Alexey-Rivkin/nixl
that referenced
this pull request
Jul 1, 2026
The container build (contrib/Dockerfile -> build-wheel.sh --build-nixl-ep, plus the nixlbench image) was only exercised by the nightly/manual nixl-ci-build-container job, never on PRs. That is how ai-dynamo#1775 - which made --torch-versions mandatory with --build-nixl-ep - broke every EP container build on main without any PR going red. Add a build-only leaf job (nixl-ci-build-container-pr) to the dispatcher fan-out. It runs the same build-container.sh (nixl, EP + debug) and nixlbench build the nightly job runs, but pushes nothing and touches no Artifactory - it only proves the images still build. To keep per-PR cost down it is x86_64-only and skips the heavy build when a PR touches no build-affecting files (contrib/, benchmark/, .ci/, meson*, pyproject/setup). The job still runs and reports green fast in that case, so it can be a required check without ever hanging a PR that only edits unrelated code.
ovidiusm
pushed a commit
that referenced
this pull request
Jul 1, 2026
…and set explicit EP meson overrides) (#1866) Fixes for nixl-ci-build-container flow ## What? 1. contrib/Dockerfile: add a new WHL_TORCH_VERSIONS build arg (default 2.11,2.12,2.13, mirroring Dockerfile.manylinux) and forward it to contrib/build-wheel.sh via --torch-versions when BUILD_NIXL_EP=true. 2. examples/device/ep/meson.build: extend the EP target's override_options from ['buildtype=release'] to ['buildtype=release', 'optimization=3', 'debug=false']. ## Why? After #1802 + #1775 landed, nixl-ci-build-container (debug, BUILD_NIXL_EP=true) has been failing in two distinct places: 1. Wheel step failure: #1775 added --torch-versions support to contrib/build-wheel.sh and updated Dockerfile.manylinux, but contrib/Dockerfile was still calling --build-nixl-ep alone. Every EP container build errored at the wheel step with --build-nixl-ep requires --torch-versions. 2. EP nvcc compile failure: #1802 sets nixl_ep target with override_options = ['buildtype=release'] so a global --buildtype=debug doesn't drop -G onto the EP nvcc line. In practice --device-debug still reached the EP .cu compiles on nixl-ci-build-containe compilation path. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for configuring which Torch versions are included when building the wheel, with a sensible default set. * **Bug Fixes** * Improved EP builds in debug environments by forcing release-style compilation settings, helping avoid debug flags affecting CUDA compilation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
nv-nmailhot
pushed a commit
that referenced
this pull request
Jul 1, 2026
…and set explicit EP meson overrides) (#1866) (#1872) Cherry pick of #1866 Fixes for nixl-ci-build-container flow 1. contrib/Dockerfile: add a new WHL_TORCH_VERSIONS build arg (default 2.11,2.12,2.13, mirroring Dockerfile.manylinux) and forward it to contrib/build-wheel.sh via --torch-versions when BUILD_NIXL_EP=true. 2. examples/device/ep/meson.build: extend the EP target's override_options from ['buildtype=release'] to ['buildtype=release', 'optimization=3', 'debug=false']. After #1802 + #1775 landed, nixl-ci-build-container (debug, BUILD_NIXL_EP=true) has been failing in two distinct places: 1. Wheel step failure: #1775 added --torch-versions support to contrib/build-wheel.sh and updated Dockerfile.manylinux, but contrib/Dockerfile was still calling --build-nixl-ep alone. Every EP container build errored at the wheel step with --build-nixl-ep requires --torch-versions. 2. EP nvcc compile failure: #1802 sets nixl_ep target with override_options = ['buildtype=release'] so a global --buildtype=debug doesn't drop -G onto the EP nvcc line. In practice --device-debug still reached the EP .cu compiles on nixl-ci-build-containe compilation path. Co-authored-by: lishapira <lishapira@nvidia.com>
nv-nmailhot
added a commit
that referenced
this pull request
Jul 1, 2026
Re-apply only the INFINIA wheel-build changes from #1841 onto release/1.3.1 (the native GitHub Actions pipeline was reverted separately): - contrib/Dockerfile.manylinux: build wheels on the public PyPA manylinux_2_28 base + NGC CUDA (the internal GitLab base isn't reachable from the AWS runners), pull the DDN "red" libs from a mirrored ECR image into /opt/ddn/red so meson auto-detects red_client and builds the INFINIA plugin, and source-build the liburing 2.14 / OpenLDAP 2.6 transitive deps EL8 lacks at the right soname. INFINIA_VARIANT=none yields an empty stub for external builds without the private image. - contrib/build-container.sh: --cuda-version (MAJOR.MINOR, validated for manylinux) and --no-infinia (INFINIA_VARIANT=none); disable buildx provenance/sbom attestations via env var. - README: document the manylinux wheel build and the --no-infinia path. Excludes the #1775-derived changes carried in #1841 (the contrib/Dockerfile EP/torch-versions edit and the manylinux wheel-build EP_BUILD_FLAGS reformat), since #1775 isn't on release/1.3.1 and its build-wheel.sh doesn't accept --torch-versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Alexey-Rivkin
added a commit
to Alexey-Rivkin/nixl
that referenced
this pull request
Jul 1, 2026
The container build (contrib/Dockerfile -> build-wheel.sh --build-nixl-ep, plus the nixlbench image) was only exercised by the nightly/manual nixl-ci-build-container job, never on PRs. That is how ai-dynamo#1775 - which made --torch-versions mandatory with --build-nixl-ep - broke every EP container build on main without any PR going red. Add a build-only leaf job (nixl-ci-build-container-pr) to the dispatcher fan-out. It runs the same build-container.sh (nixl, EP + debug) and nixlbench build the nightly job runs, but pushes nothing and touches no Artifactory - it only proves the images still build. To keep per-PR cost down it is x86_64-only and skips the heavy build when a PR touches no build-affecting files (contrib/, benchmark/, .ci/, meson*, pyproject/setup). The job still runs and reports green fast in that case, so it can be a required check without ever hanging a PR that only edits unrelated code.
Alexey-Rivkin
added a commit
to Alexey-Rivkin/nixl
that referenced
this pull request
Jul 1, 2026
The container build (contrib/Dockerfile -> build-wheel.sh --build-nixl-ep, plus the nixlbench image) was only exercised by the nightly/manual nixl-ci-build-container job, never on PRs. That is how ai-dynamo#1775 - which made --torch-versions mandatory with --build-nixl-ep - broke every EP container build on main without any PR going red. Add a build-only leaf job (nixl-ci-build-container-pr) to the dispatcher fan-out. It runs the same build-container.sh (nixl, EP + debug) and nixlbench build the nightly job runs, but pushes nothing and touches no Artifactory - it only proves the images still build. To keep per-PR cost down it is x86_64-only and skips the heavy build when a PR touches no build-affecting files (contrib/, benchmark/, .ci/, meson*, pyproject/setup). The job still runs and reports green fast in that case, so it can be a required check without ever hanging a PR that only edits unrelated code.
Alexey-Rivkin
added a commit
to Alexey-Rivkin/nixl
that referenced
this pull request
Jul 2, 2026
The container build (contrib/Dockerfile -> build-wheel.sh --build-nixl-ep, plus the nixlbench image) was only exercised by the nightly/manual nixl-ci-build-container job, never on PRs. That is how ai-dynamo#1775 - which made --torch-versions mandatory with --build-nixl-ep - broke every EP container build on main without any PR going red. Add a build-only leaf job (nixl-ci-build-container-pr) to the dispatcher fan-out. It runs the same build-container.sh (nixl, EP + debug) and nixlbench build the nightly job runs, but pushes nothing and touches no Artifactory - it only proves the images still build. To keep per-PR cost down it is x86_64-only and skips the heavy build when a PR touches no build-affecting files (contrib/, benchmark/, .ci/, meson*, pyproject/setup). The job still runs and reports green fast in that case, so it can be a required check without ever hanging a PR that only edits unrelated code.
Alexey-Rivkin
added a commit
to Alexey-Rivkin/nixl
that referenced
this pull request
Jul 2, 2026
The container build (contrib/Dockerfile -> build-wheel.sh --build-nixl-ep, plus the nixlbench image) was only exercised by the nightly/manual nixl-ci-build-container job, never on PRs. That is how ai-dynamo#1775 - which made --torch-versions mandatory with --build-nixl-ep - broke every EP container build on main without any PR going red. Add a build-only leaf job (nixl-ci-build-container-pr) to the dispatcher fan-out. It runs the same build-container.sh (nixl, EP + debug) and nixlbench build the nightly job runs, but pushes nothing and touches no Artifactory - it only proves the images still build. To keep per-PR cost down it is x86_64-only and skips the heavy build when a PR touches no build-affecting files (contrib/, benchmark/, .ci/, meson*, pyproject/setup). The job still runs and reports green fast in that case, so it can be a required check without ever hanging a PR that only edits unrelated code.
Alexey-Rivkin
added a commit
to Alexey-Rivkin/nixl
that referenced
this pull request
Jul 2, 2026
The container build (contrib/Dockerfile -> build-wheel.sh --build-nixl-ep, plus the nixlbench image) was only exercised by the nightly/manual nixl-ci-build-container job, never on PRs. That is how ai-dynamo#1775 - which made --torch-versions mandatory with --build-nixl-ep - broke every EP container build on main without any PR going red. Add a build-only leaf job (nixl-ci-build-container-pr) to the dispatcher fan-out. It runs the same build-container.sh (nixl, EP + debug) and nixlbench build the nightly job runs, but pushes nothing and touches no Artifactory - it only proves the images still build. To keep per-PR cost down it is x86_64-only and skips the heavy build when a PR touches no build-affecting files (contrib/, benchmark/, .ci/, meson*, pyproject/setup). The job still runs and reports green fast in that case, so it can be a required check without ever hanging a PR that only edits unrelated code.
Alexey-Rivkin
added a commit
to Alexey-Rivkin/nixl
that referenced
this pull request
Jul 2, 2026
The container build (contrib/Dockerfile -> build-wheel.sh --build-nixl-ep, plus the nixlbench image) was only exercised by the nightly/manual nixl-ci-build-container job, never on PRs. That is how ai-dynamo#1775 - which made --torch-versions mandatory with --build-nixl-ep - broke every EP container build on main without any PR going red. Add a build-only leaf job (nixl-ci-build-container-pr) to the dispatcher fan-out. It runs the same build-container.sh (nixl, EP + debug) and nixlbench build the nightly job runs, but pushes nothing and touches no Artifactory - it only proves the images still build. To keep per-PR cost down it is x86_64-only and skips the heavy build when a PR touches no build-affecting files (contrib/, benchmark/, .ci/, meson*, pyproject/setup). The job still runs and reports green fast in that case, so it can be a required check without ever hanging a PR that only edits unrelated code.
copy-pr-bot Bot
pushed a commit
that referenced
this pull request
Jul 2, 2026
The container build (contrib/Dockerfile -> build-wheel.sh --build-nixl-ep, plus the nixlbench image) was only exercised by the nightly/manual nixl-ci-build-container job, never on PRs. That is how #1775 - which made --torch-versions mandatory with --build-nixl-ep - broke every EP container build on main without any PR going red. Add a build-only leaf job (nixl-ci-build-container-pr) to the dispatcher fan-out. It runs the same build-container.sh (nixl, EP + debug) and nixlbench build the nightly job runs, but pushes nothing and touches no Artifactory - it only proves the images still build. To keep per-PR cost down it is x86_64-only and skips the heavy build when a PR touches no build-affecting files (contrib/, benchmark/, .ci/, meson*, pyproject/setup). The job still runs and reports green fast in that case, so it can be a required check without ever hanging a PR that only edits unrelated code.
copy-pr-bot Bot
pushed a commit
that referenced
this pull request
Jul 2, 2026
The container build (contrib/Dockerfile -> build-wheel.sh --build-nixl-ep, plus the nixlbench image) was only exercised by the nightly/manual nixl-ci-build-container job, never on PRs. That is how #1775 - which made --torch-versions mandatory with --build-nixl-ep - broke every EP container build on main without any PR going red. Add a build-only leaf job (nixl-ci-build-container-pr) to the dispatcher fan-out. It runs the same build-container.sh (nixl, EP + debug) and nixlbench build the nightly job runs, but pushes nothing and touches no Artifactory - it only proves the images still build. To keep per-PR cost down it is x86_64-only and skips the heavy build when a PR touches no build-affecting files (contrib/, benchmark/, .ci/, meson*, pyproject/setup). The job still runs and reports green fast in that case, so it can be a required check without ever hanging a PR that only edits unrelated code.
nv-nmailhot
added a commit
that referenced
this pull request
Jul 9, 2026
Re-apply only the INFINIA wheel-build changes from #1841 onto release/1.3.1 (the native GitHub Actions pipeline was reverted separately): - contrib/Dockerfile.manylinux: build wheels on the public PyPA manylinux_2_28 base + NGC CUDA (the internal GitLab base isn't reachable from the AWS runners), pull the DDN "red" libs from a mirrored ECR image into /opt/ddn/red so meson auto-detects red_client and builds the INFINIA plugin, and source-build the liburing 2.14 / OpenLDAP 2.6 transitive deps EL8 lacks at the right soname. INFINIA_VARIANT=none yields an empty stub for external builds without the private image. - contrib/build-container.sh: --cuda-version (MAJOR.MINOR, validated for manylinux) and --no-infinia (INFINIA_VARIANT=none); disable buildx provenance/sbom attestations via env var. - README: document the manylinux wheel build and the --no-infinia path. Excludes the #1775-derived changes carried in #1841 (the contrib/Dockerfile EP/torch-versions edit and the manylinux wheel-build EP_BUILD_FLAGS reformat), since #1775 isn't on release/1.3.1 and its build-wheel.sh doesn't accept --torch-versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Pack multiple torch versions into nixl wheel
Why?
See #1646
Summary by CodeRabbit
New Features
Chores