Repository navigation
ci(pytorch): add Python 3.15 release builds - #8394
Conversation
Track per-ref platform support so unsupported combinations are excluded.
Provide cp315 build, dependency mirror, prerelease setup, and promotion support.
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🎉 All checks passed! This PR is ready for review. |
[release/2.11] Point torchaudio at the legacy-setuptools fix for wheel builds release/2.11 pins torchaudio at pytorch/audio 34c52a67, which passes pathlib.Path objects as extension `sources` and fails on the builder's setuptools with "AssertionError: 'sources' must be a list of strings". Repoint at the str() fix (ethanwee1/audio 143129b5) so the ROCm 7.2.4 wheel build reaches torchaudio green. Same pin used by the last green Jenkins wheel build (#28).
Fold 3.15 next to 3.14 in allow/reject fixtures and existing matrix cases.
Pre-commit check failed⛔ pre-commit failed Please run locally:
This repo uses |
Keep the existing-list coverage; black failed CI on the set comprehension line.
Match how 3.14 was added: one RELEASE_PYTHON_VERSIONS list, no per-ref map.
Stop wrapping existing cases in extra loops; 3.15 belongs on the shared version list.
Exercise the shared release default so Python version additions cannot bypass matrix coverage.
| # images, with custom additional packages installed. We switch to | ||
| # new upstream versions as needed. | ||
| FROM quay.io/pypa/manylinux_2_28_x86_64@sha256:d632b5e68ab39e59e128dcf0e59e438b26f122d7f2d45f3eea69ffd2877ab017 | ||
| FROM quay.io/pypa/manylinux_2_28_x86_64@sha256:531d7aa844bbb0c131d4ab011d3db741c4abc8d498cd5ccc86121046f62303b4 |
There was a problem hiding this comment.
https://quay.io/repository/pypa/manylinux_2_28_x86_64?tab=tags
looking here, the new version selected has 4 cirtical securtiy issues and is 1 month old. can we swithc to something newer? if we need to have those changes for py3.15 in the first place.
There was a problem hiding this comment.
patches are strongly discouraged. considering how long this pr is open, cant we just wait until this makes it into rocm-systems? (sad to see that the pr in rocm-systems is still a draft :( )
## Problem The merged TheRock manylinux image does not contain CPython 3.15, so TheRock #8394 cannot configure or build its cp315 Linux artifacts from merged code. ## Changes - Move to the current `manylinux_2_28_x86_64` base digest containing CPython 3.15. - Build shared Python 3.15 for embedded consumers such as ROCgdb. - Install `glibc-gconv-extra` so the refreshed image retains the iconv conversion coverage needed by ROCgdb tests. ## Sequencing - Merge this image change and wait for the main-branch image publication. - Then update the published image digest across all consumers in [TheRock #8394](#8394). - [TheRock #8460](#8460) and [#8565](#8565) are the earlier image update and revert; this replaces that reverted image with the current base and the missing shared-runtime setup. ## Validation - `bash -n dockerfiles/install_shared_pythons.sh` passed. - `git diff --check` passed. - Local Docker image build passed from commit `74266a11`. - The built image contains regular CPython 3.15.2 and shared CPython 3.15.0. - `glibc-gconv-extra` is installed and `iconv -l` reports 1,180 entries. Tracked in AIPYTORCH-1241.
| - name: Set up .NET | ||
| # Hosted Windows images can omit dotnet; WiX v4 is a .NET global tool. | ||
| uses: actions/setup-dotnet@67a3573c9a986a3f9c594539f4ab511d57bb3ce9 # v4.3.1 | ||
| with: | ||
| dotnet-version: '8.x' |
There was a problem hiding this comment.
This is unrelated to Python 3.15 builds, please move to another PR.
## Summary Pin Python dependencies for the `release/2.15` bringup so CI and builds are reproducible, and make everything resolve cleanly on Python **3.10 / 3.11 / 3.12 / 3.13 / 3.14 / 3.15**. Mirrors the 2.14 bringup (#3574), plus the 3.15 handling that is new in this release. Three commits, one per file: - `requirements.txt`: pin all direct dev/install deps (with `python_version` splits for `filelock`, `hypothesis` and `networkx`). - `requirements-build.txt`: pin all build-system deps (with a `python_version` split for `numpy`). - `.ci/docker/requirements-ci.txt`: pin the previously-loose entries (`click`, `psutil`, `pytest-rerunfailures`, `unittest-xml-reporting`, `redis`, `pyzstd` split, `six`, `cuda-bindings`, `cupti-python`, `tqdm`, `uv`, `librosa`) and add `python_version` conditionals for 3.14/3.15. ## Per-version splits (dev/build) - **numpy** (build): `2.2.6` (<3.11), `2.4.6` (==3.11), `2.5.4` (>=3.12). - **networkx** (dev): `3.4.2` (<3.11), `3.6.1` (==3.11), `3.7` (>=3.12). - **filelock** (dev): `4.1.0` (<3.11), `4.2.0` (>=3.11). - **hypothesis** (dev): `6.168.5` (<3.11), `6.169.2` (>=3.11). Every other dev/build dep resolved identically across 3.10-3.15, so those are flat-pinned. ## 3.14 / 3.15 conditionals in requirements-ci.txt Restored from 2.14 (lost in the 2.15 branch): - **pywavelets**: `1.9.0` for 3.14 (1.7.0 has no cp314 wheel). - **fbscribelogger**: guarded to `< 3.14` (its `thriftpy2` dep has no cp314/cp315 wheel and would source-build in the compiler-less test image). New for 3.15 (earliest release with a cp315 wheel): - **numpy** `2.5.2` (the existing `2.4.6` pin for 3.15 has no cp315 wheel), **pandas** `3.0.6`, **optree** `0.20.0`, **pywavelets** `1.10.0`, **aiohttp** `3.14.4`. - **scikit-image**: skipped on 3.15 (no release ships a cp315 wheel yet). ## How versions were resolved Per the documented process: installed each requirements file in a fresh venv inside an `ubuntu:24.04` container with no C compiler (deadsnakes for the non-native Python versions) and pinned the `pip freeze` output. All three files install on **3.10-3.15** with `pip check` clean and no new compiler-dependent source builds (only the pre-existing pure-Python `pwlf` and `coremltools` sdists, plus `pyyaml` on 3.15, which builds without its optional C extension). ## Test plan TheRock CI `multi_arch_build_portable_linux_pytorch_wheels.yml` (gfx94X-dcgpu, ROCm 10.2.0a20261013, `pytorch_git_ref` = `pablo-garay/release-2.15-integration`, which is this PR plus the related_commits from #3816): - [x] Python 3.11: build passed, [run 38705512288](https://github.com/ROCm/TheRock/actions/runs/38705512288) - [x] Python 3.12: build passed, [run 38705525582](https://github.com/ROCm/TheRock/actions/runs/38705525582) - [x] Python 3.13: build passed, [run 38705535781](https://github.com/ROCm/TheRock/actions/runs/38705535781) - [x] Python 3.14: build passed, [run 38705545732](https://github.com/ROCm/TheRock/actions/runs/38705545732) - Python 3.10: n/a. Triton 3.9 requires Python >=3.11, and TheRock dropped 3.10 release builds in ROCm/TheRock#8396 ([run 38705500960](https://github.com/ROCm/TheRock/actions/runs/38705500960) fails installing the triton wheel). - Python 3.15: n/a until ROCm/TheRock#8394 adds 3.15 to the build image ([run 38705556076](https://github.com/ROCm/TheRock/actions/runs/38705556076) fails at setup: `Could not find python: /opt/python/cp315-cp315`). The 3.15 pins were validated by the container install above. --------- Signed-off-by: pablo-garay <pgarayfe@amd.com>
Can you please revise the PR description? The PR description will be the commit message when merged and text like this is not relevant there. These are also self-evident from the pull request itself and don't need to be mentioned here. (In general I'd recommend writing PR descriptions yourself if patterns like this continue, many coding agents are still bad at writing commit messages...) |
ScottTodd
left a comment
ScottTodd
left a comment
There was a problem hiding this comment.
Please link workflow run logs showing all Python 3.15 builds succeeding. The PR description has a whole lot in the "Validation" section but not what I'm actually interested in reviewing.
For example, manual runs of these workflows with the python version set to 3.15:
- https://github.com/ROCm/TheRock/actions/workflows/multi_arch_build_portable_linux_pytorch_wheels.yml
- https://github.com/ROCm/TheRock/actions/workflows/multi_arch_build_windows_pytorch_wheels.yml
(In general please be mindful of https://github.com/ROCm/TheRock/blob/main/CONTRIBUTING.md#ai-tool-use-policy, the text in the PR description is verbose and puts substantial extra burden on reviewers to parse through)
| - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 | ||
| with: | ||
| python-version: ${{ inputs.python_version }} | ||
| allow-prereleases: true |
There was a problem hiding this comment.
We shouldn't need allow-prereleases: true now, right? Or at least that could be a separate PR at this point to be more forward looking.
| "additional_requirements_files": [ | ||
| _get_artifact_path("share/rocprofiler-sdk/tests/requirements.txt"), | ||
| _get_script_path("requirements-test-ninja.txt"), | ||
| ], |
There was a problem hiding this comment.
There are unrelated files showing up in the diff here. Can you keep your branch up to date instead of sprinkling unrelated commits in the history? This makes review difficult.
|
The
Step-by-Step SolutionStep 1: Check and Downshift Your Python VersionCreate a new conda or virtual environment using a supported stable Python version (Python 3.12 or 3.13 are widely supported across modern PyTorch and ROCm builds): conda create -n mempalace python=3.12 -y
conda activate mempalace
Step 2: Install PyTorch with Device-All or General TargetingOnce inside a supported Python environment, reinstall the ROCm base libraries and try pointing to the general device package or adjusting the wheel selector: python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ "rocm[libraries,device-gfx1103]==10.1.0"
Then install PyTorch using the python -m pip install --index-url https://stable.repo.amd.com/rocm/whl-next/ \
"torch[device-all]==2.14.0+rocm10.1.0" \
"torchvision==0.29.0a0+rocm10.1.0" \
"torchaudio==2.11.0.3+rocm10.1.0"
Verification: Run |
|
@ethanwee1 are now all other dependencies merged? or is this pr still waiting for changes in the pytorch ecosystem? |
Summary
Enable Python 3.15 PyTorch release builds in TheRock, including regular/shared CPython paths and all portable manylinux consumers. Unsupported PyTorch
release/2.12combinations remain excluded.Risk Assessment
Level 3 (
build/ci): release matrices, reusable workflows, dependency mirroring, and portable wheel packaging change across Linux and Windows; ROCm runtime code is unchanged.Related
sha256:f1f9dd9b20404c239f01a02ffec61d6c2e80abcafe7fdac854f06169828e53ceDevice / Architecture Coverage
Testing Checklist
filelock-4.0.8in the dev index.Flags / Guardrails
No new runtime feature flag. Python 3.15 is included only in release matrices; CI defaults remain unchanged.
Technical Changes
Closes #5400
Tracked in AIPYTORCH-1241.