Repository navigation
[AUTOGENERATED] [release/2.15] Skip test_ddp_optimizer_cudagraph on ROCm - #3849
Draft
rocm-repo-management-api-6[bot] wants to merge 9 commits into
Draft
rocm-repo-management-api-6[bot] wants to merge 9 commits into
rocm-repo-management-api-6[bot] wants to merge 9 commits into
Conversation
## Summary Bringup step for `release/2.15`: add `related_commits` pinning the ROCm dependency builds. - apex: new `ROCm/apex` `release/1.15.0` (cut from master, version 1.15.0) - torchvision: upstream `pytorch/vision` `release/0.29` - torchaudio: new `ROCm/audio` `release/2.11.1`: upstream `pytorch/audio` `release/2.11.1` plus the ROCm build support from `release/2.11.0.3` and the ROCm version check (ROCm/audio#21) Same layout as the `release/2.14` `related_commits` (#3573). Tracking: AIPYTORCH-1326 --------- Signed-off-by: pablo-garay <pgarayfe@amd.com>
## Summary Pin Python dependencies for the `release/2.15` bringup so CI and builds are reproducible, and make everything resolve cleanly on Python **3.10 / 3.11 / 3.12 / 3.13 / 3.14 / 3.15**. Mirrors the 2.14 bringup (#3574), plus the 3.15 handling that is new in this release. Three commits, one per file: - `requirements.txt`: pin all direct dev/install deps (with `python_version` splits for `filelock`, `hypothesis` and `networkx`). - `requirements-build.txt`: pin all build-system deps (with a `python_version` split for `numpy`). - `.ci/docker/requirements-ci.txt`: pin the previously-loose entries (`click`, `psutil`, `pytest-rerunfailures`, `unittest-xml-reporting`, `redis`, `pyzstd` split, `six`, `cuda-bindings`, `cupti-python`, `tqdm`, `uv`, `librosa`) and add `python_version` conditionals for 3.14/3.15. ## Per-version splits (dev/build) - **numpy** (build): `2.2.6` (<3.11), `2.4.6` (==3.11), `2.5.4` (>=3.12). - **networkx** (dev): `3.4.2` (<3.11), `3.6.1` (==3.11), `3.7` (>=3.12). - **filelock** (dev): `4.1.0` (<3.11), `4.2.0` (>=3.11). - **hypothesis** (dev): `6.168.5` (<3.11), `6.169.2` (>=3.11). Every other dev/build dep resolved identically across 3.10-3.15, so those are flat-pinned. ## 3.14 / 3.15 conditionals in requirements-ci.txt Restored from 2.14 (lost in the 2.15 branch): - **pywavelets**: `1.9.0` for 3.14 (1.7.0 has no cp314 wheel). - **fbscribelogger**: guarded to `< 3.14` (its `thriftpy2` dep has no cp314/cp315 wheel and would source-build in the compiler-less test image). New for 3.15 (earliest release with a cp315 wheel): - **numpy** `2.5.2` (the existing `2.4.6` pin for 3.15 has no cp315 wheel), **pandas** `3.0.6`, **optree** `0.20.0`, **pywavelets** `1.10.0`, **aiohttp** `3.14.4`. - **scikit-image**: skipped on 3.15 (no release ships a cp315 wheel yet). ## How versions were resolved Per the documented process: installed each requirements file in a fresh venv inside an `ubuntu:24.04` container with no C compiler (deadsnakes for the non-native Python versions) and pinned the `pip freeze` output. All three files install on **3.10-3.15** with `pip check` clean and no new compiler-dependent source builds (only the pre-existing pure-Python `pwlf` and `coremltools` sdists, plus `pyyaml` on 3.15, which builds without its optional C extension). ## Test plan TheRock CI `multi_arch_build_portable_linux_pytorch_wheels.yml` (gfx94X-dcgpu, ROCm 10.2.0a20261013, `pytorch_git_ref` = `pablo-garay/release-2.15-integration`, which is this PR plus the related_commits from #3816): - [x] Python 3.11: build passed, [run 38705512288](https://github.com/ROCm/TheRock/actions/runs/38705512288) - [x] Python 3.12: build passed, [run 38705525582](https://github.com/ROCm/TheRock/actions/runs/38705525582) - [x] Python 3.13: build passed, [run 38705535781](https://github.com/ROCm/TheRock/actions/runs/38705535781) - [x] Python 3.14: build passed, [run 38705545732](https://github.com/ROCm/TheRock/actions/runs/38705545732) - Python 3.10: n/a. Triton 3.9 requires Python >=3.11, and TheRock dropped 3.10 release builds in ROCm/TheRock#8396 ([run 38705500960](https://github.com/ROCm/TheRock/actions/runs/38705500960) fails installing the triton wheel). - Python 3.15: n/a until ROCm/TheRock#8394 adds 3.15 to the build image ([run 38705556076](https://github.com/ROCm/TheRock/actions/runs/38705556076) fails at setup: `Could not find python: /opt/python/cp315-cp315`). The 3.15 pins were validated by the container install above. --------- Signed-off-by: pablo-garay <pgarayfe@amd.com>
…#3561) (#3832) ## Summary Cherry-pick the hipify `subst`-drive fix into `release/2.15`. It's the same change as `release/2.13` (#3561) and `release/2.14` (f11950f); upstream nightly has it via pytorch#194641. Windows CI builds under a `subst` drive (`B:` aliasing `C:\scratch-b`), so `CUDAExtension` compares `B:\...` with `C:\...` and hipify silently skips torchaudio's GPU sources. Resolving both sides with `os.path.realpath` fixes the match. No effect on Linux. ## Test plan - Failing without this: Windows py3.12 build of `release/2.15` ([run 38716492477](https://github.com/ROCm/TheRock/actions/runs/38716492477)) fails in torchaudio `rnnt/gpu/compute.cu` and `forced_align/gpu/compute.cu` with `'cuda_runtime_api.h' file not found`. - The diff is identical to the 2.14 fix, which builds torchaudio on Windows. - Re-running the Windows build on `pablo-garay/release-2.15-integration` with this commit for ROCm/TheRock#9031. Signed-off-by: pablo-garay <pgarayfe@amd.com>
Cherry-pick of #3808 Co-authored-by: albmalamd <albert.malewski@amd.com>
…dSmiImport (pytorch#200967) (#3846) ## Linked issue or supporting maintainer Supported by @jeffdaily Related: ROCm/TheRock#8888 (the CI lane where the test fails), pytorch#200650 (the first fix for this test) and pytorch#197482, which added the test. ## BC-breaking? No. ## Summary (human written only) Still getting failures after pytorch#200650 landed, this should fix the next failure that was hidden by the first. ## Agent details (if applicable) `test_amdsmi_bare_unversioned_path_uses_loader_first` still fails in the TheRock nightly wheel tests after pytorch#200650, now with exit code 1 during `import torch` in the child: `FileNotFoundError: Could not find rocm library 'amd_comgr' at path '/tmp/.../_rocm_sdk_core/lib'`. `_run_amdsmi_import` writes a stub `_rocm_sdk_core` package into its temp dir, which is the child's cwd and first on `PYTHONPATH`. TheRock wheels preload ROCm libraries from the real `_rocm_sdk_core` during `import torch`, so the stub shadows it and the preload fails. This creates the stub only when no real `_rocm_sdk_core` is importable. When one is installed, its versioned `libamd_smi.so` is the SDK fallback, and the test's assertion (a bare `libamd_smi.so` request uses the loader path first) is unchanged. Upstream CI has no `_rocm_sdk_core`, so nothing changes there. Validation on gfx942, done with Claude, using the TheRock nightly `torch 2.16.0a0+rocm10.2.0a20261014` (git ca883d5) with the test file at that commit. Before: `test_amdsmi_bare_unversioned_path_uses_loader_first` fails with the `amd_comgr` error above and the other four TestAmdSmiImport tests pass. After: all five pass in three runs; the child imports the real `_rocm_sdk_core` (which ships `libamd_smi.so.27`) and the bare request resolves through the loader path. The no-SDK path was not rerun, since the guard is a no-op there. lintrunner clean on main. Pull Request resolved: pytorch#200967 Approved by: https://github.com/jeffdaily Co-authored-by: Jeff Daily <jeff.daily@amd.com>
…ort (pytorch#200650) (#3845) ## Linked issue or supporting maintainer Supported by @jeffdaily Related: ROCm/TheRock#8888 (the CI lane where the test fails) and pytorch#197482, which added the test. ## BC-breaking? No. ## Summary (human written only) `test_amdsmi_bare_unversioned_path_uses_loader_first` (added in pytorch#197482) fails with `Expected 0 but got 127` when the Python running the tests needs `LD_LIBRARY_PATH` to find libpython, as with an interpreter installed by actions/setup-python. `_run_amdsmi_import` replaced `LD_LIBRARY_PATH` with its temp loader directory before spawning `sys.executable`, so the child died in the dynamic loader before running any Python. This prepends the loader directory to the inherited value instead. It is still first on `LD_LIBRARY_PATH`, and the test's assertion (the bare name loads, the SDK fallback is not used) is unchanged. ## Agent details (if applicable) Validation on gfx942 at 6c3af02, done with Claude; the helper and the amdsmi hook are unchanged between that commit and this branch's base. With the stock interpreter all five TestAmdSmiImport tests pass before and after. Under a locally built launcher that reaches libpython only through `LD_LIBRARY_PATH` (it exits 127 without it), `test_amdsmi_bare_unversioned_path_uses_loader_first` fails with `Expected 0 but got 127` before the change and all five pass after. The patched test also passes with a non-loadable `libamd_smi.so` in an inherited `LD_LIBRARY_PATH` directory. This reproduces the failure mode, not the TheRock CI lane itself. lintrunner clean. Pull Request resolved: pytorch#200650 Approved by: https://github.com/jeffdaily Co-authored-by: Jeff Daily <jeff.daily@amd.com>
… libomp140 (#3836) Cherry-picks the two upstream Windows ROCm wheel DLL install fixes onto `release/2.15` (scikit-build-core backend): `aotriton_v2.dll` + `liblzma.dll` (pytorch#200931) and `libomp140.x86_64.dll` (pytorch#191629). Without them, once TheRock's temporary force-includes are removed the DLLs drop out of the wheel and `import torch` fails with WinError 126. The aotriton/liblzma install uses `OPTIONAL` so the existence check runs at install time, because aotriton's ExternalProject only generates those DLLs during the build. ### Validation Branch merged with current `release/2.15` (now includes `related_commits` from #3816). Windows ROCm wheel build (gfx1100, py3.12) on this head (`a119523c5bc`) with ROCm/TheRock#9360, which passes no force-include flags, so every DLL can only ship via CMake install: https://github.com/ROCm/TheRock/actions/runs/39287495719. Build and the `import torch` sanity check passed. Related PR: - pytorch#191629: install `libomp140.x86_64.dll` via CMake - pytorch#200931: install `aotriton_v2.dll` + `liblzma.dll` via CMake - ROCm/TheRock#9055: drop the `_rocm_init.py` wheel force-include - ROCm/TheRock#9058: drop the `aotriton_v2.dll` + `liblzma.dll` wheel force-includes - ROCm/TheRock#9360: drop the `libomp140.x86_64.dll` wheel force-include - #3835: release/2.14 cherry-pick of pytorch#191629 + pytorch#200931 - #3836: release/2.15 cherry-pick of pytorch#191629 + pytorch#200931 (this PR) - Issue: ROCm/TheRock#6965 Tracked in AIPYTORCH-1516.
jithunnair-amd
force-pushed
the
release/2.15
branch
from
November 4, 2026 16:58
dde42be to
61dff56
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #3708
Need to resolve conflicts