pytorch-cpu v2.9.1#448
Conversation
Update for 2.9.0, and rebase patches. Signed-off-by: Michał Górny <mgorny@quansight.com>
The tested assertion started failing on AArch64 cross builds, make it print the actual value to aid debugging (now and in the future). Signed-off-by: Michał Górny <mgorny@quansight.com>
Signed-off-by: Michał Górny <mgorny@quansight.com>
Our old patch is no longer sufficient for cross-compilation, the new code checks for PYTORCH_BLAS_USE_CBLAS_DOT envvar though. Signed-off-by: Michał Górny <mgorny@quansight.com>
It looks like we now need to explicitly pass CUDA target include and `libcuda.so` stub directories while building inductor. Introduce a substitution `@CUDA_TARGET@` value in the patch, and replace it with appropriate target in `build.sh`. This leaves the unsubstituted `@CUDA_TARGET@` path on Windows, but that shouldn't do any harm -- the path will simply not exist. Signed-off-by: Michał Górny <mgorny@quansight.com>
Inductor now started requiring fmt headers, and since we are moving includes from site-packages to the top-level include directory, having PyTorch install fmt headers there would conflict with system fmt install. However, PyTorch nowadays uses fmt 12, so let's just use the system library instead. This uses a WIP patch submitted upstream along with a quick hack to make kineto build. Signed-off-by: Michał Górny <mgorny@quansight.com>
Signed-off-by: Michał Górny <mgorny@quansight.com>
Upstream now does not build fbgemm on Windows by default, and if we try to force building it, it just fails. Signed-off-by: Michał Górny <mgorny@quansight.com>
This is required by fmt headers, and upstream is doing it explicitly in PyTorch's `CMakeLists.txt`. I suppose all dependent projects will have to follow suit. Signed-off-by: Michał Górny <mgorny@quansight.com>
Signed-off-by: Michał Górny <mgorny@quansight.com>
Signed-off-by: Michał Górny <mgorny@quansight.com>
Signed-off-by: Michał Górny <mgorny@quansight.com>
Signed-off-by: Michał Górny <mgorny@quansight.com>
…and conda-forge-pinning 2025.11.05.12.33.05 Other tools: - conda-build 25.9.0 - rattler-build 0.49.0 - rattler-build-conda-compat 1.4.9
Signed-off-by: Michał Górny <mgorny@quansight.com>
|
Hi! This is the friendly automated conda-forge-linting service. I just wanted to let you know that I linted all conda-recipes in your PR ( I do have some suggestions for making it better though... For recipe/meta.yaml:
This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/19647270882. Examine the logs at this URL for more detail. |
Fixes conda-forge#429 Signed-off-by: Michał Górny <mgorny@quansight.com>
…5.11.13.12.38.23 Other tools: - conda-build 25.9.0 - rattler-build 0.49.0 - rattler-build-conda-compat 1.4.9
Signed-off-by: Michał Górny <mgorny@quansight.com>
…5.11.13.12.38.23 Other tools: - conda-build 25.9.0 - rattler-build 0.49.0 - rattler-build-conda-compat 1.4.9
|
Good news; #446 has unblocked us on osx 🥳 🚀 I've integrated those changes here now. 93301f0 should obviously move to the ci-setup, but that can happen independently. I want to see if everything works when merging the osx-on-cirun builds; the uploads obviously are not tested in a PR. |
7b5fa3a to
ae878a3
Compare
|
Sigh, don't know what's up with the linux-aarch builds here That library is definitely present at CC @conda-forge/ctng-compilers, in case this could be related to recent work on that feedstock. |
|
Maybe it's just some flake? It would really be weird if it only noticed the missing plugin this late in the build. |
|
Unfortunately it reproduces in the linux-aarch CPU job. Clearly the actual cross-compilation part works during the build. We get the failure during the test-phase when we're emulating As such, flags like |
|
The last passing build two weeks ago had That's the only difference between then and now from a compiler POV is conda-forge/ctng-compiler-activation-feedstock#168 AFAICT, but that PR only affects the bootstrap compilers, which we're not using here (also the newest run still has a 2.34 sysroot). There's also conda-forge/binutils-feedstock#101, but that looks benign. In total, the environment differences are as follows: diff of test environment from before (passing) to after (failing)@@ -3,12 +3,12 @@ The following NEW packages will be INSTALLED:
_openmp_mutex: 4.5-6_kmp_llvm conda-forge
arm-variant: 1.2.0-sbsa conda-forge
attr: 2.5.1-h4e544f5_1 conda-forge
- binutils_impl_linux-aarch64: 2.45-ha36da51_0 conda-forge
- binutils_linux-aarch64: 2.45-hf1166c9_0 conda-forge
+ binutils_impl_linux-aarch64: 2.45-default_1234567_3 conda-forge
+ binutils_linux-aarch64: 2.45-default_1234567_3 conda-forge
bzip2: 1.0.8-h4777abc_8 conda-forge
c-ares: 1.34.5-h86ecc28_0 conda-forge
ca-certificates: 2025.11.12-hbd8a1cb_0 conda-forge
- cmake: 4.1.2-hc9d863e_0 conda-forge
+ cmake: 4.2.0-hc9d863e_0 conda-forge
cuda-cccl_linux-aarch64: 12.9.27-h579c4fd_0 conda-forge
cuda-crt-dev_linux-aarch64: 12.9.86-h579c4fd_2 conda-forge
cuda-crt-tools: 12.9.86-h579c4fd_2 conda-forge
@@ -34,17 +34,17 @@ The following NEW packages will be INSTALLED:
cudnn: 9.10.2.21-h32c1c63_0 conda-forge
fmt: 12.0.0-h416241a_0 conda-forge
gcc_impl_linux-aarch64: 13.4.0-h69010b7_7 conda-forge
- gcc_linux-aarch64: 13.4.0-hbf9eca1_13 conda-forge
+ gcc_linux-aarch64: 13.4.0-hbf9eca1_14 conda-forge
gxx_impl_linux-aarch64: 13.4.0-hf6d83cf_7 conda-forge
- gxx_linux-aarch64: 13.4.0-h2dde472_13 conda-forge
+ gxx_linux-aarch64: 13.4.0-he64a8ad_14 conda-forge
kernel-headers_linux-aarch64: 5.14.0-h05a177a_2 conda-forge
keyutils: 1.6.3-h86ecc28_0 conda-forge
krb5: 1.21.3-h50a48e9_0 conda-forge
- ld_impl_linux-aarch64: 2.45-hd32f0e1_0 conda-forge
+ ld_impl_linux-aarch64: 2.45-default_1234567_3 conda-forge
libabseil: 20250512.1-cxx17_h201e9ed_0 conda-forge
- libblas: 3.9.0-38_haddc8a3_openblas conda-forge
+ libblas: 3.11.0-2_haddc8a3_openblas conda-forge
libcap: 2.77-h68e9139_0 conda-forge
- libcblas: 3.9.0-38_hd72aa62_openblas conda-forge
+ libcblas: 3.11.0-2_hd72aa62_openblas conda-forge
libcublas: 12.9.1.4-he38c790_1 conda-forge
libcudnn: 9.10.2.21-hd88968f_0 conda-forge
libcudnn-dev: 9.10.2.21-h76cf850_0 conda-forge
@@ -57,17 +57,17 @@ The following NEW packages will be INSTALLED:
libcusparse: 12.5.10.65-h8f3c8d4_2 conda-forge
libedit: 3.1.20250104-pl5321h976ea20_0 conda-forge
libev: 4.33-h31becfc_2 conda-forge
- libexpat: 2.7.1-hfae3067_0 conda-forge
+ libexpat: 2.7.3-hfae3067_0 conda-forge
libffi: 3.5.2-hd65408f_0 conda-forge
libgcc: 15.2.0-he277a41_7 conda-forge
libgcc-devel_linux-aarch64: 13.4.0-hd10b1b9_107 conda-forge
libgcc-ng: 15.2.0-he9431aa_7 conda-forge
libgfortran: 15.2.0-he9431aa_7 conda-forge
libgfortran5: 15.2.0-h87db57e_7 conda-forge
- libglib: 2.86.1-he84ff74_2 conda-forge
+ libglib: 2.86.2-hf53f6bf_1 conda-forge
libgomp: 15.2.0-he277a41_7 conda-forge
libiconv: 1.18-h90929bb_2 conda-forge
- liblapack: 3.9.0-38_h88aeb00_openblas conda-forge
+ liblapack: 3.11.0-2_h88aeb00_openblas conda-forge
liblzma: 5.8.1-h86ecc28_2 conda-forge
libmagma: 2.9.0-hef847a9_3 conda-forge
libnghttp2: 1.67.0-ha888d0e_0 conda-forge
@@ -83,17 +83,17 @@ The following NEW packages will be INSTALLED:
libstdcxx-devel_linux-aarch64: 13.4.0-hd10b1b9_107 conda-forge
libstdcxx-ng: 15.2.0-hf1166c9_7 conda-forge
libsystemd0: 258.2-h8c5a66d_1 conda-forge
- libtorch: 2.9.1-cuda129_generic_h5314ce0_200 local
+ libtorch: 2.9.1-cuda129_generic_hd20f0ec_200 local
libudev1: 258.2-h8c5a66d_1 conda-forge
libuv: 1.51.0-he30d5cf_1 conda-forge
libzlib: 1.3.1-h86ecc28_2 conda-forge
- llvm-openmp: 21.1.5-he40846f_2 conda-forge
+ llvm-openmp: 21.1.6-he40846f_0 conda-forge
nccl: 2.28.9.1-heee7246_0 conda-forge
ncurses: 6.5-ha32ae93_3 conda-forge
- ninja: 1.13.1-hdc560ac_0 conda-forge
+ ninja: 1.13.2-hdc560ac_0 conda-forge
nvtx-c: 3.3.0-h7ac5ae9_0 conda-forge
openssl: 3.6.0-h8e36d6e_0 conda-forge
- pcre2: 10.46-h15761aa_0 conda-forge
+ pcre2: 10.47-hf841c20_0 conda-forge
pkg-config: 0.29.2-hce167ba_1009 conda-forge
pybind11-abi: 11-hc364b38_1 conda-forge
rdma-core: 60.0-he839754_0 conda-forgeNot sure though with the CMake logs if the sanity checks were actually skipped or the (re)computation isn't shown in the logs; the passing run had while now we have |
|
This is only based on my intuition, but I would not exclude the possibility of a cmake 4.1.2 --> 4.2.0 regression. |
|
Same issue as conda-forge/numpy-feedstock#241. Will be fixed by conda-forge/binutils-feedstock#105 |
Thank you very much! Could you please explain a little what was the issue? Ostensibly we had the right binutils builds (latest build number, non-bootstrap, right platform), so while I noticed conda-forge/binutils-feedstock#104, that looked like a cosmetic issue to me. Also the numpy issue you're referencing is really old, and not immediately obviously related; the only hint is the title of your PR that fixed things at the time: "make sure packages have unique build strings for cross compiling". Which packages here had coinciding build strings, and how did that break the compiler detection only during the test phase? It's an impressive deduction to me - I have no idea what's going on. Would appreciate it if you could enlighten us. |
b6e57b4 to
2894cdd
Compare
|
conda extracts |
|
Ah, right, so it's a function of the package cache, and therefore not visible in the test environment (where we're not using the |
|
Sigh; I had merged after the issue had been resolved for the CPU build, but it appears there's yet another issue for the aarch + CUDA builds: This shouldn't really happen in emulation... 🤔 I see some clobber errors during environment creation, but not sure if they're related. |
|
For win+CUDA: $ gh run download 19654421211 --repo conda-forge/pytorch-cpu-feedstock --name conda_artifacts_19654421211_win_64_channel_targetsconda-forge_maincu_hca575dce
$ unzip pytorch-cpu-feedstock_conda_artifacts_.zip
$ cd bld/win-64
$ rm current_repodata.json index.html repodata*
$ ls
libtorch-2.9.1-cuda128_mkl_hbf96477_300.conda pytorch-gpu-2.9.1-cuda128_mkl_hc88b545_300.conda
pytorch-2.9.1-cuda128_mkl_py310_hcc6672c_300.conda pytorch-tests-2.9.1-cuda128_mkl_py310_h6b46e55_300.conda
pytorch-2.9.1-cuda128_mkl_py311_h7c65ee9_300.conda pytorch-tests-2.9.1-cuda128_mkl_py311_ha41f3a0_300.conda
pytorch-2.9.1-cuda128_mkl_py312_h003053b_300.conda pytorch-tests-2.9.1-cuda128_mkl_py312_hd7dd76b_300.conda
pytorch-2.9.1-cuda128_mkl_py313_h7f80487_300.conda pytorch-tests-2.9.1-cuda128_mkl_py313_h976a8e3_300.conda
$ ls | xargs anaconda upload
$ DELEGATE=h-vetinari
PACKAGE_VERSION=2.9.1
for package in libtorch pytorch pytorch-gpu pytorch-tests; do
anaconda copy --from-label main --to-label main --to-owner conda-forge ${DELEGATE}/${package}/${PACKAGE_VERSION}
done |
|
Interestingly, the win+CUDA build passed upon restarting, but the upload section is completely empty I can only infer that it didn't even attempt to upload stuff because it found artefacts with exactly matching names already on the CDN, but I still would have expected some logs. However, I checked this on another feedstock where pushing an LTS branch means CI for some past commit on |
|
Thankfully, the aarch+CUDA build passed on main. That means we'll be able to start the migration without much further delay! 🥳 |
It is very likely that the current package version for this feedstock is out of date.
Checklist before merging this PR:
license_fileis packagedInformation about this PR:
@conda-forge-admin,please add bot automergein the title and merge the resulting PR. This command will add our bot automerge feature to your feedstock.bot-rerunlabel to this PR. The bot will close this PR and schedule another one. If you do not have permissions to add this label, you can use the phrase@conda-forge-admin, please rerun botin a PR comment to have theconda-forge-adminadd it for you.Closes: #415
Closes: #433
Closes: #446
Pending Dependency Version Updates
Here is a list of all the pending dependency version updates for this repo. Please double check all dependencies before merging.
This PR was created by the regro-cf-autotick-bot. The regro-cf-autotick-bot is a service to automatically track the dependency graph, migrate packages, and propose package version updates for conda-forge. Feel free to drop us a line if there are any issues! This PR was generated by https://github.com/regro/cf-scripts/actions/runs/19320428618 - please use this URL for debugging.