Migrate for CUDA 12.9 by h-vetinari · Pull Request #7476 · conda-forge/conda-forge-pinning-feedstock

h-vetinari · 2025-06-13T04:23:55Z

Builds on top of #7005 after the problems there were rendered obsolete by dropping CUDA 11.8 (c.f. #7404, #7431)

As a demo, I've opened conda-forge/pytorch-cpu-feedstock#393 ~~though this currently needs a smithy PR (conda-forge/conda-smithy#2335) due to an issue with the variant algebra for exactly the case we want to do here: conda-forge/conda-smithy#2331~~

Closes #7005
Closes #6980

conda-forge-admin · 2025-06-13T04:25:14Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

h-vetinari · 2025-06-13T06:09:00Z

Draft until we fix smithy to update correctly

h-vetinari · 2025-06-14T20:36:32Z

The smithy changes have landed and were released in 3.50.1 (thanks @beckermr!), so this is ready for review!

PTAL especially @conda-forge/cuda.

Perhaps relevant: there's some weird linker errors in conda-forge/pytorch-cpu-feedstock#393 that seem to be due to the CUDA 12.9 toolchain (or some interaction with it).

hmaarrfk

I beleive you have addressed all the technical blockers correct.

h-vetinari · 2025-06-17T23:32:27Z

Blockers from the infrastructure side should all be resolved, but we haven't got a passing 12.9 build for pytorch yet, and I'd like to understand what's going wrong in the toolchain there (aside from some input or green light from @conda-forge/cuda on this in general).

Feedstocks that want to build with 12.9 can do so of course (and feedback would be welcome!): simply copy the migrator from this PR, add use_local: true and then rerender.

hmaarrfk · 2025-06-17T23:39:30Z

Perfect. I guess my approval is on the structure of this PR and once the PyTorch build is ready this can be merged without further input from me

bdice · 2025-06-18T16:01:25Z

+  - 12.9                       # [((linux and (x86_64 or aarch64)) or win64) and os.environ.get("CF_CUDA_ENABLED", "False") == "True"]
+
+c_compiler_version:            # [(linux and (x86_64 or aarch64)) and os.environ.get("CF_CUDA_ENABLED", "False") == "True"]
+  - 13                         # [(linux and (x86_64 or aarch64)) and os.environ.get("CF_CUDA_ENABLED", "False") == "True"]


CUDA 12.8 and 12.9 both support GCC 14. I haven't tracked the GCC 14 migration elsewhere on conda-forge enough to know if this should be bumped to 14 or not.

I know, and I'm planning to make use of this. If #7421 gets merged first, I'll update to 14 here. Or if this PR gets merged first, I'll bump the pin in the cuda129.yaml file in the other PR.

On the topic of GCC 14 (which we'll bump to in a few days), it seems that we maybe should stay on GCC 13 for CUDA 12.9 for now. At least on the pytorch side, this combination ran into issues, namely

Looks like GCC 14 might be premature, at least for pytorch (or at least without turning off -Wincompatible-pointer-types):

$SRC_DIR/third_party/XNNPACK/src/f16-conv-hwc2chw/f16-conv-hwc2chw-3x3s2p1c3x4-neonfp16arith-2x2.c:53:62: error: passing argument 1 of 'vld1_dup_u16' from incompatible pointer type [-Wincompatible-pointer-types] 53 | const float16x4_t vmax = vreinterpret_f16_u16(vld1_dup_u16(&params->scalar.max)); | ^~~~~~~~~~~~~~~~~~~ | | | const xnn_float16 * {aka const _Float16 *} In file included from $SRC_DIR/third_party/XNNPACK/src/f16-conv-hwc2chw/f16-conv-hwc2chw-3x3s2p1c3x4-neonfp16arith-2x2.c:8: $BUILD_PREFIX/lib/gcc/aarch64-conda-linux-gnu/14.3.0/include/arm_neon.h:13130:31: note: expected 'const uint16_t *' {aka 'const short unsigned int *'} but argument is of type 'const xnn_float16 *' {aka 'const _Float16 *'} 13130 | vld1_dup_u16 (const uint16_t* __a) | ~~~~~~~~~~~~~~~~^~~

Curious also that this doesn't seem to be an issue on x64, only on aarch64.

At first glance it appears that the type of params->scalar.max gets messed up, because casting from _Float16 to uint16 sounds very risky, and the vld1_dup_u16 in GCC really is about integers (so I can't see how it'd be a case of picking the wrong overload).

So #7421 has been merged now. But for now I'm leaving CUDA 12.9 on GCC 13, until the above issue gets fixed or someone tells me that the issue is somehow specific to pytorch.

Sounds good to me!

[...] or someone tells me that the issue is somehow specific to pytorch.

FYI, since it turns out that the problems were specific to pytorch, I'm bumping the CUDA 12.9 migrator to GCC 14 now, to match the rest of the pinning: #7563

h-vetinari · 2025-07-02T00:37:03Z

We finally merged conda-forge/pytorch-cpu-feedstock#393, though on windows we had to downgrade to 12.8 because 12.9 was OOM-ing even on the largest possible machine. I'm fine with keeping this specific to pytorch (which is a beast to build anyway), as long as we're reasonably confident that there are no big unresolved issues with 12.9 on windows. It does seem like the toolchain has a problem (or a regression) there though.

h-vetinari · 2025-07-08T23:23:32Z

@conda-forge/cuda can someone please comment whether this is good to go from your end. Several feedstocks are waiting to support the new architectures.

I think the remaining open points encountered specifically on the pytorch feedstock (win+12.9 OOMs but works with 12.8; linux compilation errors when using GCC 14) aren't big enough to be blockers for getting this started.

bdice

This looks fine to me. Maybe @jakirkham or @carterbox can take a quick peek before merging?

bdice · 2025-07-11T03:07:00Z

+  - 12.9                       # [((linux and (x86_64 or aarch64)) or win64) and os.environ.get("CF_CUDA_ENABLED", "False") == "True"]
+
+c_compiler_version:            # [(linux and (x86_64 or aarch64)) and os.environ.get("CF_CUDA_ENABLED", "False") == "True"]
+  - 13                         # [(linux and (x86_64 or aarch64)) and os.environ.get("CF_CUDA_ENABLED", "False") == "True"]


Sounds good to me!

Co-authored-by: Daniel Ching <9604511+carterbox@users.noreply.github.com>

h-vetinari · 2025-07-12T00:15:10Z

Alright, thanks for the inputs @bdice @carterbox. I'll merge this in 72h unless there are other comments.

xref: conda-forge/conda-forge-pinning-feedstock#7476

jakirkham and others added 5 commits February 7, 2025 16:23

Add a migrator for CUDA 12.8

2254197

Merge remote-tracking branch 'upstream/main' into add_cuda128

f62cd96

drop keys in cuda 11.8 migrator that aren't part of zip anymore

2252dbd

use CUDA 12.9

31cbc59

remove operation: key_add

0390e04

h-vetinari requested a review from a team as a code owner June 13, 2025 04:23

h-vetinari mentioned this pull request Jun 13, 2025

Rebuild for CUDA 12.9 conda-forge/pytorch-cpu-feedstock#393

Merged

fix ordering to keep CPU builds

57df0d7

h-vetinari force-pushed the cuda129 branch from cd22ce7 to d0ec4c8 Compare June 13, 2025 04:46

do not set {c,cxx,fortran}_compiler_version on windows

6bd655a

h-vetinari force-pushed the cuda129 branch from d0ec4c8 to 6bd655a Compare June 13, 2025 04:52

h-vetinari mentioned this pull request Jun 13, 2025

Adding CUDA 12.9 migration #6980

Closed

h-vetinari marked this pull request as draft June 13, 2025 06:08

make cuda 12.9 migrator compatible with cuda11.8 opt-in

859c0ea

h-vetinari mentioned this pull request Jun 13, 2025

Add CUDA 11.8 manual migrator (redux) #7472

Merged

h-vetinari marked this pull request as ready for review June 14, 2025 20:33

hmaarrfk approved these changes Jun 14, 2025

View reviewed changes

weiji14 mentioned this pull request Jun 16, 2025

flash-attn v2.8.0.post2 conda-forge/flash-attn-feedstock#36

Closed

3 tasks

hmaarrfk approved these changes Jun 17, 2025

View reviewed changes

bdice reviewed Jun 18, 2025

View reviewed changes

update architecture list and instructions in cuda129 migrator

3d8cab8

h-vetinari mentioned this pull request Jun 29, 2025

GCC 14 & Clang 19 #7421

Merged

19 tasks

h-vetinari mentioned this pull request Jul 7, 2025

Adding CUDA 12.8 Support conda-forge/pytorch-cpu-feedstock#390

Closed

weiji14 mentioned this pull request Jul 11, 2025

flash-attn v2.8.1 conda-forge/flash-attn-feedstock#37

Closed

3 tasks

bdice approved these changes Jul 11, 2025

View reviewed changes

carterbox reviewed Jul 11, 2025

View reviewed changes

Comment thread recipe/migrations/cuda129.yaml

add explanation about arches set by nvcc already

824e51d

Co-authored-by: Daniel Ching <9604511+carterbox@users.noreply.github.com>

leofang added a commit to regro-cf-autotick-bot/cupy-feedstock that referenced this pull request Jul 13, 2025

Use CUDA 12.9 to build

e00f4cf

xref: conda-forge/conda-forge-pinning-feedstock#7476

leofang mentioned this pull request Jul 13, 2025

use different name for CUDA 11.8 opt-in migrator #7487

Closed

carterbox approved these changes Jul 14, 2025

View reviewed changes

h-vetinari merged commit cab1800 into conda-forge:main Jul 14, 2025
3 checks passed

h-vetinari deleted the cuda129 branch July 14, 2025 21:48

This was referenced Jul 15, 2025

Bump GCC versions in CUDA 12.9 migrator #7563

Merged

RFC: Drop CUDA support on ppc64le #7571

Closed

h-vetinari mentioned this pull request Aug 18, 2025

NEW: Migrate recipes to CUDA 13.0 #7653

Merged

11 tasks

Uh oh!

Conversation

h-vetinari commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

conda-forge-admin commented Jun 13, 2025

Uh oh!

h-vetinari commented Jun 13, 2025

Uh oh!

h-vetinari commented Jun 14, 2025

Uh oh!

hmaarrfk left a comment

Choose a reason for hiding this comment

Uh oh!

h-vetinari commented Jun 17, 2025

Uh oh!

hmaarrfk commented Jun 17, 2025

Uh oh!

Uh oh!

bdice Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

h-vetinari Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

h-vetinari Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

h-vetinari Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

bdice Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

h-vetinari Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

h-vetinari commented Jul 2, 2025

Uh oh!

h-vetinari commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bdice left a comment

Choose a reason for hiding this comment

Uh oh!

bdice Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

h-vetinari commented Jul 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

h-vetinari commented Jun 13, 2025 •

edited

Loading

h-vetinari Jul 15, 2025 •

edited

Loading

h-vetinari commented Jul 8, 2025 •

edited

Loading