{ai}[foss/2024a] PyTorch v2.9.1 w/ CUDA 12.6.0 by Flamefire · Pull Request #24365 · easybuilders/easybuild-easyconfigs

Flamefire · 2025-10-24T16:33:41Z

(created using eb --new-pr)

Requires:

github-actions · 2025-10-24T16:34:27Z

Diff of new easyconfig(s) against existing ones is too long for a GitHub comment. Use --review-pr (and --review-pr-filter / --review-pr-max) locally.

Flamefire · 2025-12-10T15:26:53Z

Test report by @Flamefire
FAILED
Build succeeded for 0 out of 1 (total: 27 hours 27 mins 43 secs) (1 easyconfigs in total)
c144 - Linux Rocky Linux 9.6, x86_64, AMD EPYC 9334 32-Core Processor (zen4), 4 x NVIDIA NVIDIA H100, 580.65.06, Python 3.9.21
See https://gist.github.com/Flamefire/50cdd13305fd9a33c6140c223aeab6cd for a full test report.

boegel · 2025-12-13T10:31:58Z

Test report by @boegel
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3803
FAILED
Build succeeded for 4 out of 6 (total: 8 mins 45 secs) (6 easyconfigs in total)
node3907.accelgor.os - Linux RHEL 9.6, x86_64, AMD EPYC 7413 24-Core Processor (zen3), 1 x NVIDIA NVIDIA A100-SXM4-80GB, 580.95.05, Python 3.9.21
See https://gist.github.com/boegel/e06f98f956452bcb8a132f816e55927b for a full test report.

easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2024a-CUDA-12.6.0.eb

easybuild/easyconfigs/t/Triton/Triton-3.5.0-gfbf-2024a-CUDA-12.6.0.eb

boegel · 2025-12-13T16:00:22Z

I'm also seeing a crash with the triton_test.py script:

== FAILED: Installation ended unsuccessfully: Sanity check failed: sanity check command TRITON_HOME=$TMPDIR/eb-triton_home python
/software/Triton/3.5.0-gfbf-2024a-CUDA-12.6.0/test/triton_test.py 8.0 failed with exit code 1 (output: Traceback (most recent call last):
  File "/software/Triton/3.5.0-gfbf-2024a-CUDA-12.6.0/test/triton_test.py", line 13, in <module>
    src = triton.compiler.ASTSource(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/software/Triton/3.5.0-gfbf-2024a-CUDA-12.6.0/lib/python3.12/site-packages/triton/compiler/compiler.py", line 67, in __init__
    for k in self.signature.keys():
             ^^^^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'keys'

Flamefire · 2025-12-13T17:36:33Z

That's why the checksum changed: Breaking change in Triton 3.5 and I updated the test script accordingly. It is in #24793

Flamefire · 2025-12-15T16:17:23Z

I had to use tlparse 0.4.0 (also separate PR in #24882) as the older one isn't compatible with PyTorch output, see pytorch/pytorch@92c2dae

The lowest tlparse version that works is 0.3.42.

Not sure if this causes conflicts in EB. The alternative is to drop this dependency as it is optional

Flamefire · 2025-12-16T17:05:22Z

Rebased to remove EasyConfigs present in develop from this branch.

Also added 2 more patches to avoid remaining failures.

Flamefire · 2025-12-16T17:10:33Z

Test report by @Flamefire
FAILED
Build succeeded for 4 out of 5 (total: 4 mins 48 secs) (5 easyconfigs in total)
c144 - Linux Rocky Linux 9.6, x86_64, AMD EPYC 9334 32-Core Processor (zen4), 4 x NVIDIA NVIDIA H100, 580.65.06, Python 3.9.21
See https://gist.github.com/Flamefire/6f2e5c7a020ec72dff9d2f5c8220fba5 for a full test report.

Flamefire · 2025-12-18T09:53:22Z

Test report by @Flamefire
SUCCESS
Build succeeded for 5 out of 5 (total: 24 hours 42 mins 21 secs) (5 easyconfigs in total)
c144 - Linux Rocky Linux 9.6, x86_64, AMD EPYC 9334 32-Core Processor (zen4), 4 x NVIDIA NVIDIA H100, 580.65.06, Python 3.9.21
See https://gist.github.com/Flamefire/a6bd885643124f5fac4864060e0e18cd for a full test report.

…es: PyTorch-2.9.0_fix-nccl-test-env.patch, PyTorch-2.9.0_readd-support-for-nvidia-cutlass-python-package.patch

boegel · 2026-01-20T20:48:42Z

@Flamefire Can we sync this with develop after merge of #23923?

boegel · 2026-01-21T19:01:44Z

@boegelbot please test @ jsc-zen3-a100
CORE_CNT=16

boegel · 2026-01-21T19:04:38Z

easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2024a-CUDA-12.6.0.eb

+
+runtest = (
+    ' TORCH_DISABLE_ADDR2LINE=1'
+    ' TORCHINDUCTOR_CUTLASS_DIR=%(start_dir)s/third_party/cutlass'


@Flamefire Just wondering: a nvidia-cutlass dependency is not a (good) option for PyTorch 2.9.1?

I tried but the structure is different. PyTorch expects a literal git checkout -.-

At least there is nothing to be built, just a plain copy

easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2024a-CUDA-12.6.0.eb

boegel · 2026-01-21T19:07:00Z

@Flamefire I guess we'll need an increase for max_failed_tests here too, like we did in #23923 (w.r.t. test results on V100 system)?
I can look into submitting a test report from a V100 system to see how "bad" it is...

boegelbot · 2026-01-21T19:10:09Z

@boegel: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=24365 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_24365 --ntasks="16" --partition=jsczen3g --gres=gpu:1 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 9464

Test results coming soon (I hope)...

Details

- notification for comment with ID 3780659015 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegel · 2026-01-23T05:02:49Z

Test report by @boegel
FAILED
Build succeeded for 5 out of 6 (total: 34 hours 38 mins 31 secs) (3 easyconfigs in total)
node4302.litleo.os - Linux RHEL 9.6, x86_64, AMD EPYC 9454P 48-Core Processor (zen4), 1 x NVIDIA NVIDIA H100 NVL, 580.95.05, Python 3.9.21
See https://gist.github.com/boegel/aeb6cd33ac5a5e07c7b5039b6906a789 for a full test report.

boegel · 2026-01-23T07:40:50Z

Test report by @boegel
FAILED
Build succeeded for 5 out of 6 (total: 37 hours 16 mins 29 secs) (3 easyconfigs in total)
node3902.accelgor.os - Linux RHEL 9.6, x86_64, AMD EPYC 7413 24-Core Processor (zen3), 1 x NVIDIA NVIDIA A100-SXM4-80GB, 590.48.01, Python 3.9.21
See https://gist.github.com/boegel/64f44a20710dd30f7c235e8a420f3b09 for a full test report.

Flamefire · 2026-01-23T10:57:09Z

export/test_export_opinfo (431 failed, 201 passed, 40 skipped, 0 errors)

Something's off there...

The latest (below) has 2 missing suites:

Missing: inductor/test_flex_attention, inductor/test_flex_decoding

boegel · 2026-01-23T17:16:05Z

Test report by @boegel
FAILED
Build succeeded for 5 out of 6 (total: 45 hours 50 mins 52 secs) (3 easyconfigs in total)
node3302.joltik.os - Linux RHEL 9.6, x86_64, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz (cascadelake), 1 x NVIDIA Tesla V100-SXM2-32GB, 580.95.05, Python 3.9.21
See https://gist.github.com/boegel/7326c174113e98762f39ea9c1da74aa9 for a full test report.

boegelbot · 2026-01-23T20:31:59Z

Test report by @boegelbot
FAILED
Build succeeded for 4 out of 5 (total: 49 hours 21 mins 18 secs) (3 easyconfigs in total)
jsczen3g1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.7, x86_64, AMD EPYC-Milan Processor (zen3), 1 x NVIDIA NVIDIA A100 80GB PCIe, 590.44.01, Python 3.9.23
See https://gist.github.com/boegelbot/18f3b5048f76bfd992289b7dfd84b363 for a full test report.

boegel · 2026-01-24T14:10:21Z

Very similar result from our A100 and H100 systems. For H100:

WARNING: 438 test failures, 0 test errors (out of 269217):
        distributed/elastic/test_control_plane (1 failed, 8 passed, 0 skipped, 0 errors)
        distributed/optim/test_zero_redundancy_optimizer (2 failed, 10 passed, 30 skipped, 0 errors)
        distributed/test_store (1 failed, 40 passed, 12 skipped, 0 errors)
        dynamo/test_error_messages (1 failed, 40 passed, 0 skipped, 0 errors)
        dynamo/test_package (1 failed, 31 passed, 17 skipped, 0 errors)
        export/test_export_opinfo (431 failed, 201 passed, 40 skipped, 0 errors)
        inductor/test_fxir_backend (1 failed, 37 passed, 0 skipped, 0 errors)

So export/test_export_opinfo is the bad guy here...
We already have a patch for this test.

Similar issue on jsc-zen3:

== 2026-01-23 20:31:52,673 build_log.py:440 WARNING 436 test failures, 0 test errors (out of 269029):
	distributed/optim/test_zero_redundancy_optimizer (2 failed, 10 passed, 30 skipped, 0 errors)
	export/test_export_opinfo (431 failed, 201 passed, 40 skipped, 0 errors)
	functorch/test_ops (2 failed, 7495 passed, 2725 skipped, 0 errors)
	inductor/test_fxir_backend (1 failed, 37 passed, 0 skipped, 0 errors)

Flamefire · 2026-02-01T14:40:16Z

So export/test_export_opinfo is the bad guy here...

Yes see also my previous comment. Can you attach the log? I assume it is a common issue that will easily fix all/most tests

Flamefire · 2026-02-04T05:21:28Z

Test report by @Flamefire
FAILED
Build succeeded for 2 out of 3 (total: 9 hours 34 mins 53 secs) (3 easyconfigs in total)
n1270.barnard.hpc.tu-dresden.de - Linux RHEL 9.6, x86_64, Intel(R) Xeon(R) Platinum 8470 (sapphirerapids), Python 3.9.21
See https://gist.github.com/Flamefire/f845f5c9c51b5e3b91bed7ff4df881cf for a full test report.

Flamefire · 2026-02-06T06:29:46Z

Test report by @Flamefire
SUCCESS
Build succeeded for 3 out of 3 (total: 10 hours 24 mins 14 secs) (3 easyconfigs in total)
n1195.barnard.hpc.tu-dresden.de - Linux RHEL 9.6, x86_64, Intel(R) Xeon(R) Platinum 8470 (sapphirerapids), Python 3.9.21
See https://gist.github.com/Flamefire/7037cde389d6fd22527c67d0f2fad5d7 for a full test report.

Flamefire · 2026-02-10T23:31:54Z

Test report by @Flamefire
SUCCESS
Build succeeded for 3 out of 3 (total: 24 hours 15 mins 20 secs) (3 easyconfigs in total)
c96 - Linux Rocky Linux 9.6, x86_64, AMD EPYC 9334 32-Core Processor (zen4), 4 x NVIDIA NVIDIA H100, 580.65.06, Python 3.9.21
See https://gist.github.com/Flamefire/9032201f62e9b79550a110fe7b561779 for a full test report.

Flamefire · 2026-02-13T02:15:27Z

Test report by @Flamefire
SUCCESS
Build succeeded for 3 out of 3 (total: 10 hours 7 mins 8 secs) (3 easyconfigs in total)
n1366.barnard.hpc.tu-dresden.de - Linux RHEL 9.6, x86_64, Intel(R) Xeon(R) Platinum 8470 (sapphirerapids), Python 3.9.21
See https://gist.github.com/Flamefire/2e975fd309b691ebb4b1976275e7e2c5 for a full test report.

…h290

Flamefire marked this pull request as draft October 24, 2025 16:33

github-actions bot added the update label Oct 24, 2025

Thyre added the 2024a issues & PRs related to 2024a common toolchains label Oct 25, 2025

This was referenced Nov 14, 2025

pyannote.audio 4.0.1 or newer + WhisperX vscentrum/vsc-software-stack#612

Open

whisperx vscentrum/vsc-software-stack#613

Open

Flamefire marked this pull request as ready for review December 9, 2025 11:31

Flamefire changed the title ~~{ai}[foss/2024a] PyTorch v2.9.0 w/ CUDA 12.6.0~~ {ai}[foss/2024a] PyTorch v2.9.1 w/ CUDA 12.6.0 Dec 9, 2025

Flamefire force-pushed the 20251024183337_new_pr_PyTorch290 branch from 9849937 to 15b85aa Compare December 9, 2025 11:45

github-actions bot added the new label Dec 9, 2025

boegel requested changes Dec 13, 2025

View reviewed changes

easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2024a-CUDA-12.6.0.eb Show resolved Hide resolved

boegel requested changes Dec 13, 2025

View reviewed changes

easybuild/easyconfigs/t/Triton/Triton-3.5.0-gfbf-2024a-CUDA-12.6.0.eb Show resolved Hide resolved

Flamefire force-pushed the 20251024183337_new_pr_PyTorch290 branch from 2b8bc42 to 64a4d67 Compare December 16, 2025 17:04

github-actions bot removed the new label Dec 16, 2025

Flamefire force-pushed the 20251024183337_new_pr_PyTorch290 branch 2 times, most recently from 01180ef to 381c028 Compare December 17, 2025 09:09

boegel added this to the release after 5.2.0 milestone Dec 18, 2025

Flamefire added 3 commits December 18, 2025 10:56

adding easyconfigs: PyTorch-2.9.0-foss-2024a-CUDA-12.6.0.eb and patch…

5cf4096

…es: PyTorch-2.9.0_fix-nccl-test-env.patch, PyTorch-2.9.0_readd-support-for-nvidia-cutlass-python-package.patch

Add dependencies

9beff55

Use tlparse 0.4.0

867d1a2

Flamefire force-pushed the 20251024183337_new_pr_PyTorch290 branch from 9e1d5c8 to 867d1a2 Compare December 18, 2025 09:57

boegel mentioned this pull request Dec 18, 2025

Fix issues with PyTorch test-results (XML files) parsing and add tests easybuilders/easybuild-easyblocks#3803

Merged

Merge branch 'develop' into 20251024183337_new_pr_PyTorch290

f5fbb91

boegel reviewed Jan 21, 2026

View reviewed changes

easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2024a-CUDA-12.6.0.eb Show resolved Hide resolved

Flamefire added 2 commits January 22, 2026 16:17

Add patch avoiding infinite test hang

935c57a

More patches

874f0c5

Add comment on TORCH_DISABLE_ADDR2LINE variable

8a7320e

Flamefire added 2 commits February 5, 2026 14:22

Fix patched skip markers

7a46620

Add comment for DISABLE_ADDR2LINE

de9d0f4

Flamefire added 2 commits February 9, 2026 15:51

Set test timeout

6015d47

Add GCC 14 patch

c342afd

Add patches for test fixes and skip slow&disabled tests

a02dba5

Flamefire force-pushed the 20251024183337_new_pr_PyTorch290 branch from 9b793dd to a02dba5 Compare February 12, 2026 15:37

Remove duplicate source_url

1a86b08

Flamefire and others added 2 commits February 13, 2026 11:04

Add PyTorch-2.6.0_fix-server-in-test_control_plane

7cc912c

Merge branch 'easybuilders:develop' into 20251024183337_new_pr_PyTorc…

096f3ad

…h290

Conversation

Flamefire commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Flamefire commented Dec 10, 2025

Uh oh!

boegel commented Dec 13, 2025

Uh oh!

Uh oh!

Uh oh!

boegel commented Dec 13, 2025

Uh oh!

Flamefire commented Dec 13, 2025

Uh oh!

Flamefire commented Dec 15, 2025

Uh oh!

Flamefire commented Dec 16, 2025

Uh oh!

Flamefire commented Dec 16, 2025

Uh oh!

Flamefire commented Dec 18, 2025

Uh oh!

boegel commented Jan 20, 2026

Uh oh!

boegel commented Jan 21, 2026

Uh oh!

boegel Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

Flamefire Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

boegel commented Jan 21, 2026

Uh oh!

boegelbot commented Jan 21, 2026

Uh oh!

boegel commented Jan 23, 2026

Uh oh!

boegel commented Jan 23, 2026

Uh oh!

Flamefire commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel commented Jan 23, 2026

Uh oh!

boegelbot commented Jan 23, 2026

Uh oh!

boegel commented Jan 24, 2026

Uh oh!

Flamefire commented Feb 1, 2026

Uh oh!

Flamefire commented Feb 4, 2026

Uh oh!

Flamefire commented Feb 6, 2026

Uh oh!

Flamefire commented Feb 10, 2026

Uh oh!

Flamefire commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Flamefire commented Oct 24, 2025 •

edited

Loading

github-actions bot commented Oct 24, 2025 •

edited

Loading

Flamefire commented Jan 23, 2026 •

edited

Loading