Skip to content

{tools}[GCCcore/14.2.0] ROCm-LLVM v19.0.0 w/ ROCm 6.4.1#23304

Merged
bedroge merged 15 commits intoeasybuilders:developfrom
Thyre:20250703205604_new_pr_ROCm-LLVM641
Jan 23, 2026
Merged

{tools}[GCCcore/14.2.0] ROCm-LLVM v19.0.0 w/ ROCm 6.4.1#23304
bedroge merged 15 commits intoeasybuilders:developfrom
Thyre:20250703205604_new_pr_ROCm-LLVM641

Conversation

@Thyre
Copy link
Collaborator

@Thyre Thyre commented Jul 3, 2025

(created using eb --new-pr)

Requires:


This is the first major step towards building ROCm and it's components. ROCm-LLVM is a fork of upstream LLVM with several major additions. This complicates the build process a lot, e.g. requiring to basically build ROCm-LLVM twice just to get some offload components working correctly.

Most of the heavy lifting was done by @bedroge, most of my work was just to bring this to GCCcore/14.2.0, figure out build issues and adding the patch to get the OpenMP Tools Interface working.

We skip building and running tests, as the build already takes ages.

One important difference to "upstream" (i.e. AMD's fork) is that we build amdflang from LLVM. We therefore choose to adopt the new flang earlier than AMD does for their official packages. I don't see a reason to still try to go with the old flang.

@Thyre Thyre marked this pull request as draft July 3, 2025 18:56
@github-actions github-actions bot added the new label Jul 3, 2025
…Cm-LLVM-6.4.1_llvm-project-19.0.0_fix-offload-build.patch

Co-Authored-By: Bob Dröge <b.e.droge@rug.nl>
@Thyre Thyre force-pushed the 20250703205604_new_pr_ROCm-LLVM641 branch from 2bf841b to 7c46968 Compare July 3, 2025 19:03
@Thyre
Copy link
Collaborator Author

Thyre commented Jul 28, 2025

MI250X node with --amdgcn-capabilities=gfx90a and --cuda-compute-capabilities=7.5,8.0

Test report by @Thyre
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3823
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
jrc0850.jureca - Linux Rocky Linux 9.5 (Blue Onyx), x86_64, AMD EPYC 7443 24-Core Processor (zen3), Python 3.9.21
See https://gist.github.com/Thyre/7779db041e343ac7d2cd3408787be6c4 for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jul 28, 2025

@boegelbot please test @ jsc-zen3
EB_ARGS="--include-easyblocks-from-pr=3823 --amdgcn-capabilities=gfx90a --installpath=/tmp/$USER/pr23304"

@boegelbot
Copy link
Collaborator

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=23304 EB_ARGS="--include-easyblocks-from-pr=3823 --amdgcn-capabilities=gfx90a --installpath=/tmp/$USER/pr23304" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_23304 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 7376

Test results coming soon (I hope)...

Details

- notification for comment with ID 3126806295 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Collaborator

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3823
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
jsczen3c2.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/532c6af9a7fd099de71a95e064894054 for a full test report.

@bedroge
Copy link
Contributor

bedroge commented Jul 29, 2025

MI100+MI210 node with --amdgcn-capabilities=gfx90a,gfx908.

Test report by @bedroge
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3823
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
il-c04 - Linux Rocky Linux 9.5 (Blue Onyx), x86_64, AMD EPYC 7542 32-Core Processor, 1 x AMD Arcturus GL-XL [Instinct MI100] (model: 0x738c, driver: "6.8.5"), 1 x AMD Instinct MI210 (model: 0x740f, driver: "6.8.5"), Python 3.9.21
See https://gist.github.com/bedroge/e7b726fa933f4f7f83065786d5fe22b8 for a full test report.

@Thyre Thyre marked this pull request as ready for review July 29, 2025 11:42
@Thyre Thyre added the 2025a issues & PRs related to 2025a common toolchains label Aug 1, 2025
@boegel boegel added this to the 5.x milestone Aug 18, 2025
Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>
@Thyre Thyre changed the title {tools}[GCCcore/14.2.0] ROCm-LLVM v6.4.1 {tools}[GCCcore/14.2.0] ROCm-LLVM v19.0.0 w/ ROCm 6.4.1 Oct 22, 2025
@Thyre
Copy link
Collaborator Author

Thyre commented Oct 24, 2025

@boegelbot please test @ jsc-zen3
EB_ARGS="--include-easyblocks-from-pr=3823 --amdgcn-capabilities=gfx90a --installpath=/tmp/$USER/pr23304"

@boegelbot
Copy link
Collaborator

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=23304 EB_ARGS="--include-easyblocks-from-pr=3823 --amdgcn-capabilities=gfx90a --installpath=/tmp/$USER/pr23304" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_23304 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 8540

Test results coming soon (I hope)...

Details

- notification for comment with ID 3444812706 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Collaborator

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3823
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/d3dbf1e3a2ecff938dd662e159b5a75a for a full test report.

Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>
@Thyre
Copy link
Collaborator Author

Thyre commented Oct 24, 2025

@boegelbot please test @ jsc-zen3
EB_ARGS="--include-easyblocks-from-pr=3823 --amdgcn-capabilities=gfx90a --installpath=/tmp/$USER/pr23304"

@boegelbot
Copy link
Collaborator

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=23304 EB_ARGS="--include-easyblocks-from-pr=3823 --amdgcn-capabilities=gfx90a --installpath=/tmp/$USER/pr23304" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_23304 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 8541

Test results coming soon (I hope)...

Details

- notification for comment with ID 3444883950 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Collaborator

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3823
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/2da3b69bcbe235dce38526c83c693e02 for a full test report.

…2.0-ROCm-6.4.1.eb

Co-authored-by: Bob Dröge <b.e.droge@rug.nl>
@boegelbot
Copy link
Collaborator

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3823
FAILED
Build succeeded for 1 out of 2 (total: 5 hours 53 mins 20 secs) (2 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/cba049031884d0c329eae34bc801e6ff for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Dec 4, 2025

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3823
FAILED
Build succeeded for 1 out of 2 (total: 5 hours 53 mins 20 secs) (2 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/cba049031884d0c329eae34bc801e6ff for a full test report.

Maybe this is happening due to us working with a bundle? Could be tested by creating an artificial EasyConfig moving LLVM to a bundle with just LLVM and then running --module-only

@Thyre
Copy link
Collaborator Author

Thyre commented Dec 4, 2025

After hacking around in the Bundle EasyBlock (since the sanity check is broken for --module-only and --sanity-check-only), we finally have our answer. It's most likely a bug in the Bundle EasyBlock:

  >> running command 'cd /tmp/eb-cs7_qwi5/tmpv1ymtzr5 && g++ minimal.cpp -o minimal_cpp $(llvm-config --link-static --system-libs all)' ...
  >> result for command 'cd /tmp/eb-cs7_qwi5/tmpv1ymtzr5 && g++ minimal.cpp -o minimal_cpp $(llvm-config --link-static --system-libs all)': FAILED
== ... (took 12 secs)
== FAILED: Installation ended unsuccessfully: Sanity check failed: sanity check command cd /tmp/eb-cs7_qwi5/tmpv1ymtzr5 && g++ minimal.cpp -o minimal_cpp $(llvm-config --link-static --system-libs all) failed with exit code 1 (output:
/opt/EasyBuild/apps/software/binutils/2.44-GCCcore-14.3.0/bin/ld: cannot find -lzstd: No such file or directory
/opt/EasyBuild/apps/software/binutils/2.44-GCCcore-14.3.0/bin/ld: cannot find -lxml2: No such file or directory

I also encountered a failure in the LLVM EasyBlock once I got something hacky working, but I wouldn't rule out that this was because of my hacky solution...

I'll write up an issue for this at some point.

@bedroge
Copy link
Contributor

bedroge commented Dec 5, 2025

Test report by @bedroge
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3823
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
il-c04 - Linux Rocky Linux 9.5 (Blue Onyx), x86_64, AMD EPYC 7542 32-Core Processor, 1 x AMD Arcturus GL-XL [Instinct MI100] (model: 0x738c, driver: "6.8.5"), 1 x AMD Instinct MI210 (model: 0x740f, driver: "6.8.5"), Python 3.9.21
See https://gist.github.com/bedroge/d46641ca3b91e3a20cfb08813711a573 for a full test report.

@bedroge
Copy link
Contributor

bedroge commented Dec 6, 2025

Test report by @bedroge
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3823
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
x86-64-amd-zen2-node2.int.aws-rocky88-202310.eessi.io - Linux Rocky Linux 8.10 (Green Obsidian), x86_64, AMD EPYC 7R32, Python 3.13.4
See https://gist.github.com/bedroge/c278348ee6c41e1ecb7bcc15e2c61273 for a full test report.

edit: this was a build on top of EESSI 2025.06.

@bedroge
Copy link
Contributor

bedroge commented Dec 6, 2025

Test report by @bedroge
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3823
SUCCESS
Build succeeded for 1 out of 1 (total: 1 hour 33 mins 53 secs) (1 easyconfigs in total)
il-c04 - Linux Rocky Linux 9.5, x86_64, AMD EPYC 7542 32-Core Processor, 1 x AMD Arcturus GL-XL [Instinct MI100] (model: 0x738c, driver: "6.8.5"), 1 x AMD Instinct MI210 (model: 0x740f, driver: "6.8.5"), Python 3.9.21
See https://gist.github.com/bedroge/15b9d95caf8d189d182118f7dad431ea for a full test report.

edit: this was with EB-develop.

@Thyre
Copy link
Collaborator Author

Thyre commented Jan 7, 2026

@boegelbot please test @ jsc-zen3
EB_ARGS="--include-easyblocks-from-pr=3823,4037 --amdgcn-capabilities=gfx90a --installpath=/tmp/$USER/pr23304"

@boegelbot
Copy link
Collaborator

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=23304 EB_ARGS="--include-easyblocks-from-pr=3823,4037 --amdgcn-capabilities=gfx90a --installpath=/tmp/$USER/pr23304" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_23304 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 9345

Test results coming soon (I hope)...

Details

- notification for comment with ID 3719122925 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Collaborator

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3823, easybuilders/easybuild-easyblocks#4037
SUCCESS
Build succeeded for 1 out of 1 (total: 5 hours 55 mins 3 secs) (1 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.7, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.23
See https://gist.github.com/boegelbot/14e5a7c8d16cf0c2a8912f4bed7d8cf6 for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jan 21, 2026

@boegelbot please test @ jsc-zen3
EB_ARGS="--include-easyblocks-from-pr=3823 --amdgcn-capabilities=gfx90a --installpath=/tmp/$USER/pr23304"

@boegelbot
Copy link
Collaborator

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=23304 EB_ARGS="--include-easyblocks-from-pr=3823 --amdgcn-capabilities=gfx90a --installpath=/tmp/$USER/pr23304" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_23304 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 9456

Test results coming soon (I hope)...

Details

- notification for comment with ID 3777377265 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Collaborator

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3823
SUCCESS
Build succeeded for 1 out of 1 (total: 5 hours 56 mins 42 secs) (1 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.7, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.23
See https://gist.github.com/boegelbot/965c550879504e7bf968fb11fb983d88 for a full test report.

bedroge and others added 3 commits January 22, 2026 10:31
Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>
Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>
@Thyre
Copy link
Collaborator Author

Thyre commented Jan 22, 2026

Test report by @Thyre
SUCCESS
Build succeeded for 1 out of 1 (total: 4 hours 22 mins 58 secs) (1 easyconfigs in total)
jrc0850.jureca - Linux Rocky Linux 9.6 (Blue Onyx), x86_64, AMD EPYC 7443 24-Core Processor (zen3), 8 x AMD AMD Instinct MI250X / MI250 (model: 0x740c, driver: "6.12.12"), Python 3.9.21
See https://gist.github.com/Thyre/d4a5c4950c2f49e177e3a8d2c3dcfe41 for a full test report.

@bedroge bedroge modified the milestones: 5.x, next release (5.2.1?) Jan 23, 2026
Copy link
Contributor

@bedroge bedroge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm! 🎉

@bedroge
Copy link
Contributor

bedroge commented Jan 23, 2026

Going in, thanks @Thyre!

@bedroge bedroge merged commit 32e3ea7 into easybuilders:develop Jan 23, 2026
8 checks passed
@Thyre
Copy link
Collaborator Author

Thyre commented Jan 23, 2026

Lets do a final, persistent build.

@boegelbot please test @ jsc-zen3
EB_ARGS="--amdgcn-capabilities=gfx90a"

@boegelbot
Copy link
Collaborator

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=23304 EB_ARGS="--amdgcn-capabilities=gfx90a" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_23304 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 9475

Test results coming soon (I hope)...

Details

- notification for comment with ID 3791054979 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Collaborator

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (total: 5 hours 59 mins 25 secs) (1 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.7, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.23
See https://gist.github.com/boegelbot/a7368c82ff81207ae94888bdd9704841 for a full test report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2025a issues & PRs related to 2025a common toolchains new

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants