Skip to content

enhance LAMMPS easyblock to use FFT_KOKKOS configure option#3784

Merged
laraPPr merged 11 commits intoeasybuilders:developfrom
lorisercole:fix/lammps-kokkos-fft
Aug 20, 2025
Merged

enhance LAMMPS easyblock to use FFT_KOKKOS configure option#3784
laraPPr merged 11 commits intoeasybuilders:developfrom
lorisercole:fix/lammps-kokkos-fft

Conversation

@lorisercole
Copy link
Contributor

From version 29Aug2024 it is possible to specify the FFT library used in Kokkos by the Cmake setting FFT_KOKKOS.

LAMMPS is shipped with the internal KISS FFT library, which is selected by default.
However, for CUDA builds, messages appear at runtime advising the user to use cuFFT for better performance.
For CPU-only builds, it is also advised to use FFTW or MKL, if available.

From v `29Aug2024` it is possible to specify the FFT library used in Kokkos by the Cmake setting `FFT_KOKKOS`.
LAMMPS is shipped with the internal KISS FFT library, which is selected by default.
However, for CUDA builds, messages appear at runtime advising the user to use `cuFFT` for better performance.
For CPU-only builds, it is also advised to use FFTW or MKL, if available.
@boegel boegel added this to the release after 5.1.1 milestone Jul 2, 2025
@boegel
Copy link
Member

boegel commented Jul 2, 2025

This should be thoroughly tested to make sure we don't run into surprises...

@laraPPr Are you up for taking a look at this?

@boegel boegel changed the title LAMMPS: select best performing FFT lib enhance LAMMPS easyblock to select best performing FFT lib Jul 2, 2025
@boegel boegel changed the title enhance LAMMPS easyblock to select best performing FFT lib enhance LAMMPS easyblock to select best performing FFT library Jul 2, 2025
@laraPPr
Copy link
Contributor

laraPPr commented Jul 2, 2025

Yes I'll tackle this

@laraPPr
Copy link
Contributor

laraPPr commented Jul 3, 2025

@lorisercole
Copy link
Contributor Author

@laraPPr
Copy link
Contributor

laraPPr commented Jul 9, 2025

Test report by @laraPPr

Overview of tested easyconfigs (in order)

  • SUCCESS LAMMPS-29Aug2024-foss-2023b-kokkos.eb
  • SUCCESS archspec-0.2.5-GCCcore-12.3.0.eb
  • SUCCESS archspec-0.2.5-GCCcore-13.3.0.eb
  • SUCCESS LAMMPS-2Aug2023_update2-foss-2023a-kokkos.eb
  • SUCCESS LAMMPS-29Aug2024_update2-foss-2023a-kokkos.eb
  • SUCCESS Voro++-0.4.6-GCCcore-13.3.0.eb
  • SUCCESS ffnvcodec-12.2.72.0.eb
  • SUCCESS kim-api-2.4.1-GCC-13.3.0.eb
  • SUCCESS x264-20240513-GCCcore-13.3.0.eb
  • SUCCESS PLUMED-2.9.3-foss-2024a.eb
  • SUCCESS SDL2-2.30.6-GCCcore-13.3.0.eb
  • SUCCESS Yasm-1.3.0-GCCcore-13.3.0.eb
  • SUCCESS x265-3.6-GCCcore-13.3.0.eb
  • SUCCESS FFmpeg-7.0.2-GCCcore-13.3.0.eb
  • SUCCESS MDI-1.4.26-gompi-2024a.eb
  • SUCCESS ScaFaCoS-1.0.4-foss-2024a.eb
  • SUCCESS re2c-3.1-GCCcore-13.3.0.eb
  • SUCCESS assimp-5.4.3-GCCcore-13.3.0.eb
  • SUCCESS DBus-1.15.8-GCCcore-13.3.0.eb
  • SUCCESS NSPR-4.35-GCCcore-13.3.0.eb
  • SUCCESS NSS-3.104-GCCcore-13.3.0.eb
  • FAIL (build issue) Qt6-6.7.2-GCCcore-13.3.0.eb (partial log available at https://gist.github.com/laraPPr/61fa2c25d4b7d807e42694c687ae5fce)
  • FAIL (build issue) VTK-9.3.1-foss-2024a.eb (partial log available at https://gist.github.com/laraPPr/43cef140840366be6f87c726af392d0c)
  • FAIL (build issue) LAMMPS-29Aug2024_update2-foss-2024a-kokkos.eb (partial log available at https://gist.github.com/laraPPr/35d51055421d0bd84afc5b7a7faf0cb9)

Build succeeded for 21 out of 24 (4 easyconfigs in total)
node4210.shinx.os - Linux RHEL 9.4 (Plow), x86_64, AMD EPYC 9654 96-Core Processor, Python 3.9.18
See https://gist.github.com/laraPPr/9e6be7caf2e9d3ab0ab26a3f7c628c11 for a full test report.

@lorisercole
Copy link
Contributor Author

Hi @laraPPr, I actually have a greater update about using Kokkos that I'd like to include in this PR.

I discovered that with the KOKKOS_ARCH_{processor_arch} flag, Kokkos' CMake will add the correspondent -march and -mtune flags to the compilation line. Generally, these will override the -march=native flag that is added by EasyBuild, and I don't think it's the behavior we desire. Note that these flags will be used in compiling most of LAMMPS' C++ files, and not only for the Kokkos-related modules.
In principle this shouldn't cause big issues, but I've seen some weird warnings while trying to write an easyconfig for LAMMPS with the intel toolchain (that I'll publish soon).

Since version 2Aug2023, LAMMPS/KOKKOS supports the KOKKOS_ARCH_NATIVE flag, that will use -march=native as expected. I'm suggesting to switch to this by default, without the need for more complex interventions.

@lorisercole lorisercole changed the title enhance LAMMPS easyblock to select best performing FFT library enhance LAMMPS easyblock Jul 9, 2025
This reverts commit 00f237b.
It is not needed as Easybuild's wrappers already take care or using the
correct rpath-ing flags. LAMMPS Cmake option is not useful and does not
work properly.
@lorisercole
Copy link
Contributor Author

lorisercole commented Aug 11, 2025

I finished testing this PR, now I'm done.
I reverted commit 00f237b: the flag -DLAMMPS_INSTALL_RPATH=ON does not actually change the build/link behavior. It only causes this flag to be added to the link command:
-Wl,-rpath,:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
so it does not seem to work properly.
Anyway, Easybuild's RPATH wrappers already take care of adding the correct rpath-ing flags. With easybuilders/easybuild-framework#4970 they will also work correctly for Intel compilers.

@laraPPr
Copy link
Contributor

laraPPr commented Aug 14, 2025

@boegelbot please test @ jsc-zen3 EB_ARGS="LAMMPS-2Aug2023_update2-foss-2023a-kokkos.eb LAMMPS-23Jun2022-foss-2021a-kokkos.eb LAMMPS-23Jun2022-foss-2021b-kokkos.eb LAMMPS-23Jun2022-foss-2022a-kokkos.eb LAMMPS-29Aug2024-foss-2023b-kokkos.eb LAMMPS-29Aug2024_update2-foss-2023a-kokkos.eb LAMMPS-29Aug2024_update2-foss-2024a-kokkos.eb"

@boegelbot
Copy link

@laraPPr: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3784 EB_ARGS="LAMMPS-2Aug2023_update2-foss-2023a-kokkos.eb LAMMPS-23Jun2022-foss-2021a-kokkos.eb LAMMPS-23Jun2022-foss-2021b-kokkos.eb LAMMPS-23Jun2022-foss-2022a-kokkos.eb LAMMPS-29Aug2024-foss-2023b-kokkos.eb LAMMPS-29Aug2024_update2-foss-2023a-kokkos.eb LAMMPS-29Aug2024_update2-foss-2024a-kokkos.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3784 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 7589

Test results coming soon (I hope)...

Details

- notification for comment with ID 3187765122 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@laraPPr
Copy link
Contributor

laraPPr commented Aug 14, 2025

Test report by @laraPPr

Overview of tested easyconfigs (in order)

Build succeeded for 3 out of 4 (4 easyconfigs in total)
node3520.doduo.os - Linux RHEL 9.4 (Plow), x86_64, AMD EPYC 7552 48-Core Processor, Python 3.9.18
See https://gist.github.com/laraPPr/2e043567582869b8749b23685a6a5705 for a full test report.

@laraPPr
Copy link
Contributor

laraPPr commented Aug 14, 2025

LAMMPS-29Aug2024_update2-foss-2024a-kokkos.eb failed because of missing dependencies. restarted it now but should be ok

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS LAMMPS-2Aug2023_update2-foss-2023a-kokkos.eb
  • SUCCESS LAMMPS-23Jun2022-foss-2022a-kokkos.eb
  • SUCCESS LAMMPS-29Aug2024-foss-2023b-kokkos.eb
  • SUCCESS LAMMPS-29Aug2024_update2-foss-2023a-kokkos.eb
  • SUCCESS kim-api-2.2.1-GCC-10.3.0.eb
  • SUCCESS kim-api-2.3.0-GCC-11.2.0.eb
  • SUCCESS archspec-0.2.5-GCCcore-13.3.0.eb
  • SUCCESS ScaFaCoS-1.0.1-foss-2021a.eb
  • SUCCESS VTK-9.1.0-foss-2021b.eb
  • SUCCESS kim-api-2.4.1-GCC-13.3.0.eb
  • SUCCESS LAMMPS-23Jun2022-foss-2021a-kokkos.eb
  • SUCCESS LAMMPS-23Jun2022-foss-2021b-kokkos.eb
  • SUCCESS MDI-1.4.26-gompi-2024a.eb
  • SUCCESS ScaFaCoS-1.0.4-foss-2024a.eb
  • SUCCESS LAMMPS-29Aug2024_update2-foss-2024a-kokkos.eb

Build succeeded for 15 out of 15 (7 easyconfigs in total)
jsczen3c2.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/be886e514e366498920f293795c7da03 for a full test report.

@laraPPr
Copy link
Contributor

laraPPr commented Aug 14, 2025

@boegelbot please test @ jsc-zen3-a100 EB_ARGS="LAMMPS-2Aug2023_update2-foss-2023a-kokkos-CUDA-12.1.1.eb LAMMPS-2Aug2023_update2-foss-2024a-kokkos-CUDA-12.6.0.eb LAMMPS-23Jun2022-foss-2021a-kokkos-CUDA-11.3.1.eb LAMMPS-23Jun2022-foss-2021b-kokkos-CUDA-11.4.1.eb LAMMPS-28Oct2024-foss-2023a-kokkos-mace-CUDA-12.1.1.eb LAMMPS-29Aug2024_update2-foss-2023b-kokkos-CUDA-12.4.0.eb LAMMPS-29Aug2024_update2-foss-2024a-kokkos-CUDA-12.6.0.eb"

@boegelbot
Copy link

@laraPPr: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3784 EB_ARGS="LAMMPS-2Aug2023_update2-foss-2023a-kokkos-CUDA-12.1.1.eb LAMMPS-2Aug2023_update2-foss-2024a-kokkos-CUDA-12.6.0.eb LAMMPS-23Jun2022-foss-2021a-kokkos-CUDA-11.3.1.eb LAMMPS-23Jun2022-foss-2021b-kokkos-CUDA-11.4.1.eb LAMMPS-28Oct2024-foss-2023a-kokkos-mace-CUDA-12.1.1.eb LAMMPS-29Aug2024_update2-foss-2023b-kokkos-CUDA-12.4.0.eb LAMMPS-29Aug2024_update2-foss-2024a-kokkos-CUDA-12.6.0.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3784 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 7594

Test results coming soon (I hope)...

Details

- notification for comment with ID 3188426479 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

Build succeeded for 3 out of 8 (7 easyconfigs in total)
jsczen3c2.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/6c643a77c35fd1134e615e157b3cfa72 for a full test report.

@laraPPr
Copy link
Contributor

laraPPr commented Aug 18, 2025

Test report by @laraPPr

Overview of tested easyconfigs (in order)

  • SUCCESS VTK-9.3.1-foss-2024a.eb
  • SUCCESS LAMMPS-29Aug2024_update2-foss-2024a-kokkos.eb

Build succeeded for 2 out of 2 (1 easyconfigs in total)
node3514.doduo.os - Linux RHEL 9.4 (Plow), x86_64, AMD EPYC 7552 48-Core Processor, Python 3.9.18
See https://gist.github.com/laraPPr/12958a7ad1a243cfd2e698a83c03c7ce for a full test report.

@laraPPr
Copy link
Contributor

laraPPr commented Aug 18, 2025

CPU builds seem ok but the CUDA builds ran in all kinds of trouble. Am now looking at LAMMPS-2Aug2023_update2-foss-2024a-kokkos-CUDA-12.6.0.eb

@lorisercole
Copy link
Contributor Author

I wonder, do these testing nodes have CUDA drivers installed? Do you normally use them to build CUDA software?

@laraPPr
Copy link
Contributor

laraPPr commented Aug 18, 2025

Yes it was build on the test system at JSC (jsc-zen3-a100) where we test all GPU builds. So it should be.

@laraPPr
Copy link
Contributor

laraPPr commented Aug 18, 2025

Test report by @laraPPr

Overview of tested easyconfigs (in order)

  • SUCCESS LAMMPS-29Aug2024_update2-foss-2024a-kokkos.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node4007.donphan.os - Linux RHEL 9.4 (Plow), x86_64, Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz (cascadelake), 1 x NVIDIA NVIDIA A2, 570.133.20, Python 3.11.3
See https://gist.github.com/laraPPr/5989ec2c610b11517e72a459b38816f7 for a full test report.

@Thyre
Copy link
Collaborator

Thyre commented Aug 18, 2025

The boegelbot results look weird. jsczen3c2.int.jsc-zen3.fz-juelich.de is not the GPU node.
That should be jsczen3g1.int.jsc-zen3.fz-juelich.de.

See e.g.: easybuilders/easybuild-easyconfigs#23653 (comment)

I think you managed to find a bug here.

@laraPPr
Copy link
Contributor

laraPPr commented Aug 19, 2025

I'm gonna ignore the failure of LAMMPS-28Oct2024-foss-2023a-kokkos-mace-CUDA-12.1.1.eb this uses the nvcc-wrapper so is failing with missing RPATH as well. Perhaps this will also be solved by easybuilders/easybuild-framework#4981 but I don't think we should delay this pr on that one.

@laraPPr
Copy link
Contributor

laraPPr commented Aug 19, 2025

Ok let me retrigger this one to see if it submits to the right system
@boegelbot please test @ jsc-zen3-a100 EB_ARGS="LAMMPS-2Aug2023_update2-foss-2024a-kokkos-CUDA-12.6.0.eb"

@Thyre
Copy link
Collaborator

Thyre commented Aug 19, 2025

Ok let me retrigger this one to see if it submits to the right system boegelbot please test @ jsc-zen3-a100 EB_ARGS="LAMMPS-2Aug2023_update2-foss-2024a-kokkos-CUDA-12.6.0.eb"

I've pinged Kenneth and Sebastian on this yesterday evening.
Looks like the script boegelbot uses for the EasyBlock repo is missing the GPU stuff. Tests from the EasyConfig repo are correct.

This requires a small change in the shell script used for it.

@boegelbot
Copy link

@laraPPr: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3784 EB_ARGS="LAMMPS-2Aug2023_update2-foss-2024a-kokkos-CUDA-12.6.0.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3784 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 7707

Test results coming soon (I hope)...

Details

- notification for comment with ID 3199602034 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@laraPPr
Copy link
Contributor

laraPPr commented Aug 19, 2025

Ok will test on our clusters in Gent in the mean time

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/d2603e2bbdc87ad8b0b4131c82b5415b for a full test report.

@laraPPr
Copy link
Contributor

laraPPr commented Aug 19, 2025

Test report by @laraPPr

Overview of tested easyconfigs (in order)

  • SUCCESS LAMMPS-2Aug2023_update2-foss-2023a-kokkos-CUDA-12.1.1.eb
  • SUCCESS LAMMPS-2Aug2023_update2-foss-2024a-kokkos-CUDA-12.6.0.eb
  • SUCCESS LAMMPS-29Aug2024_update2-foss-2023b-kokkos-CUDA-12.4.0.eb
  • SUCCESS LAMMPS-29Aug2024_update2-foss-2024a-kokkos-CUDA-12.6.0.eb

Build succeeded for 4 out of 4 (4 easyconfigs in total)
node3904.accelgor.os - Linux RHEL 9.4 (Plow), x86_64, AMD EPYC 7413 24-Core Processor, 1 x NVIDIA NVIDIA A100-SXM4-80GB, 570.133.20, Python 3.9.18
See https://gist.github.com/laraPPr/0ff2de22772ed52f991ca80d348e5294 for a full test report.

@laraPPr
Copy link
Contributor

laraPPr commented Aug 19, 2025

Test report by @laraPPr

Overview of tested easyconfigs (in order)

  • SUCCESS LAMMPS-2Aug2023_update2-foss-2023a-kokkos-CUDA-12.1.1.eb
  • SUCCESS LAMMPS-2Aug2023_update2-foss-2024a-kokkos-CUDA-12.6.0.eb
  • SUCCESS LAMMPS-29Aug2024_update2-foss-2023b-kokkos-CUDA-12.4.0.eb
  • SUCCESS LAMMPS-29Aug2024_update2-foss-2024a-kokkos-CUDA-12.6.0.eb

Build succeeded for 4 out of 4 (4 easyconfigs in total)
node4303.litleo.os - Linux RHEL 9.4 (Plow), x86_64, AMD EPYC 9454P 48-Core Processor, 1 x NVIDIA NVIDIA H100 NVL, 570.133.20, Python 3.9.18
See https://gist.github.com/laraPPr/66ea0ee53ee44abd489d7dfa684eb1fa for a full test report.

@laraPPr
Copy link
Contributor

laraPPr commented Aug 20, 2025

Test report by @laraPPr

Overview of tested easyconfigs (in order)

  • SUCCESS LAMMPS-29Aug2024_update2-foss-2024a-kokkos-CUDA-12.6.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
node3303.joltik.os - Linux RHEL 9.4 (Plow), x86_64, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz (cascadelake), 1 x NVIDIA Tesla V100-SXM2-32GB, 570.133.20, Python 3.11.3
See https://gist.github.com/laraPPr/bb50ccc46dbc919b0a12c31efab4e709 for a full test report.

Copy link
Contributor

@laraPPr laraPPr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@laraPPr
Copy link
Contributor

laraPPr commented Aug 20, 2025

Going in, thanks @lorisercole!

@laraPPr laraPPr merged commit 2c400f0 into easybuilders:develop Aug 20, 2025
17 checks passed
@Thyre
Copy link
Collaborator

Thyre commented Aug 20, 2025

@boegelbot please test @ jsc-zen3-a100 EB_ARGS="LAMMPS-2Aug2023_update2-foss-2024a-kokkos-CUDA-12.6.0.eb"

@boegelbot
Copy link

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3784 EB_ARGS="LAMMPS-2Aug2023_update2-foss-2024a-kokkos-CUDA-12.6.0.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3784 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 7735

Test results coming soon (I hope)...

Details

- notification for comment with ID 3207096727 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 1 (1 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/d6572f40acb38362aaa3e2e0e16894d3 for a full test report.

@boegel boegel changed the title enhance LAMMPS easyblock enhance LAMMPS easyblock to use FFT_KOKKOS configure option Sep 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants