EPIC: Initial TriBITS Modernization and Generalization

[bartlettroscoe]

Description

This Epic will plan for and track the execution of the TriBITS modernization effort to address the immediate needs of Trilinos, Kokkos and the ECP Spack ecosystem. The goal is to do this in a way that will provide as much backward compatibility as reasonably possible to avoid having to rewrite thousands of CMakeLists.txt and *.cmake files in Trilinos, Drekar, Charon2, CASL VERA and other projects using TriBITS or breaking external customers using Trilinos as much as is reasonable.

Refactoring plan

• EPIC: Merge TriBITS concepts and implementation of Packages and TPLs #63: Rename branch ‘63-merge-tpls-packages’ to ‘63-merge-tpls-packages-1’, do some clean-up to improve understanding without changing any behavior (knowing it is incomplete) (Merge to ‘master’ and snapshot to Trilinos ‘develop’.)
• EPIC: Merge TriBITS concepts and implementation of Packages and TPLs #63: Branch ‘63-merge-tpls-packages-2' (PR Second refactoring cleaning up to get ready for #63 #374) for more cleanup of code building dependency graph.
• Learnings from the VTK Module Upgrade to modern CMake #337: Extract whatever info/advice we can get out of the similar VTK Module upgrade to modern CMake.
• Change from upper-case SOME_FUNCTION_NAME() to lower-cases some_function_name() #274: Create tool to make all macro, function, and command names and calls lower case and apply to TriBITS 'master' branch. (This will make the future refactorings easier by making the code easier to read.)
• Dropping support for Makefile.export.* files with TriBITS/Trilinos refactoring to support modern CMake features trilinos/Trilinos#8498: Announce and then remove support for Makefile.export.* files and push to Trilinos ‘develop’.
• EPIC: Merge TriBITS concepts and implementation of Packages and TPLs #63: Provide feedback on TriBITS package dependencies data-structures #377
• Split TribitsDevelopersGuide.* into TribitsUsersGuide.* and TribitsMaintainersGuide.* #381: Split TribitsDevelopersGuide.* into TribitsUsersGuide.* and TribitsMaintainersGuide.* (in support of EPIC: Merge TriBITS concepts and implementation of Packages and TPLs #63 and Move to modern CMake target-based propagation of build information #299).
• Set up TriBITS testing using GitHub Actions #363: Have full GitHub Actions testing set up and posting to CDash for all needed use cases.
• Refactoring target for TriBITS and packages using TriBITS #342: Refine and agree to initial targets for the refactoring.
• Move to modern CMake target-based propagation of build information #299: Refactor the guts of TriBITS to use modern CMake target-based propagation of everything instead of variables. (Merge to ‘master’ and snapshot to Trilinos ‘develop’.)
• EPIC: Merge TriBITS concepts and implementation of Packages and TPLs #63: Finish up basics of treating TriBITS package and TPLs as just “packages” and provide a few examples/tests in TriBITS itself (Use Case 3: Configure/build pointing to a subset of already installed TriBITS packages in same repo). (Merge to ‘master’ and snapshot to Trilinos ‘develop’.)
• Refactor Trilinos (and perhaps Kokkos some) to allow pre-building and installing Kokkos and the pointing to that installed Kokkos (ignoring packages/kokkos) when building the rest of Trilinos. (Merge to Trilinos ‘develop’) (see Build and install downstream Trilinos packages against pre-installed Kokkos using native CMake build system trilinos/Trilinos#11545)
• Update Spack 'develop' Trilinos package.py file to depend on kokkos version 4.1.00 (4.1.0) when using Trilinos 'develop' (and all future releases of Trilinos starting with 14.4) ... See [email protected] +kokkos: use external kokkos spack/spack#39397
• Refactor Trilinos (and perhaps Kokkos and KokkosKernels some) to allow pre-building and installing Kokkos and KokkosKernels and the pointing to those installed Kokkos and KokkosKernels when building the rest of Trilinos (see Build and install downstream Trilinos packages against pre-installed KokkosKernels and SEACAS using their native CMake build systems trilinos/Trilinos#12002) ... See KokkosKernelsConfig.cmake: add all_libs target and necessary aliases kokkos/kokkos-kernels#2020
• [ ] Refactor the independent SEACAS TriBITS project to pull in pre-installed Kokkos and Zoltan packages (will allow separate Kokkos, KokkosKernels, SEACAS and remaining Trilinos packages to co-exist as Spack packages and linked into single downstream packages and APPs). ... Descoped (see below)
• Address other refactorings and cleanups as they are exposed (as funding and time permits) ... Has been deployed for more than a year and several Trilinos and Spack releases!

[bartlettroscoe]

CC: @keitat

FYI: The ExaWind project has specifically asked to have Trilinos build against pre-installed Kokkos before SC 2021. Given the plan above, we should have this done by the end of FY21.

[bartlettroscoe]

CC: @keita, @fnrizzi, @marcinwrobel1986, @MikolajZuzek

The merge of the branch '63-merge-tpls-packages-1' to TriBITS 'master' is now complete (see #63 (comment)). I can now continue with some much more aggressive refactorings towards #63 and #299. Next, I will do some more clean up refactorings to improve understanding of the code base to make the next steps easier.

[bartlettroscoe]

FYI: I talked with @gsjaardema about this plan and added the sub-story above:

• Refactor the independent SEACAS TriBITS project to pull in pre-installed Kokkos and Zoltan packages (will allow separate Kokkos, KokkosKernels, SEACAS and remaining Trilinos packages to co-exist as Spack packages and linked into single downstream packages and APPs)

That should be easy once we can build Trilinos against pre-installed Kokkos and KokkosKernels.

[keitat]

xSDK members are looking for a way to import Kokkos' build information. They need optimization flags and compiler (nvcc_wrapper for CUDA build) information.

[bartlettroscoe]

xSDK members are looking for a way to import Kokkos' build information. They need optimization flags and compiler (nvcc_wrapper for CUDA build) information.

@keitat, from Spack? This is not a Spack-related epic. This epic will enable more flexibility with Spack but does not directly address Spack packages.

[bartlettroscoe]

FYI: The move to modern CMake targets for TriBITS and Trilinos is complete. See #299 (comment).

[bartlettroscoe]

FYI:

We have a demonstration ready for pre-building/installing Kokkos, then KokkosKernels, then the rest of Trilinos. This demos the building and installing Kokkos and KokkosKernels separately as their own Spack packages and then building the rest of Trilinos as its own Spack package (and similar use cases). (And we can further partition Trilinos into smaller Spack packages in the future as we would like starting with SEACAS.)

All of the critical changes on the TriBITS side should be complete. The question is what to do about the current incompatibilities between the TriBITS builds of Kokkos and KokkosKernels and the native non-TriBITS CMake builds for those packages and what is expected by downstream Trilinos packages. (Changes can be made on either side to accommodate some of the differences.)

[bartlettroscoe]

FYI, with the merge of PR:

• Get pre-installed Kokkos and Trilinos working with native Kokkos non-TriBITS CMake build system (#11545)  trilinos/Trilinos#11863

Trilinos 'develop' and Kokkos 'develop' now support pre-isntalling Kokkos with the native Kokkos non-TriBITS CMake build system.

[bartlettroscoe]

FYI: Here is the Spack PR that has the Trilinos Spack package using the Kokkos Spack package:

• [email protected] +kokkos: use external kokkos spack/spack#39397

[bartlettroscoe]

FYI: Looks like the changes to KokkosKernels was made to allow it to be installed using its native non-TriBITS CMake build system. See:

• KokkosKernelsConfig.cmake: add all_libs target and necessary aliases kokkos/kokkos-kernels#2020

[bartlettroscoe]

I am going to descope to remove the SEACAS work (but I think has been implicitly done by SEACAS developers). Looking at:

• https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/seacas/package.py

it does not looks like the SEACAS Spack package has yet been updated to use external Kokkos or Zoltan, but someone added a ToDo for this a long time ago:

• spack/spack@ccce3b7

I don't think that is actually hard to do at this point, and SEACAS has already been updated for external Pamgen (see here and the commit sandialabs/seacas@2d4c7ce).

[bartlettroscoe]

Closing as complete (which should have been done a long time ago ...)

[bartlettroscoe]

Closing as complete (which should have been done a long time ago ...)

[bartlettroscoe]

FYI: Someone updated the Spack build of Trilinos to use external KokkosKernels last week. See:

• Kokkos complex_align variant, Trilinos+PETSc enforcement for Kokkos~complex_align spack/spack#47686 (comment)