Build systems

[christoph-conrads]

As of today, there are three different build systems in LAPACK:

• Makefiles-only,
• CMake, and
• Meson.

The Meson build is incomplete and unmaintained since its introduction in January 2019. The CMake and the Makefile-only build are not identical: The -frecursive flag is missing. Having more than one build system has at least three effects:

• It increases the amount of work,
• the issues Segmentation fault when testing xeigtstz #276 and Segfault in ./EIG/xeigtstz #335 do not occur when building with CMake but
• only the CMake-based builds may crash when LAPACK is called concurrently, see the comment about the need for -frecursive in PR #486.

Especially the last point is great because even if someone delivers an accurate problem report, it is impossible to retrace the issue when you built with Makefiles.

I strongly suggest to stick with one build system.

[martin-frbg]

To add to the list, CMake builds currently (appear to) support the option to build only select parts of the library (REAL, DOUBLE, COMPLEX and/or DOUBLE COMPLEX), while Makefile-only builds do not. (I have changed the Makefiles in the copy we distribute with OpenBLAS and could produce a PR if desired - would need to rediff to leave out some changes not relevant here)

[christoph-conrads]

CMake builds currently (appear to) support the option to build only select parts of the library (REAL, DOUBLE, COMPLEX and/or DOUBLE COMPLEX)

This option worked the last time I tried it (around May 2020).

The CMake build has several options, I list the most important ones for me here:

• building shared or static libraries
• enabled or disabling tests
• 64- or 32-bit indices
• optimized and debugging builds
• options for matrix generator library (TMG), usage of other BLAS libraries, BLAS++, LAPACK++...

[thijssteel]

Good idea. Builds working on my machine with make but not on the ci with CMake has driven me crazy a few times.

Technically, i think the Makefile also supports a lot of those options (you can just edit your make.inc). If you want a build without the tests you can build with make lib. But crucially, building a shared library is not supported. Most people use LAPACK as shared library so that makes CMake the clear winner for me.

If we end up going for it, I suggest we add some extra build instructions to the readme. How to build and run the tests, how to add flags, especially the frecursive flag. Perhaps even some small note on how to use the shared library.

[martin-frbg]

I suspect there will be many patches floating around already (e.g. in linux distributions that package lapack) to build shared libraries with gmake. Should not be too complicated to add a BUILD_SHARED option to make.inc like the BUILD_DEPRECATED that is already there (and both are pretty obscure in the CMake build unless one already knows that ccmake and cmake-gui exist).

[christoph-conrads]

Should not be too complicated to add a BUILD_SHARED option to make.inc like the BUILD_DEPRECATED that is already there (and both are pretty obscure in the CMake build unless one already knows that ccmake and cmake-gui exist).

There is no need to work with ccmake to set any of these options. You can use the command line similar to the configure script generated by autotools:

$ cmake -DBUILD_SHARED_LIBS=ON -DBUILD_COMPLEX=OFF -DBUILD_DEPRECATED=ON -- ../lapack/
[snip]
-- Build deprecated routines: ON
-- Build single precision real: ON
-- Build double precision real: ON
-- Build single precision complex: OFF
-- Build double precision complex: ON
[snip]
$ git -C ~/lapack rev-parse HEAD
6e125a41a7d4905d905a7467d3239d3f0d14b22c

[martin-frbg]

You certainly can, my point is that there is no obvious documentation except reading the CMakelists.txt (whereas the README points one to the preconfigured make.inc files for gmake). ccmake and cmake-gui show a list of available options, but this will be lost on anyone new to cmake.

[langou]

If we write a documentation on how to use CMAKE, we can use me as the guinea pig user since I have never used CMAKE.

Reading the whole conversation, I feel that the Makefile is in LAPACK there only for me. ;)

Well, I do not know.

Yes: maintaining several build system leads to inconsistencies. So I see the argument to only have one. (Whether it be CMAKE, make or meson.) (Oh yes, I have never used meson either, if you wonder.)

I am worried about moving to CMAKE because, not only do I not know how to use it, I do not know how maintain it. However it seems like we have a wonderful community who can help, so me not being able to work on this part of the project does not seem an issue, in addition, I probably should learn CMAKE.

All: Please express your opinion in this thread.

It seems that most of us would like to (1) remove make and remove meson; (2) move to CMAKE only; (3) add some good documentation on how to use CMAKE. Fine with me.

I am going to send a few emails to other package maintainers here and there to ask how they are doing and what their opinion on this issue is.

@martin-frbg: how is this done in OpenBLAS?

[martin-frbg]

OpenBLAS has Makefiles as its "traditional" build system that is "known to work", and CMAKE as an originally user-contributed alternative that still spits out a warning that the files it generates may not correspond exactly to what a Makefile build would do.
I do not like CMAKE much, but I have learned to create working (though perhaps sometimes unorthodox) cmake build files. In my opinion, the Makefiles should remain (and I believe gmake is still more widespread than cmake). I do not know the history of the meson support - if this was a one-time "thrown over the fence" contribution that nobody knows or cares enough to keep updated, I guess there is little point in keeping it around.

[christoph-conrads]

I do not know the history of the meson support - if this was a one-time "thrown over the fence" contribution that nobody knows or cares enough to keep updated, I guess there is little point in keeping it around.

The Meson build was parachuted into LAPACK in PR #311.

[martin-frbg]

Hm. If it got accepted into 3.9.0 (even if just as a proof of concept), it would look a bit strange to throw it out again with the very next release...

[langou]

I contacted @therault so that he can give us his opinion, here is what @therault says. Thanks @therault!

Open MPI uses only autoconf (automake, autoconf, configure, Makefiles).

For PaRSEC and DPLASMA, we only use CMake, and to help users who are used to configure, @abouteiller made a script that uses the syntax of configure and calls CMake with its own syntax.

I know of one major project that tried to maintain both configure and CMake. It slowly evolved in a situation where 100% of the features were available in CMake, and less and less new features were supported in configure. I believe that they have stopped maintaining the configure version.

Maintaining two build systems for a code is hard: in theory, the build system should work with any code, and thus maintaining two should be possible (with a high cost for the maintainers, to keep them in sync, which is the reason why the configure was dropped in that other project, the maintainer decided he had better things to do with his time). In practice, it is sometimes necessary to adapt the code to the build system to make things cleaner. When that happens, one of the two build systems needs to use workarounds and hacks to behave as the other one and in accordance with the code.

In the end, CMake or configure, both will disappoint. We are programmers / mathematicians / we do algorithms... Spending time to understand why on that machine, with this combination of compilers and flags, the code doesn't compile properly, nobody likes that. Worse, trying to find how to do that thing or that other using CMake or autoconf will drive you up the wall :) Working with both, I can guarantee you will complain about both of them :) Maintaining a single one is a sure way to complain twice less.

Today, I'm in favor of CMake for a single reason: CMake does a good job at generating Ninja files instead of Makefiles. Ninja compiles PaRSEC twice to three times faster than Make -j. Now, maybe configure/autoconf can also generate Ninja files, I never tried... If you never tried ninja before, and if LAPACK has a CMake version, I suggest you try cmake -G Ninja (after installing Ninja on your machine). To compile, you call 'ninja' in the directory where you executed cmake instead of 'make'. You'll see, that's amazing :)

[abouteiller]

I am happy to contribute said configure-to-cmake glue script to LAPACK if that's something you are interested in.
(to clarify, that script is not autoconf/m4 based, it is a simple script that converts configure --with-feature=value into a call to cmake -DFEATURE=value, with a little more polish).

[martin-frbg]

LAPACK does not even use configure - and I agree it would probably be a pain to keep an autoconf/automake/configure-based system concurrent with cmake, with the autotools themselves constantly "evolving" and being dependent on m4 macros and whatnot.

[christoph-conrads]

I am worried about moving to CMAKE because, not only do I not know how to use it, I do not know how maintain it.

This is a major reason to stick to Makefiles.

All: Please express your opinion in this thread.

The Meson build should be removed. No one is using it, no one knows how to maintain it, the original author submitted an incomplete build, and never came back after his changes were merged. How many people knew there was a Meson build before I opened this issue?

CMake is a portable build system with simple syntax and fully configurable builds. One can set file-specific, compiler-specific, or target-specific options (e.g., to selectively enable functions that are not in the C standard library like gethostname()). CMake has a robust object file management; this allows you to keep typing make to update your builds even if you switched git branches. CMake interacts nicely with the rest of the UNIX ecosystem: you can call arbitrary programs and act on their output. For example, you can call pkg-config to locate other packages.

The first time I used CMake was in 2016 and I have used it continuously ever since to build code for Linux, Mac, iOS, and Android systems, on x86, x86-64, armv7, and aarch64 instruction set architectures. It is available on all major Linux distributions and on all supercomputers that I have access to (Grid 5000, Jean Zay, JUWELS, GRICAD, Eagle II). The supercomputers usually have multiple versions available including old releases and very recent ones (3.18+). The only platform where I struggled with CMake availability is CentOS 7 but you can always download a CMake tarball and use the bootstrap script inside the tarball to build CMake only with GNU Make.

CMake just works for me.

My experience with other build systems is as follows:

• Bazel: a huge memory hog, impossible to run on a Raspberry Pi with 1GB of memory, insists to build all dependencies, and link everything statically
• Autotools: as a developer no experience, as a user the configure script is very convenient because it can show you all available options (CMake does that with ccmake only after running cmake once).

[christoph-conrads]

Today, I'm in favor of CMake for a single reason: CMake does a good job at generating Ninja files instead of Makefiles. Ninja compiles PaRSEC twice to three times faster than Make -j.

Great, Ninja could not compile Fortran in the past, cf. ninja #1265, i.e., on some distributions this will not work (Debian 9?).

[zerothi]

I would definitely say regular makefiles are easier and sufficient in this case.
Maintaining software at a HPC cluster I am very disappointed with Cmake uses.

1. Developers have a hard time following standard naming schemes making building a hard time due to excessive documentation reading. Every package do some "tweak" of names that fits their projects. Thus ending up in non-standard naming schemes. (a bit true for autoconf+, but here things have standardised more)
2. When stuff breaks in cmake, it requires a cmake expert to debug it, I still have a very hard time debugging stuff... :(
3. Hard to use configuration files (no make.inc), this forces all bulild-options to be specified on the command-line at build-time.

On the other hand, cmake also does nice things:

1. Easier windows support
2. More automatic handling of dependencies and inclusion of cmake files for other projects (this is a nice benefit)

However, for LAPACK's very simple build system I don't think there is a good and bad. The current build system has worked for decades, and people are used to it.
Also, for fast parallel build, this could trivially be added to the current makefile system...

I don't really have a preference, but it should be absolutely clear which options are supported by which build-system if more are supported...

[thijssteel]

I feel like i'm a similar situation as you are @langou .

I don't know cmake. I only managed to use it for this project because all the configuration was already in place and by opening the travis config and copy pasting the lines it executes. I don't think its just because I'm inexperienced. A Makefile is a little less magical and feels more low level. That's something that likely resonates with the kind of people that tend to work on this project.
If purely for my personal use, I'd keep using gmake, but then we'd have to put all the features that are currently only available in cmake into the Makefile. Especially a shared library target.

However, i can't ignore all the nice features cmake seems to offer. Ultimately, i think either cmake or gmake will do fine. We just need to pick one.

[h-vetinari]

I am an an outside voice, but as someone who helps maintain lapack in conda-forge, having the build system be CMake would have several advantages (less hacky, native windows support, etc.).

It takes some getting used to the CMake syntax, but my impression is that it is a well-maintained and well-documented piece of software that has successfully solved / abstracted away a lot of gnarly build problems. I am involved in packaging other libraries as well, and (subjectively) see more and more of a move to CMake.

[ilayn]

Just another data point from outside; in our work which involves quite a bit of C/C++ building, we are moving to Meson after struggling too much with CMake. The long list of issues with it are well-documented elsewhere so I'm skipping those.

In SciPy project we will also try to use CMake or Meson at some point since Python distutils is being decommissioned.

[weslleyspereira]

I agree maintaining three build systems leads to inconsistencies and/or extra work for the maintainers. However, based on the comments above and in my opinion, each user has their own way to build the code. Particularly, I agree with @therault's comment:

We are programmers / mathematicians / we do algorithms... Spending time to understand why on that machine, with this combination of compilers and flags, the code doesn't compile properly, nobody likes that. Worse, trying to find how to do that thing or that other using CMake or autoconf will drive you up the wall :) Working with both, I can guarantee you will complain about both of them :) Maintaining a single one is a sure way to complain twice less.

One idea to circumvent the problem with multiple build systems is:

• Create more configurations in the CI so that it covers all supported build systems. This way we guarantee all build systems work with the basic features.
• Present all and suggest the use of one of the build systems in the README. Ex.: We suggest the use of CMake if you need to build shared and static libraries, ..., as mentioned by @martin-frbg, @christoph-conrads, and @thijssteel in this thread. (I didn't know there was the Meson build system on the repository. I think we should either add it in the README or stop maintaining this option.)

---

As far as I understand, the issue with the recursive flag has to do with the testing routines xeigtstz (see #335 (comment)). When this issue is fixed, my opinion is that we shall decide to either

1. add the flag to the default CMake configuration and wherever else it is missing, or
2. remove the flag from the make.inc.* files.

[martin-frbg]

Slight misunderstanding w.r.t -frecursive I think - this is required for correct functioning of multithreaded code, so should not be removed from the build files in general. However, a side effect is that it puts local arrays on the stack, which happen to be extremely large in the xeigtstz case. Removing this option from (only) the TESTING/EIG Makefile would appear to help in the simple case, but it appears to be implied when compiling with `-fopenmp' so this is no general solution for the unusually large stacklimit requirements of that test.

[julielangou]

so here is the why.
I agree LAPACK is a "simple" library - just having a make system should be enough, and it was enough for a long time.
But but but.. there is Windows.. and only because of Windows, we had to bring cmake. Originally, the people from CMake worked on the configuration, but now I guess we are on our own, and like @langou, we are not really mastering it.
Now most the softwares based on LAPACK are using Cmake and we know from our user feedback, they like it.

So if we have to pick one build system: this has to be Cmake... unfortunately for the present but fortunately for the future.

My vote: following @abouteiller and @@therault 's advice and have only one build system, thus cmake.

[christoph-conrads]

Now most the softwares based on LAPACK are using Cmake and we know from our user feedback, they like it.

I opened this issue and proposed to have just one build system. My assumption was that Makefile and CMake skills are about evenly distributed. Please correct if I am wrong but my impression is that this is not true among the regular LAPACK contributors: Makefiles seem to be popular and CMake knowledge seem not to be strong. Moreover, after two days of discussion, the -frecursive flags is apparently the only difference between the CMake and the Makefile build and the fact that only CMake is used in Travis CI. It does not make sense to force one build system that makes users happy if it drives away core contributors.

(My judgement about who is a regular contributor is based on looking at the first five pages of accepted pull requests.)

[weslleyspereira]

Agree.. BTW, the recursive flags are in the default CMake and Makefile configurations with #492. So, I think we just need to include the Makefile build in the Travis CI.

[weslleyspereira]

PR #500 mitigates the inconsistencies between the Makefile and CMake build systems reported here. See #500 (comment)

[langou]

Hi @ilayn. Thanks for the Meson / CMake / Makefile information. It is great to have outside feedback, and get information on what other projects are doing. We cannot support all three systems and we prefer CMake and Makefile for now, so we will likely decommissioned Meson. Not saying that we will never use Meson, but for now, we do not have the resource to maintain it. This is my opinion. I think we will submit a PR soon decommissioning Meson. Julien.

[smijolovic]

My experience with other build systems is as follows:

* Bazel: a huge memory hog, impossible to run on a Raspberry Pi with 1GB of memory, insists to build all dependencies, and link everything statically

* Autotools: as a developer no experience, as a user the `configure` script is very convenient because it can show you all available options (CMake does that with `ccmake` only after running `cmake` once).

I really wish people would stop using these abstraction oriented build systems. Bazel in particular is a security vulnerability mass propagation system, and is never designed to support air-gapped/dark-site development in ANY project its used for. The expectation of internet connectivity to build code is a recipe for disaster. Can we stick to good ol make....and stop trying to reinvent the wheel with all of these poorly designed build systems?