WIP: CMake

[yuyichao]

So after struggling for a few month, I finally decided to do the cmake port myself during the free time of a physics conference. A few things that I want to mention.

1. This is a little bit cheating since I'm calling make for the part I've not ported yet. It does have the advantage that the build is somewhat functional as I'm porting it.
2. I know there's concern about cross compilation. In my experience, cmake can be used to cross compile with simple environment variable settings for many projects without explicit support in the project. For more complex stuff, a toolchain file works pretty well too. For julia, since we need to compile the system image, we will need to do sth ourselves but that shouldn't be too bad.
3. This is probably the worst (public) code I've ever written so far in many sense. (style, commit log, commits etc). It is mainly because of a mixture between the cmake style and the makefile style. It will go through clean ups and squashes before merging for sure.

In it's current state,

1. Out-of-tree build of upto julia excutable works. Dependencies and sysimg are still calling the old makefile
2. Compilation of sysimg shouldn't be hard and so does dependencies. However, they are reasonably well decoupled from each other and I think it's a good time to get help / feedback about compilation options (many of them are not supported yet)

@tkelman

Ref #9422 #11645 #1832

[ScottPJones]

👍 💯 I'll try this out tonight

[carnaval]

Looks cool. What are the advantages (or future ones) ? I don't know much about buildsystems but from the outside what we have now seems to work well. If it eases up the maintenance for people actually making this work then of course all in :-)

Maybe the only problem I ever encounter is parallel build from scratch sometimes fails on dependencies (but restarting fixes it).

[tkelman]

#1832 (comment)

I don't think rolling our own will be maintainable forever. LLVM plans to deprecate autotools entirely in another version or two. I think the right immediate path forward for this is to add it as an alternative option coexisting with the makefiles for a while (use the snippet from #9422 to disallow in-tree builds, so this doesn't overwrite the existing makefiles), until people get used to it.

[yuyichao]

@carnaval

I don't know much about buildsystems but from the outside what we have now seems to work well.

I totally agree with you. In fact, that's the main reason I've been struggling whether to implement this even though I'm personally a cmake fan. It should provide certain benefit (see @tkelman 's comment above and I'll also list some of mine below) but it's quite boring to do... (It's not that hard but can get quite boring after a while, which is, btw, when I decided to push it out...)

A few benefit that I can think of

1. I really like the cmake template system (configure_file), this makes it easier to add compile time options (without having to manually edit a git managed file options.h)
2. Out of tree build. Same with @tkelman , I'm also a big fan of this.
3. The structure can be cleaner (not a big Make.inc). I guess it's not really convincing from this PR but should be much better after a clean up.
4. (This one is not a good argument but) Personally I use cmake in almost all other projects (most noticably KDE) and it's quite easy to manage. (I know there must be autotool and Makefile fans but in my experience and from my friends in other distros, build systems based on cmake has a better average quality and is easier to deal with for packagers)

A few possibly down side

1. Cross-compilation. In my experience, I've used the toolchain file in one of my other project for cross compilation for ARM and it works. I've also made a few lib32 package with cmake by setting certain compiler/cmake options and it was as easy as other build systems. Therefore, I think it is possible and shouldn't be worse than the old system but others with more experience should definitely comment on this.
2. The support for custom command is not great. (i.e. correct cross-directory rebuild/dependency rules). I think I've figured out how to do this when dealing with this with my other projects and I think it should not be a big issue.

[yuyichao]

Oh and MSVC support (which is convered by @tkelman 's comment). Just want to mention here because that's the argument I hear a lot. (Although personally I couldn't care less...)

[tkelman]

With spec file macros or an equivalent toolchain file, cross-compilation in cmake works fine. I've used it for Lapack, Metis, HDF5, AMPL-MP, and others on the opensuse build service, it's not that bad. More verbose than --host=x86_64-w64-mingw32 sure, but it just takes getting used to how to do things correctly. (The multiarch issue in #10010 is actually pkg-config's fault, it works fine if you don't have pkg-config installed.) For running cross-compiled executables, we can put in some logic for whether to run them via wine or qemu, like we have in the makefiles with the spawn definition but generalized even further for ARM-cross.

The custom-commands-with-side-effects has gotten better in recent versions of cmake, there are some new features for that. Though you can possibly get into trouble on old distros if you require a feature that's only in the newest cmake - there are PPA's or generic binary installs of cmake that work though.

This is tedious manual work to port over makefiles into cmake and @yuyichao is a saint for putting the work into it. Once we can start using ExternalProject as a standard template for dependencies, a lot of the nasty boilerplate from deps/Makefile goes away and it'll really be worth it. Cmake has solved that problem quite well.

[yuyichao]

For running cross-compiled executables, we can put in some logic for whether to run them via wine or qemu, like we have in the makefiles with the spawn definition but generalized even further for ARM-cross.

and

Once we can start using ExternalProject as a standard template for dependencies, a lot of the nasty boilerplate from deps/Makefile goes away and it'll really be worth it.

Yes that's basically the plan. I've never personally used external_project but I saw projects that uses it and works.

The custom-commands-with-side-effects has gotten better in recent versions of cmake, there are some new features for that.

The pattern I'm using is this. I don't have a old cmake to test it though.

I also have some functions for download/untar with the expected behavior here and I'll bring those if necessary. (That file is GPL but I'm the author so don't worry about it...)

[yuyichao]

@tkelman I was trying how to get cmake to link the sysimg object file to a shared object file. Can you check if this trick works with MSVC when you have time?

[yuyichao]

Another thing is the generation of .ji and .o files feels slower now (although it's possibly just the slowdown of my laptop when powered with battery). I'll do a careful comparison of flags and speed later (after porting is done, which will not be very soon...) but before that, does anyone know if there's any compiler flags that can cause this?

[tkelman]

I think it'll need to be using a newer LLVM than 3.3 to emit comdat sections for the MSVC linker to accept it, and I don't have an MSVC build of recent LLVM handy. Based on some recent commits to master (3a63b2a) I think @vtjnash may have been doing something related to using the MSVC linker but whatever he was doing wasn't exactly openly-worked-on through an issue/PR. I suspect that was more about linking a mingw-built sys.dll into an msvc-built application though, otherwise he probably would have also fixed the alignment segfault in jl_setjmp that's prevented an MSVC-built Julia from working properly.

[yuyichao]

That's a pack of issues with MSVC that I've never heard of before....
I'll just interpret it as this trick should work in general but I don't have to worry about MSVC for the time being.

Anyway, I'm not seeking MSVC support in this PR and I feel like this PR should be post 0.4 (or even post Arraymageddon depending on how long it takes).

[tkelman]

Agreed, MSVC support does not need to be fully working within the scope of this PR. I haven't even opened a separate issue for the jl_setjmp segfault because to the best of my knowledge I'm the only person who tests building Julia with MSVC on a remotely regular basis. But future enhancements to a cmake build system would allow MSVC support to be much more first-class, and entirely avoid relying on a posix environment (assuming binary dll's are used for all ccall'ed dependencies that do not use cmake themselves).

[vtjnash]

I think @vtjnash may have been doing something related to using the MSVC linker but whatever he was doing wasn't exactly openly-worked-on through an issue/PR

it wasn't anything related to msvc, sorry. i just sometimes get asked to do unusual tests like 366740a to certain use cases more reliable (or in that case, to fail with certainty on any pointer bugs). this was inspired by the default behavior on OS X as documented in http://www.unix.com/man-page/osx/1/ld/ -pagezero_size

i've never attempted a msvc build, although it might make some aspects of debugging / line numbers easier if we could get it working.

[tkelman]

-fPIC gives a warning with mingw, that's why it's set to empty on windows

[yuyichao]

I see. These compiler flags are far from complete. I'm planing to do a more careful port of the flags after the infrastructure is there.

[yuyichao]

Just curious. What warning does it give?

[tkelman]

../Source/umf_symbolic_usage.c:1:0: warning: -fPIC ignored for target (all code is position independent) [enabled by default]
 /* ========================================================================== */
 ^

Harmless, but annoying if it happens on every file.

[yuyichao]

This is probably one of the stupidest warning I've seen.....

[Luthaf]

There is a CMAKE_POSITION_INDEPENDENT_CODE variable in cmake (and the corresponding POSITION_INDEPENDENT_CODE property for targets) which should handle this.

[yuyichao]

Thanks. Will use that.

[yuyichao]

BTW, this is cleared for mingw or do I still need to check that.

[tkelman]

Collecting some todo's from me as a comment instead of inline so github doesn't eat them on a rebase:

• don't use -fPIC flag on windows
• replace calls to perl with built-in cmake regexes
• use the built-in default cmake variables e.g. MSVC instead of our current USEMSVC, etc
• error out for CYGWIN (or msys or msys2 targets) with the suggestion to cross-compile (double check that cygwin-build, mingw-host cross compile sets MINGW and WIN32 but not CYGWIN)
• eventually put in a toolchain file for mingw cross compile (from cygwin or linux)
• error out for obsolete mingw.org (distinguish from mingw-w64 via -dumpmachine)
• figure out right approach for settings files, either via configure_file or a CMakeCache.txt-to-Make.user translation or something else
• error on in-tree builds with cmake so this doesn't have to break the makefiles right away
• check for build-time dependencies like m4 when not using system gmp, python (2.x) when not using system LLVM or libgit2, etc
• _setjmp and _longjmp files are assembly, not c (need test)
• deal with needing to change JL_SYSTEM_IMAGE_PATH on install
• see if newer cmake dependency-tracking features can replace most of the confusing utility functions

[yuyichao]

Actually for breaking makefile, can we rename the current Makefiles to makefile?

[yuyichao]

NVM doesn't work on case insensitive system.... = = ...............................

[tkelman]

these are assembly, .S

[yuyichao]

Ooops.

[yuyichao]

BTW let's move the discussion to the main comment. It's easier to get lost and I find the main comment easier to find. We can link to the code if necessary.

[yuyichao]

So the part I would like to keep is basically this, which I think is pretty straightforward.

The logic in builtins.c is curretly just an ad-hoc solution and can be changed. I was actually thinking about a separate PR for that but I'd like to work out what I need and what would be useful first.

[tkelman]

That foreach, on its own, makes sense for dealing with timestamps. The automatic generation of dependency file lists doesn't, it's too confusing and indirect.

[yuyichao]

The automatic generation of dependency file lists doesn't, it's too confusing and indirect.

Do you mean, how the file is generated? how it is read? why it is needed?

[tkelman]

All of the code for dealing with those lists is currently confusing. I can see roughly what you're aiming for, but I think it's total overkill for now.

Reconfigure is necessary because otherwise the command will not be re-run when the added file is edited.

Sure. And I think that's just fine - we say that you have to re-run the cmake configure step (or do make clean which would force re-running the command by virtue of its target no longer existing) any time you add or remove a julia file, if you want the dependency tracking to be perfectly up-to-date. I think that's a perfectly acceptable limitation in exchange for removing confusing code from the build system.

[yuyichao]

I guess you are confused by the autodeps file.

[yuyichao]

And you'd be right about it. I'm actually using that file as a time stamp to rerun the command when the compilation is aborted. I totally agree that needs to be changed. For the autodeps you can just replace it with the one without the suffix and it should still works.

[tkelman]

That and the .fnames logging to populate them, which seems messy. Might the .ji file (or the soon-to-be .o file with embedded bitcode) already have source file information encoded into it in some retrievable way?

[tkelman]

And wouldn't setting DEPENDS in a standard add_custom_command accomplish the same thing as the timestamp-checking foreach?

[tkelman]

Given the discussion in #8745 (comment) it sounds like we might need to implement the checksumming/listing of source files included during sysimg build anyway, either as a separate file or embedded into a data section of the shared library. So I'm not as opposed to that now as I originally was.

[yuyichao]

Yeah, I noticed that and I'm just going to wait for a decision to be made there before I replace what I have here with the new mechanism.

This might mean that I could not have a generic version of it and rather a sysimg specific macro but that's totally find and will make the wrapper easier to understand as well.

[tkelman]

Actually now I'm thinking generic is good, we could install a few cmake definitions in share or something that teach cmake how to build packages or other Julia modules. That would solve the build system issue, avoiding reinventing make by instead just using cmake. But for now the build system checksumming/timestamping parts of the question are not the problem that PR is trying to address.

[yuyichao]

By "generic" here I mean apply to other kind of targets as well (i.e. not only julia sysimg/module compilation), which is probably not what you want.

But for now the build system checksumming/timestamping parts of the question are not the problem that PR is trying to address.

Yeah. these helper functions will be in a separate file and figuring out how to reuse them can be done later.

[ViralBShah]

I'm closing this one since there doesn't seem to be an immediate need for it, and probably needs new work anyways given how old it is.