Ccache

[juj]

This PR adds support for Ninja build system and ccache, to be built from source.

Needs a bit of code duplication cleanup in order to be able to land, but posting for visibility so that users can try it out.

The corresponding PR for emscripten is at emscripten-core/emscripten#13498 and the corresponding changes to ccache itself are at https://github.com/juj/ccache/tree/emscripten.

The general mechanism is that after activating ccache in Emscripten SDK, emcc and em++ will automatically take usage of the installed ccache tool.

Usage:

emsdk install sdk-upstream-master-64bit ccache-git-emscripten-64bit
emsdk activate sdk-upstream-master-64bit  ccache-git-emscripten-64bit
cd %EMSCRIPTEN%
git remote add juj https://github.com/juj/emscripten.git
git fetch emscripten
git checkout ccache

# Example 1: rebuilding libc is fast from cache:
ccache -s # check initial ccache statistics: zero compiled files in cache
emcc --clear-cache

python embuilder.py build libc # cache miss
ccache -s # will show 862 files missed cache

ccache -z # zero ccache statistics

emcc --clear-cache
python embuilder.py build libc # cache hit!
ccache -s # now shows 862 files found from cache

# Example 2: rebuilding custom code is fast from cache:
ccache -z # zero statistics
emcc -c tests\hello_world.c -o a.o # cache miss
ccache -s # shows one file missed cache
emcc -c tests\hello_world.c -o a.o # cache hit!
ccache -s # shows one file hit cache

[sbc100]

Why not just treat ninja like node an download the prebuilt binaries?

Looks like the zip files are easy to link to: https://github.com/ninja-build/ninja/releases

Or maybe just treat it like cmake itself and assume the developer has already installed it? Building from source seems like a less trodden path so have extra dependencies seems reasonable maybe? At least it seems like cmake and ninja should be treated in the same way.

[juj]

Mainly because of being able to change up ninja code in a custom branch, because of getting direct Apple M1 Mac support this way, and to get backwards compatibility support against whichever Xcode SDK and Windows SDK that I compile for myself. (I don't know the guaranteed min OS versions of the precompiled releases).

[sbc100]

Should we at least try it? or at least have "download binary" as the default way to download ninja? (then only folks with unusual OS/hardware need to build from source, like we do for llvm)

[sbc100]

Do you have have patches to ninja planned?

[juj]

I would not oppose adding prebuilt ninja as well, though I am not keen to work on that at the moment. Fortunately emsdk does allow a good structure to have both.

I do plan to change ninja to fix stdout printing issues on Windows. When one uses ninja, they lose colored output for error messages and all stdout gets needlessly buffered, resulting in EMCC_DEBUG being uncomfortable to use on Windows.

[sbc100]

I would prefer to simply use ninja pre-built bit I'm OK landing this as is.

Regarding the specifics of the ninja buffering issue. Are you talking about how the buffering causes the tools to skip adding color to their output? I agree it would be nice to find a fix for this, but I also really like how ninja buffers command and output rather than the nasty interleaving one gets with make VERVOSE=1 -jN.

[juj]

Ping, any other reviews? If not, I'll proceed and land this.

[sbc100]

lgtm

[sbc100]

I would prefer to simply use ninja pre-built bit I'm OK landing this as is.

Regarding the specifics of the ninja buffering issue. Are you talking about how the buffering causes the tools to skip adding color to their output? I agree it would be nice to find a fix for this, but I also really like how ninja buffers command and output rather than the nasty interleaving one gets with make VERVOSE=1 -jN.

[juj]

Are you talking about how the buffering causes the tools to skip adding color to their output?

Yeah, that's right.

I also really like how ninja buffers command and output rather than the nasty interleaving one gets with make VERVOSE=1 -jN.

I think the right way is to do line buffering per each thread, and not character/printf segment buffering, but mingw32-make -j on Windows fails on doing that proper. Having multiple threads interleave the printed lines of their output is more desirable to me for responsiveness, rather than get a long silence that first makes you wonder if the build is hung (which can happen with python multiprocessing) and then a sudden flood of everything that got printed.

[juj]

I would prefer to simply use ninja pre-built bit I'm OK landing this as is.

Not opposed at all with also having pre-built ninja packages in the manifest, but I don't find that so useful, mostly due to not getting M1 Mac Mini native ARM64 code, so not spending the time to work on that for now.

[sbc100]

I would prefer to simply use ninja pre-built bit I'm OK landing this as is.

Not opposed at all with also having pre-built ninja packages in the manifest, but I don't find that so useful, mostly due to not getting M1 Mac Mini native ARM64 code, so not spending the time to work on that for now.

I see.. it hadn't occurred to me that there might not be a ninja binary for that platform yet. Thanks for clarifying.

[mr-c]

Thank you @juj for this work and the fork of ccache; have you submitted your enhancements upstream?