-
Notifications
You must be signed in to change notification settings - Fork 3.5k
Ccache #13498
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ccache #13498
Changes from all commits
bfd21a6
cb8c881
fd22b8a
ace00c0
3bad5f5
1693577
c6cf4f8
7f52afa
f613c53
79400fd
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,4 +8,11 @@ | |
| set EM_PY=python | ||
| ) | ||
|
|
||
| @"%EM_PY%" "%~dp0\%~n0.py" %* | ||
| @if "%EMCC_CCACHE%"=="" ( | ||
| :: Do regular invocation of em++.py compiler | ||
| "%EM_PY%" "%~dp0\%~n0.py" %* | ||
| ) else ( | ||
| :: Remove the ccache env. var, invoke ccache and re-enter this script to take the above branch. | ||
| set EMCC_CCACHE= | ||
| ccache "%~dp0\%~n0.bat" %* | ||
| ) | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These launcher scripts are tool-generated (by Or as an alternative we would inject the
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Injecting to the inner clang call is considerably weaker in terms of performance than injecting to the outer python emcc call.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Updated the create_entry_points.py script. |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,25 +7,28 @@ | |
| """Tool for creating/maintains the python launcher scripts for all the emscripten | ||
| python tools. | ||
| This tools makes copies or `run_python.sh` and `run_python.bat` script for each | ||
| entry point. On UNIX we previously used symbolic links for simplicity but this | ||
| breaks MINGW users on windows who want use the shell script launcher but don't | ||
| have symlink support (sigh). | ||
| This tools makes copies or `run_python.sh/.bat` and `run_python_compiler.sh/.bat` | ||
| script for each entry point. On UNIX we previously used symbolic links for | ||
| simplicity but this breaks MINGW users on windows who want use the shell script | ||
| launcher but don't have symlink support. | ||
| """ | ||
|
|
||
| import os | ||
| import shutil | ||
| import sys | ||
|
|
||
| compiler_entry_points = ''' | ||
| emcc | ||
| em++ | ||
| '''.split() | ||
|
|
||
| entry_points = ''' | ||
| emar | ||
| embuilder | ||
| emcc | ||
| emcmake | ||
| em-config | ||
| emconfigure | ||
| emmake | ||
| em++ | ||
| emranlib | ||
| emrun | ||
| emscons | ||
|
|
@@ -39,13 +42,23 @@ | |
| def main(): | ||
| tools_dir = os.path.dirname(os.path.abspath(__file__)) | ||
| root_dir = os.path.dirname(tools_dir) | ||
| sh_file = os.path.join(tools_dir, 'run_python.sh') | ||
| bat_file = os.path.join(tools_dir, 'run_python.bat') | ||
| for entry_point in entry_points: | ||
| if os.path.exists(os.path.join(root_dir, entry_point)): | ||
| os.remove(os.path.join(root_dir, entry_point)) | ||
| shutil.copy2(sh_file, os.path.join(root_dir, entry_point)) | ||
| shutil.copy2(bat_file, os.path.join(root_dir, entry_point) + '.bat') | ||
|
|
||
| def generate_entry_points(cmd, path): | ||
| sh_file = path + '.sh' | ||
| bat_file = path + '.bat' | ||
| for entry_point in cmd: | ||
| dst = os.path.join(root_dir, entry_point) | ||
| if os.path.exists(dst): | ||
| os.remove(dst) | ||
| shutil.copy2(sh_file, dst) | ||
|
|
||
| dst += '.bat' | ||
| if os.path.exists(dst): | ||
| os.remove(dst) | ||
| shutil.copy2(bat_file, dst) | ||
|
|
||
| generate_entry_points(entry_points, os.path.join(tools_dir, 'run_python')) | ||
| generate_entry_points(compiler_entry_points, os.path.join(tools_dir, 'run_python_compiler')) | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I guess there are two new files here that need to be added?
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks, indeed missed adding those. |
||
|
|
||
|
|
||
| if __name__ == '__main__': | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| :: Entry point for running python scripts on windows systems. | ||
| :: To modify this file, edit `tools/run_python.bat` and then run | ||
| :: `tools/create_entry_points.py` | ||
|
|
||
| @setlocal | ||
| @set EM_PY=%EMSDK_PYTHON% | ||
| @if "%EM_PY%"=="" ( | ||
| set EM_PY=python | ||
| ) | ||
|
|
||
| @if "%EMCC_CCACHE%"=="" ( | ||
| :: Do regular invocation of em++.py compiler | ||
| "%EM_PY%" "%~dp0\%~n0.py" %* | ||
| ) else ( | ||
| :: Remove the ccache env. var, invoke ccache and re-enter this script to take the above branch. | ||
| set EMCC_CCACHE= | ||
| ccache "%~dp0\%~n0.bat" %* | ||
| ) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| #!/bin/sh | ||
| # Copyright 2020 The Emscripten Authors. All rights reserved. | ||
| # Emscripten is available under two separate licenses, the MIT license and the | ||
| # University of Illinois/NCSA Open Source License. Both these licenses can be | ||
| # found in the LICENSE file. | ||
| # | ||
| # Entry point for running python scripts on UNIX systems. | ||
| # | ||
| # To modify this file, edit `tools/run_python.sh` and then run | ||
| # `tools/create_entry_points.py` | ||
|
|
||
| if [ -z "$PYTHON" ]; then | ||
| PYTHON=$EMSDK_PYTHON | ||
| fi | ||
|
|
||
| if [ -z "$PYTHON" ]; then | ||
| PYTHON=$(which python3 2> /dev/null) | ||
| fi | ||
|
|
||
| if [ -z "$PYTHON" ]; then | ||
| PYTHON=$(which python 2> /dev/null) | ||
| fi | ||
|
|
||
| if [ -z "$PYTHON" ]; then | ||
| echo 'unable to find python in $PATH' | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [ -z "$EMCC_CCACHE" ]; then | ||
| exec "$PYTHON" "$0.py" "$@" | ||
| else | ||
| unset EMCC_CCACHE | ||
| exec ccache "$0" "$@" | ||
| fi |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need to handle injecting ccache like this when other compilers such as clang and gcc don't? Can't we ask folks to use cache in the normal way (which I believe is to perpend it to compile command?)
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the/a normal way. Ccache supports two methods of installation: 1) explicit invocation with
ccache gcc ..., and 2) automatic invocation withgcc .... See the documentation of ccache.That is not the full picture. See the docs above. There are two recommended ways to install ccache, this integration is ensuring that both usages work.
E.g. man documentation: https://www.freebsd.org/cgi/man.cgi?query=ccache&sektion=1&n=1
The reason that symbolic links are not used by this integration is because that would intrude with emsdk installation of either ccache or emscripten packages (say, e.g. if installing ccache package would modify installed emscripten package)
Also while Linux and macOS could use symbolic links to inject ccache, that is not well supported on Windows.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My thoughts about this in general: if neither gcc nor clang require explicit in-tree modification to make ccache work I would hope that we do not need this either.
If seems that we are adding a third
RUN MODEhere that is emscripten-specific:ccache <compiler><compiler> -> ccachesymlink in your PATHEMCC_CACHEto the environment.Given that mode (3) does not exist today with existing compilers I don't see why we in emscripten want to be special and add this third mode. I understand that mode (2) is hard/impossible on windows but that is pre-existing condition unrelated to emscripten, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't agree here. Given that we can offer an easy "emsdk activate ccache" feature that just does the full integration, I don't understand why we should not do it. It gives a straightforward "fully set up out of the box" experience without manual hassles. Option 2 is a complex burden on Windows, but also on non-Windows platforms.
Also, the step (3) does not need to be something public, but an internal implementation aid to carry the integration together. I.e. we are not offering a "third run mode" here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I get that we want this to work out-of-the-box for all emsdk users including windows users.
I'd still like find a solution that doesn't involved modifying emscripten. How about this solution:
emsdk/ccache/bin/emccandemsdk/ccache/bin/em++(these are bat/sh wrappers rather than symlinks)emsdk activate ccacheputemsdk/ccache/binfirst in the PATH beforeemscripten/bin.This avoids changing emscripten at all and works more in like the symlink approach that exists today for ccache (but also happens to work on windows). It also works with older/current versions of emscripten too.
This means figuring out how to chain
emsdk/ccache/bin/emcc->emscripten/bin/emccin the wrapper script. I don't know of the top of my head how to do that. If its too tricky I'm not going to resist this emscripten-side change too much, but it does seem wrong to me that we are hacking the compiler to be specifically aware of ccache (unlike gcc or clang).