Skip to content
Merged

Ccache #13498

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/emcc.txt
Original file line number Diff line number Diff line change
Expand Up @@ -645,6 +645,8 @@ Environment variables

* "EM_CONFIG" [general]

* "EM_LLVM_ROOT" [compile+link]

Search for 'os.environ' in emcc.py to see how these are used. The most
interesting is possibly "EMCC_DEBUG", which forces the compiler to
dump its build and temporary files to a temporary directory where they
Expand Down
7 changes: 6 additions & 1 deletion em++
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,9 @@ if [ -z "$PYTHON" ]; then
exit 1
fi

exec "$PYTHON" "$0.py" "$@"
if [ -z "$EMCC_CCACHE" ]; then
exec "$PYTHON" "$0.py" "$@"
else
unset EMCC_CCACHE
exec ccache "$0" "$@"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to handle injecting ccache like this when other compilers such as clang and gcc don't? Can't we ask folks to use cache in the normal way (which I believe is to perpend it to compile command?)

Copy link
Collaborator Author

@juj juj Mar 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we ask folks to use cache in the normal way

This is the/a normal way. Ccache supports two methods of installation: 1) explicit invocation with ccache gcc ..., and 2) automatic invocation with gcc .... See the documentation of ccache.

(which I believe is to perpend it to compile command?)

That is not the full picture. See the docs above. There are two recommended ways to install ccache, this integration is ensuring that both usages work.

E.g. man documentation: https://www.freebsd.org/cgi/man.cgi?query=ccache&sektion=1&n=1

RUN MODES
       There are two ways to use ccache. You can either	prefix your
       compilation commands with ccache	or you can let ccache masquerade as
       the compiler by creating	a symbolic link	(named as the compiler)	to
       ccache. The first method	is most	convenient if you just want to try out
       ccache or wish to use it	for some specific projects. The	second method
       is most useful for when you wish	to use ccache for all your
       compilations.

Why do we need to handle injecting ccache like this when other compilers such as clang and gcc don't?

The reason that symbolic links are not used by this integration is because that would intrude with emsdk installation of either ccache or emscripten packages (say, e.g. if installing ccache package would modify installed emscripten package)

Also while Linux and macOS could use symbolic links to inject ccache, that is not well supported on Windows.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thoughts about this in general: if neither gcc nor clang require explicit in-tree modification to make ccache work I would hope that we do not need this either.

If seems that we are adding a third RUN MODE here that is emscripten-specific:

  1. Run explicitly via ccache <compiler>
  2. Run implicitly via having a <compiler> -> ccache symlink in your PATH
  3. Run implicitly via add EMCC_CACHE to the environment.

Given that mode (3) does not exist today with existing compilers I don't see why we in emscripten want to be special and add this third mode. I understand that mode (2) is hard/impossible on windows but that is pre-existing condition unrelated to emscripten, right?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't agree here. Given that we can offer an easy "emsdk activate ccache" feature that just does the full integration, I don't understand why we should not do it. It gives a straightforward "fully set up out of the box" experience without manual hassles. Option 2 is a complex burden on Windows, but also on non-Windows platforms.

Also, the step (3) does not need to be something public, but an internal implementation aid to carry the integration together. I.e. we are not offering a "third run mode" here.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I get that we want this to work out-of-the-box for all emsdk users including windows users.

I'd still like find a solution that doesn't involved modifying emscripten. How about this solution:

  1. create emsdk/ccache/bin/emcc and emsdk/ccache/bin/em++ (these are bat/sh wrappers rather than symlinks)
  2. Have emsdk activate ccache put emsdk/ccache/bin first in the PATH before emscripten/bin.

This avoids changing emscripten at all and works more in like the symlink approach that exists today for ccache (but also happens to work on windows). It also works with older/current versions of emscripten too.

This means figuring out how to chain emsdk/ccache/bin/emcc -> emscripten/bin/emcc in the wrapper script. I don't know of the top of my head how to do that. If its too tricky I'm not going to resist this emscripten-side change too much, but it does seem wrong to me that we are hacking the compiler to be specifically aware of ccache (unlike gcc or clang).

fi
9 changes: 8 additions & 1 deletion em++.bat
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,11 @@
set EM_PY=python
)

@"%EM_PY%" "%~dp0\%~n0.py" %*
@if "%EMCC_CCACHE%"=="" (
:: Do regular invocation of em++.py compiler
"%EM_PY%" "%~dp0\%~n0.py" %*
) else (
:: Remove the ccache env. var, invoke ccache and re-enter this script to take the above branch.
set EMCC_CCACHE=
ccache "%~dp0\%~n0.bat" %*
)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These launcher scripts are tool-generated (by tools/create_entry_points.py), so it want to move forward with this I think we would want to modify that script.

Or as an alternative we would inject the cache command around the inner call to clang rather than the out call to python emcc?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Injecting to the inner clang call is considerably weaker in terms of performance than injecting to the outer python emcc call.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the create_entry_points.py script.

7 changes: 6 additions & 1 deletion emcc
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,9 @@ if [ -z "$PYTHON" ]; then
exit 1
fi

exec "$PYTHON" "$0.py" "$@"
if [ -z "$EMCC_CCACHE" ]; then
exec "$PYTHON" "$0.py" "$@"
else
unset EMCC_CCACHE
exec ccache "$0" "$@"
fi
9 changes: 8 additions & 1 deletion emcc.bat
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,11 @@
set EM_PY=python
)

@"%EM_PY%" "%~dp0\%~n0.py" %*
@if "%EMCC_CCACHE%"=="" (
:: Do regular invocation of em++.py compiler
"%EM_PY%" "%~dp0\%~n0.py" %*
) else (
:: Remove the ccache env. var, invoke ccache and re-enter this script to take the above branch.
set EMCC_CCACHE=
ccache "%~dp0\%~n0.bat" %*
)
1 change: 1 addition & 0 deletions site/source/docs/tools_reference/emcc.rst
Original file line number Diff line number Diff line change
Expand Up @@ -540,6 +540,7 @@ Environment variables
- ``EMCC_SKIP_SANITY_CHECK`` [general]
- ``EM_IGNORE_SANITY`` [general]
- ``EM_CONFIG`` [general]
- ``EM_LLVM_ROOT`` [compile+link]

Search for 'os.environ' in `emcc.py <https://github.com/emscripten-core/emscripten/blob/main/emcc.py>`_ to see how these are used. The most interesting is possibly ``EMCC_DEBUG``, which forces the compiler to dump its build and temporary files to a temporary directory where they can be reviewed.

Expand Down
2 changes: 1 addition & 1 deletion src/settings.js
Original file line number Diff line number Diff line change
Expand Up @@ -1187,7 +1187,7 @@ var DYNAMIC_EXECUTION = 1;
var BOOTSTRAPPING_STRUCT_INFO = 0;

// Add some calls to emscripten tracing APIs
// [link]
// [compile+link]
var EMSCRIPTEN_TRACING = 0;

// Specify the GLFW version that is being linked against. Only relevant, if you
Expand Down
39 changes: 26 additions & 13 deletions tools/create_entry_points.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,25 +7,28 @@
"""Tool for creating/maintains the python launcher scripts for all the emscripten
python tools.
This tools makes copies or `run_python.sh` and `run_python.bat` script for each
entry point. On UNIX we previously used symbolic links for simplicity but this
breaks MINGW users on windows who want use the shell script launcher but don't
have symlink support (sigh).
This tools makes copies or `run_python.sh/.bat` and `run_python_compiler.sh/.bat`
script for each entry point. On UNIX we previously used symbolic links for
simplicity but this breaks MINGW users on windows who want use the shell script
launcher but don't have symlink support.
"""

import os
import shutil
import sys

compiler_entry_points = '''
emcc
em++
'''.split()

entry_points = '''
emar
embuilder
emcc
emcmake
em-config
emconfigure
emmake
em++
emranlib
emrun
emscons
Expand All @@ -39,13 +42,23 @@
def main():
tools_dir = os.path.dirname(os.path.abspath(__file__))
root_dir = os.path.dirname(tools_dir)
sh_file = os.path.join(tools_dir, 'run_python.sh')
bat_file = os.path.join(tools_dir, 'run_python.bat')
for entry_point in entry_points:
if os.path.exists(os.path.join(root_dir, entry_point)):
os.remove(os.path.join(root_dir, entry_point))
shutil.copy2(sh_file, os.path.join(root_dir, entry_point))
shutil.copy2(bat_file, os.path.join(root_dir, entry_point) + '.bat')

def generate_entry_points(cmd, path):
sh_file = path + '.sh'
bat_file = path + '.bat'
for entry_point in cmd:
dst = os.path.join(root_dir, entry_point)
if os.path.exists(dst):
os.remove(dst)
shutil.copy2(sh_file, dst)

dst += '.bat'
if os.path.exists(dst):
os.remove(dst)
shutil.copy2(bat_file, dst)

generate_entry_points(entry_points, os.path.join(tools_dir, 'run_python'))
generate_entry_points(compiler_entry_points, os.path.join(tools_dir, 'run_python_compiler'))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess there are two new files here that need to be added?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, indeed missed adding those.



if __name__ == '__main__':
Expand Down
18 changes: 18 additions & 0 deletions tools/run_python_compiler.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
:: Entry point for running python scripts on windows systems.
:: To modify this file, edit `tools/run_python.bat` and then run
:: `tools/create_entry_points.py`

@setlocal
@set EM_PY=%EMSDK_PYTHON%
@if "%EM_PY%"=="" (
set EM_PY=python
)

@if "%EMCC_CCACHE%"=="" (
:: Do regular invocation of em++.py compiler
"%EM_PY%" "%~dp0\%~n0.py" %*
) else (
:: Remove the ccache env. var, invoke ccache and re-enter this script to take the above branch.
set EMCC_CCACHE=
ccache "%~dp0\%~n0.bat" %*
)
34 changes: 34 additions & 0 deletions tools/run_python_compiler.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/bin/sh
# Copyright 2020 The Emscripten Authors. All rights reserved.
# Emscripten is available under two separate licenses, the MIT license and the
# University of Illinois/NCSA Open Source License. Both these licenses can be
# found in the LICENSE file.
#
# Entry point for running python scripts on UNIX systems.
#
# To modify this file, edit `tools/run_python.sh` and then run
# `tools/create_entry_points.py`

if [ -z "$PYTHON" ]; then
PYTHON=$EMSDK_PYTHON
fi

if [ -z "$PYTHON" ]; then
PYTHON=$(which python3 2> /dev/null)
fi

if [ -z "$PYTHON" ]; then
PYTHON=$(which python 2> /dev/null)
fi

if [ -z "$PYTHON" ]; then
echo 'unable to find python in $PATH'
exit 1
fi

if [ -z "$EMCC_CCACHE" ]; then
exec "$PYTHON" "$0.py" "$@"
else
unset EMCC_CCACHE
exec ccache "$0" "$@"
fi