Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Fix build_ccache_wrappers: #14631

Merged
merged 1 commit into from
Jun 24, 2019
Merged

Fix build_ccache_wrappers: #14631

merged 1 commit into from
Jun 24, 2019

Conversation

larroy
Copy link
Contributor

@larroy larroy commented Apr 5, 2019

Description

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
  • Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
  • Code is well-documented:
  • For user-facing API changes, API doc string has been updated.
  • For new C++ functions in header files, their functionalities and arguments are documented.
  • For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
  • Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
  • To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

@larroy
Copy link
Contributor Author

larroy commented Apr 5, 2019

@mxnet-label-bot add [CI]

@marcoabreu marcoabreu added the CI label Apr 5, 2019
@larroy
Copy link
Contributor Author

larroy commented Apr 5, 2019

@mxnet-label-bot add [pr-awaiting-review]

@marcoabreu marcoabreu added the pr-awaiting-review PR is waiting for code review label Apr 5, 2019
@larroy larroy force-pushed the ccache_wrappers branch 3 times, most recently from d0c9197 to 13708f8 Compare April 5, 2019 22:04
ln -s ccache /usr/local/bin/clang++-5.0
ln -s ccache /usr/local/bin/clang-5.0
ln -s ccache /usr/local/bin/clang++-6.0
ln -s ccache /usr/local/bin/clang-6.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also restore the links in /usr/local/bin/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need, also they fail with permission denied when executing locally or without root privileges.

No need because they are added to the path.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you aware that the links you created previously were dangling? (basically didn't work)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you think the links to /usr/local/bin should be restored? Could you explain?

Copy link
Contributor

@marcoabreu marcoabreu Apr 19, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We spoke about this already a dozen times, so I will only link the official guide: https://ccache.dev/manual/latest.html#_run_modes

Back when I developed this, I tested it only by relaying on the PATH (hence I introduced /tmp/ccache-redirects) and it didn't work. Thus, I added the /usr/local/bin method.

The links did work - why shouldn't they?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It didn't work because your links were broken. So with my changes it works in /tmp/ccache-redirects . there's no need to touch /usr/local/bin this is not allowed for users, requires sudo privileges, it's not necessary, etc. Please elaborate why you don't accept my fix, which uses the ccache wrappers in the /tmp folder. The guide says "something like this" you are taking the guide too seriously, having in any other folder which is then injected first in the path is enough.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate on why are you requesting changes when my changes actually fix the problem and make ccache work? or do you think there is something that is not working with my changes? Is not clear from your comments.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are not answering my comments, can you get into the container and see that this PR is fixing your broken links? That's the reason your approach didn't work, and you still left the /tmp/ccache-redirect broken links. I fixed this, thus no need to write in /usr/local/bin which prevents the build functions from being used locally or without root permissions.

@marcoabreu
Copy link
Contributor

Thanks for making it idempotent!

@larroy
Copy link
Contributor Author

larroy commented Apr 6, 2019

ccache/ccache#373

@larroy larroy force-pushed the ccache_wrappers branch 3 times, most recently from 549d1f1 to 01540a2 Compare April 10, 2019 18:16
@larroy
Copy link
Contributor Author

larroy commented Apr 10, 2019

@marcoabreu can you have another look please? CI is passing.

@larroy
Copy link
Contributor Author

larroy commented Apr 10, 2019

@KellenSunderland you might want to chime in as well.

@Roshrini
Copy link
Member

@marcoabreu @KellenSunderland Can you help review this PR?

@marcoabreu
Copy link
Contributor

marcoabreu commented Apr 19, 2019

This code produces the following output:

    ls -l /usr/local/bin
    ln -s ccache /usr/local/bin/gcc
    ln -s ccache /usr/local/bin/gcc-8
    ln -s ccache /usr/local/bin/g++
    ln -s ccache /usr/local/bin/g++-8
    ln -s ccache /usr/local/bin/nvcc
    ln -s ccache /usr/local/bin/clang++-3.9
    ln -s ccache /usr/local/bin/clang-3.9
    ln -s ccache /usr/local/bin/clang++-5.0
    ln -s ccache /usr/local/bin/clang-5.0
    ln -s ccache /usr/local/bin/clang++-6.0
    ln -s ccache /usr/local/bin/clang-6.0
    ls -l /usr/local/bin
+ ls -l /usr/local/bin
total 872
drwxr-xr-x 1 jenkins_slave root   4096 Apr 16 22:48 __pycache__
-rwxr-xr-x 1 jenkins_slave root    229 Apr 16 22:48 backend-test-tools
-rwxr-xr-x 1 jenkins_slave root    216 Apr 16 22:49 breathe-apidoc
-rwxr-xr-x 1 jenkins_slave root 497848 Apr 16 22:36 ccache
-rwxr-xr-x 1 jenkins_slave root    224 Apr 16 22:37 chardetect
-rwxr-xr-x 1 jenkins_slave root    232 Apr 16 22:48 check-model
-rwxr-xr-x 1 jenkins_slave root    230 Apr 16 22:48 check-node
-rwxr-xr-x 1 jenkins_slave root    228 Apr 16 22:49 cm2html
-rwxr-xr-x 1 jenkins_slave root    230 Apr 16 22:49 cm2latex
-rwxr-xr-x 1 jenkins_slave root    226 Apr 16 22:49 cm2man
-rwxr-xr-x 1 jenkins_slave root    238 Apr 16 22:49 cm2pseudoxml
-rwxr-xr-x 1 jenkins_slave root    230 Apr 16 22:49 cm2xetex
-rwxr-xr-x 1 jenkins_slave root    226 Apr 16 22:49 cm2xml
lrwxrwxrwx 1 jenkins_slave root     20 Apr 16 22:35 cmake -> /opt/cmake/bin/cmake
-rwxr-xr-x 1 jenkins_slave root   1108 Apr 16 22:49 cmark.py
-rw-r--r-- 1 jenkins_slave root   1462 Apr 16 22:49 cmark.pyc
-rwxr-xr-x 1 jenkins_slave root    218 Apr 16 22:48 coverage
-rwxr-xr-x 1 jenkins_slave root    218 Apr 16 22:48 coverage-2.7
-rwxr-xr-x 1 jenkins_slave root    218 Apr 16 22:48 coverage-3.5
-rwxr-xr-x 1 jenkins_slave root    218 Apr 16 22:48 coverage2
-rwxr-xr-x 1 jenkins_slave root    218 Apr 16 22:48 coverage3
-rwxr-xr-x 1 jenkins_slave root    209 Apr 16 22:37 cpplint
-rwxr-xr-x 1 jenkins_slave root    223 Apr 16 22:46 cygdb
-rwxr-xr-x 1 jenkins_slave root    244 Apr 16 22:46 cython
-rwxr-xr-x 1 jenkins_slave root    224 Apr 16 22:46 cythonize
-rwxr-xr-x 1 jenkins_slave root    233 Apr 16 22:36 easy_install
-rwxr-xr-x 1 jenkins_slave root    233 Apr 16 22:36 easy_install-2.7
-rwxr-xr-x 1 jenkins_slave root    233 Apr 16 22:36 easy_install-3.5
-rwxr-xr-x 1 jenkins_slave root    222 Apr 16 22:49 epylint
-rwxr-xr-x 1 jenkins_slave root    758 Apr 16 22:37 f2py
-rw-r--r-- 1 jenkins_slave root  18864 Apr 16 22:47 gflags2man.py
-rw-r--r-- 1 jenkins_slave root  18063 Apr 16 22:47 gflags2man.pyc
-rwxr-xr-x 1 jenkins_slave root    234 Apr 16 22:47 iptest
-rwxr-xr-x 1 jenkins_slave root    234 Apr 16 22:47 iptest2
-rwxr-xr-x 1 jenkins_slave root    227 Apr 16 22:47 ipython
-rwxr-xr-x 1 jenkins_slave root    227 Apr 16 22:47 ipython2
-rwxr-xr-x 1 jenkins_slave root    212 Apr 16 22:37 isort
-rwxr-xr-x 1 jenkins_slave root   1675 Apr 16 22:37 jp.py
-rw-r--r-- 1 jenkins_slave root   1846 Apr 16 22:37 jp.pyc
-rwxr-xr-x 1 jenkins_slave root  12538 Apr 16 22:38 lein
-rwxr-xr-x 1 jenkins_slave root    214 Apr 16 22:37 nosetests
-rwxr-xr-x 1 jenkins_slave root    214 Apr 16 22:36 nosetests-2.7
-rwxr-xr-x 1 jenkins_slave root    214 Apr 16 22:37 nosetests-3.4
-rwxr-xr-x 1 jenkins_slave root    214 Apr 16 22:49 pbr
-rwxr-xr-x 1 jenkins_slave root    215 Apr 16 22:36 pip
-rwxr-xr-x 1 jenkins_slave root    215 Apr 16 22:36 pip2
-rwxr-xr-x 1 jenkins_slave root    215 Apr 16 22:36 pip2.7
-rwxr-xr-x 1 jenkins_slave root    215 Apr 16 22:36 pip3
-rwxr-xr-x 1 jenkins_slave root    215 Apr 16 22:36 pip3.5
-rwxr-xr-x 1 jenkins_slave root    208 Apr 16 22:48 py.test
-rwxr-xr-x 1 jenkins_slave root    225 Apr 16 22:49 pybabel
-rwxr-xr-x 1 jenkins_slave root    218 Apr 16 22:48 pygmentize
-rwxr-xr-x 1 jenkins_slave root    220 Apr 16 22:49 pylint
-rwxr-xr-x 1 jenkins_slave root    226 Apr 16 22:49 pyreverse
-rwxr-xr-x 1 jenkins_slave root    208 Apr 16 22:48 pytest
-rwxr-xr-x 1 jenkins_slave root    594 Apr 16 22:37 rst2html.py
-rw-r--r-- 1 jenkins_slave root    604 Apr 16 22:37 rst2html.pyc
-rwxr-xr-x 1 jenkins_slave root    714 Apr 16 22:37 rst2html4.py
-rw-r--r-- 1 jenkins_slave root    726 Apr 16 22:37 rst2html4.pyc
-rwxr-xr-x 1 jenkins_slave root   1139 Apr 16 22:37 rst2html5.py
-rw-r--r-- 1 jenkins_slave root    730 Apr 16 22:37 rst2html5.pyc
-rwxr-xr-x 1 jenkins_slave root    791 Apr 16 22:37 rst2latex.py
-rw-r--r-- 1 jenkins_slave root    734 Apr 16 22:37 rst2latex.pyc
-rwxr-xr-x 1 jenkins_slave root    600 Apr 16 22:37 rst2man.py
-rw-r--r-- 1 jenkins_slave root    709 Apr 16 22:37 rst2man.pyc
-rwxr-xr-x 1 jenkins_slave root    764 Apr 16 22:37 rst2odt.py
-rw-r--r-- 1 jenkins_slave root    803 Apr 16 22:37 rst2odt.pyc
-rwxr-xr-x 1 jenkins_slave root   1698 Apr 16 22:37 rst2odt_prepstyles.py
-rw-r--r-- 1 jenkins_slave root   2044 Apr 16 22:37 rst2odt_prepstyles.pyc
-rwxr-xr-x 1 jenkins_slave root    601 Apr 16 22:37 rst2pseudoxml.py
-rw-r--r-- 1 jenkins_slave root    600 Apr 16 22:37 rst2pseudoxml.pyc
-rwxr-xr-x 1 jenkins_slave root    637 Apr 16 22:37 rst2s5.py
-rw-r--r-- 1 jenkins_slave root    649 Apr 16 22:37 rst2s5.pyc
-rwxr-xr-x 1 jenkins_slave root    871 Apr 16 22:37 rst2xetex.py
-rw-r--r-- 1 jenkins_slave root    817 Apr 16 22:37 rst2xetex.pyc
-rwxr-xr-x 1 jenkins_slave root    602 Apr 16 22:37 rst2xml.py
-rw-r--r-- 1 jenkins_slave root    612 Apr 16 22:37 rst2xml.pyc
-rwxr-xr-x 1 jenkins_slave root    670 Apr 16 22:37 rstpep2html.py
-rw-r--r-- 1 jenkins_slave root    680 Apr 16 22:37 rstpep2html.pyc
-rwxr-xr-x 1 jenkins_slave root    223 Apr 16 22:47 skivi
-rwxr-xr-x 1 jenkins_slave root    215 Apr 16 22:49 sphinx-apidoc
-rwxr-xr-x 1 jenkins_slave root    233 Apr 16 22:49 sphinx-autogen
-rwxr-xr-x 1 jenkins_slave root    208 Apr 16 22:49 sphinx-build
-rwxr-xr-x 1 jenkins_slave root    219 Apr 16 22:49 sphinx-quickstart
-rwxr-xr-x 1 jenkins_slave root    222 Apr 16 22:49 symilar
-rwxr-xr-x 1 jenkins_slave root    212 Apr 16 22:48 tabulate
-rwxr-xr-x 1 jenkins_slave root    211 Apr 16 22:36 wheel
+ ln -s ccache /usr/local/bin/gcc
+ ln -s ccache /usr/local/bin/gcc-8
+ ln -s ccache /usr/local/bin/g++
+ ln -s ccache /usr/local/bin/g++-8
+ ln -s ccache /usr/local/bin/nvcc
+ ln -s ccache /usr/local/bin/clang++-3.9
+ ln -s ccache /usr/local/bin/clang-3.9
+ ln -s ccache /usr/local/bin/clang++-5.0
+ ln -s ccache /usr/local/bin/clang-5.0
+ ln -s ccache /usr/local/bin/clang++-6.0
+ ln -s ccache /usr/local/bin/clang-6.0
+ ls -l /usr/local/bin
total 872
drwxr-xr-x 1 jenkins_slave root            4096 Apr 16 22:48 __pycache__
-rwxr-xr-x 1 jenkins_slave root             229 Apr 16 22:48 backend-test-tools
-rwxr-xr-x 1 jenkins_slave root             216 Apr 16 22:49 breathe-apidoc
-rwxr-xr-x 1 jenkins_slave root          497848 Apr 16 22:36 ccache
-rwxr-xr-x 1 jenkins_slave root             224 Apr 16 22:37 chardetect
-rwxr-xr-x 1 jenkins_slave root             232 Apr 16 22:48 check-model
-rwxr-xr-x 1 jenkins_slave root             230 Apr 16 22:48 check-node
lrwxrwxrwx 1 jenkins_slave jenkins_slave      6 Apr 19 01:20 clang++-3.9 -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave      6 Apr 19 01:20 clang++-5.0 -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave      6 Apr 19 01:20 clang++-6.0 -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave      6 Apr 19 01:20 clang-3.9 -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave      6 Apr 19 01:20 clang-5.0 -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave      6 Apr 19 01:20 clang-6.0 -> ccache
-rwxr-xr-x 1 jenkins_slave root             228 Apr 16 22:49 cm2html
-rwxr-xr-x 1 jenkins_slave root             230 Apr 16 22:49 cm2latex
-rwxr-xr-x 1 jenkins_slave root             226 Apr 16 22:49 cm2man
-rwxr-xr-x 1 jenkins_slave root             238 Apr 16 22:49 cm2pseudoxml
-rwxr-xr-x 1 jenkins_slave root             230 Apr 16 22:49 cm2xetex
-rwxr-xr-x 1 jenkins_slave root             226 Apr 16 22:49 cm2xml
lrwxrwxrwx 1 jenkins_slave root              20 Apr 16 22:35 cmake -> /opt/cmake/bin/cmake
-rwxr-xr-x 1 jenkins_slave root            1108 Apr 16 22:49 cmark.py
-rw-r--r-- 1 jenkins_slave root            1462 Apr 16 22:49 cmark.pyc
-rwxr-xr-x 1 jenkins_slave root             218 Apr 16 22:48 coverage
-rwxr-xr-x 1 jenkins_slave root             218 Apr 16 22:48 coverage-2.7
-rwxr-xr-x 1 jenkins_slave root             218 Apr 16 22:48 coverage-3.5
-rwxr-xr-x 1 jenkins_slave root             218 Apr 16 22:48 coverage2
-rwxr-xr-x 1 jenkins_slave root             218 Apr 16 22:48 coverage3
-rwxr-xr-x 1 jenkins_slave root             209 Apr 16 22:37 cpplint
-rwxr-xr-x 1 jenkins_slave root             223 Apr 16 22:46 cygdb
-rwxr-xr-x 1 jenkins_slave root             244 Apr 16 22:46 cython
-rwxr-xr-x 1 jenkins_slave root             224 Apr 16 22:46 cythonize
-rwxr-xr-x 1 jenkins_slave root             233 Apr 16 22:36 easy_install
-rwxr-xr-x 1 jenkins_slave root             233 Apr 16 22:36 easy_install-2.7
-rwxr-xr-x 1 jenkins_slave root             233 Apr 16 22:36 easy_install-3.5
-rwxr-xr-x 1 jenkins_slave root             222 Apr 16 22:49 epylint
-rwxr-xr-x 1 jenkins_slave root             758 Apr 16 22:37 f2py
lrwxrwxrwx 1 jenkins_slave jenkins_slave      6 Apr 19 01:20 g++ -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave      6 Apr 19 01:20 g++-8 -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave      6 Apr 19 01:20 gcc -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave      6 Apr 19 01:20 gcc-8 -> ccache
-rw-r--r-- 1 jenkins_slave root           18864 Apr 16 22:47 gflags2man.py
-rw-r--r-- 1 jenkins_slave root           18063 Apr 16 22:47 gflags2man.pyc
-rwxr-xr-x 1 jenkins_slave root             234 Apr 16 22:47 iptest
-rwxr-xr-x 1 jenkins_slave root             234 Apr 16 22:47 iptest2
-rwxr-xr-x 1 jenkins_slave root             227 Apr 16 22:47 ipython
-rwxr-xr-x 1 jenkins_slave root             227 Apr 16 22:47 ipython2
-rwxr-xr-x 1 jenkins_slave root             212 Apr 16 22:37 isort
-rwxr-xr-x 1 jenkins_slave root            1675 Apr 16 22:37 jp.py
-rw-r--r-- 1 jenkins_slave root            1846 Apr 16 22:37 jp.pyc
-rwxr-xr-x 1 jenkins_slave root           12538 Apr 16 22:38 lein
-rwxr-xr-x 1 jenkins_slave root             214 Apr 16 22:37 nosetests
-rwxr-xr-x 1 jenkins_slave root             214 Apr 16 22:36 nosetests-2.7
-rwxr-xr-x 1 jenkins_slave root             214 Apr 16 22:37 nosetests-3.4
lrwxrwxrwx 1 jenkins_slave jenkins_slave      6 Apr 19 01:20 nvcc -> ccache
-rwxr-xr-x 1 jenkins_slave root             214 Apr 16 22:49 pbr
-rwxr-xr-x 1 jenkins_slave root             215 Apr 16 22:36 pip
-rwxr-xr-x 1 jenkins_slave root             215 Apr 16 22:36 pip2
-rwxr-xr-x 1 jenkins_slave root             215 Apr 16 22:36 pip2.7
-rwxr-xr-x 1 jenkins_slave root             215 Apr 16 22:36 pip3
-rwxr-xr-x 1 jenkins_slave root             215 Apr 16 22:36 pip3.5
-rwxr-xr-x 1 jenkins_slave root             208 Apr 16 22:48 py.test
-rwxr-xr-x 1 jenkins_slave root             225 Apr 16 22:49 pybabel
-rwxr-xr-x 1 jenkins_slave root             218 Apr 16 22:48 pygmentize
-rwxr-xr-x 1 jenkins_slave root             220 Apr 16 22:49 pylint
-rwxr-xr-x 1 jenkins_slave root             226 Apr 16 22:49 pyreverse
-rwxr-xr-x 1 jenkins_slave root             208 Apr 16 22:48 pytest
-rwxr-xr-x 1 jenkins_slave root             594 Apr 16 22:37 rst2html.py
-rw-r--r-- 1 jenkins_slave root             604 Apr 16 22:37 rst2html.pyc
-rwxr-xr-x 1 jenkins_slave root             714 Apr 16 22:37 rst2html4.py
-rw-r--r-- 1 jenkins_slave root             726 Apr 16 22:37 rst2html4.pyc
-rwxr-xr-x 1 jenkins_slave root            1139 Apr 16 22:37 rst2html5.py
-rw-r--r-- 1 jenkins_slave root             730 Apr 16 22:37 rst2html5.pyc
-rwxr-xr-x 1 jenkins_slave root             791 Apr 16 22:37 rst2latex.py
-rw-r--r-- 1 jenkins_slave root             734 Apr 16 22:37 rst2latex.pyc
-rwxr-xr-x 1 jenkins_slave root             600 Apr 16 22:37 rst2man.py
-rw-r--r-- 1 jenkins_slave root             709 Apr 16 22:37 rst2man.pyc
-rwxr-xr-x 1 jenkins_slave root             764 Apr 16 22:37 rst2odt.py
-rw-r--r-- 1 jenkins_slave root             803 Apr 16 22:37 rst2odt.pyc
-rwxr-xr-x 1 jenkins_slave root            1698 Apr 16 22:37 rst2odt_prepstyles.py
-rw-r--r-- 1 jenkins_slave root            2044 Apr 16 22:37 rst2odt_prepstyles.pyc
-rwxr-xr-x 1 jenkins_slave root             601 Apr 16 22:37 rst2pseudoxml.py
-rw-r--r-- 1 jenkins_slave root             600 Apr 16 22:37 rst2pseudoxml.pyc
-rwxr-xr-x 1 jenkins_slave root             637 Apr 16 22:37 rst2s5.py
-rw-r--r-- 1 jenkins_slave root             649 Apr 16 22:37 rst2s5.pyc
-rwxr-xr-x 1 jenkins_slave root             871 Apr 16 22:37 rst2xetex.py
-rw-r--r-- 1 jenkins_slave root             817 Apr 16 22:37 rst2xetex.pyc
-rwxr-xr-x 1 jenkins_slave root             602 Apr 16 22:37 rst2xml.py
-rw-r--r-- 1 jenkins_slave root             612 Apr 16 22:37 rst2xml.pyc
-rwxr-xr-x 1 jenkins_slave root             670 Apr 16 22:37 rstpep2html.py
-rw-r--r-- 1 jenkins_slave root             680 Apr 16 22:37 rstpep2html.pyc
-rwxr-xr-x 1 jenkins_slave root             223 Apr 16 22:47 skivi
-rwxr-xr-x 1 jenkins_slave root             215 Apr 16 22:49 sphinx-apidoc
-rwxr-xr-x 1 jenkins_slave root             233 Apr 16 22:49 sphinx-autogen
-rwxr-xr-x 1 jenkins_slave root             208 Apr 16 22:49 sphinx-build
-rwxr-xr-x 1 jenkins_slave root             219 Apr 16 22:49 sphinx-quickstart
-rwxr-xr-x 1 jenkins_slave root             222 Apr 16 22:49 symilar
-rwxr-xr-x 1 jenkins_slave root             212 Apr 16 22:48 tabulate
-rwxr-xr-x 1 jenkins_slave root             211 Apr 16 22:36 wheel

Thus, I can not confirm the statement that the creation of the simlinks don't work.

I have made the experience that in some cases the PATH method with /usr/local/bin was not picked up. Thus, I added /usr/local/bin as suggested by CCache.

When I developed this integration, I did my due diligence to compare the ccache hit rates. I don't have the cycles to dive deep into this again and stand by the stance that my developed solution is working without major issues. Since you have not provided any instructions to reproduce your described problem, I can only execute the code our CI is running - which is running fine. This modification would reduce the CCache hitrate and thus my veto stands - as I have thoroughly explained to you over the course of the last months.

I have already given my review and still stand by it. Thus there are no further actions required from the reviewer side until the comments have been addressed.

@marcoabreu marcoabreu added pr-awaiting-response PR is reviewed and waiting for contributor to respond and removed pr-awaiting-review PR is waiting for code review labels Apr 19, 2019
@larroy
Copy link
Contributor Author

larroy commented Apr 19, 2019

We don't want to write in /usr/local/bin because then the script requires root privileges. The links in /tmp/ccache-redirects didn't work for you because the links were broken. Check the manpage of ln for details on usage.

Here is an example of what I'm trying to explain to you, what I fixed, and that links in /usr/local/bin are not necessary.

piotr@ip-172-31-27-211:0:~/mxnet_master ((ff04de0a7...))+$ docker \          
>         run \                                                            
>         --cap-add \                                                      
>         SYS_PTRACE \                                                     
>         --rm \                                                     
>         --shm-size=500m \                                            
>         -v \                                                       
>         /home/piotr/mxnet_master:/work/mxnet \                       
>         -v \                                                        
>         /home/piotr/mxnet_master/build:/work/build \                                   
>         -v \
>         /home/piotr/.ccache:/work/ccache \
>         -u \
>         1001:1002 \
>         -e \
>         CCACHE_MAXSIZE=500G \
>         -e \
>         CCACHE_TEMPDIR=/tmp/ccache \
>         -e \
>         CCACHE_DIR=/work/ccache \
>         -e \
>         CCACHE_LOGFILE=/tmp/ccache.log \
>         -ti \
>         mxnetci/build.ubuntu_cpu
jenkins_slave@9c9d63f274c2:/work/mxnet$ /work/runtime_functions.sh build_ccache_wrappers
+ NOSE_COVERAGE_ARGUMENTS='--with-coverage --cover-inclusive --cover-xml --cover-branches --cover-package=mxnet'
+ NOSE_TIMER_ARGUMENTS='--with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error'
+ CI_CUDA_COMPUTE_CAPABILITIES='-gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_70,code=sm_70'
+ CI_CMAKE_CUDA_ARCH_BIN=52,70
+ set +x
+ '[' -z ']'
+ echo 'No $CC set, defaulting to gcc'
No $CC set, defaulting to gcc
+ export CC=gcc
+ CC=gcc
+ '[' -z ']'
+ echo 'No $CXX set, defaulting to g++'
No $CXX set, defaulting to g++
+ export CXX=g++
+ CXX=g++
+ mkdir /tmp/ccache-redirects
+ export PATH=/tmp/ccache-redirects:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ ln -s ccache /tmp/ccache-redirects/gcc
+ ln -s ccache /tmp/ccache-redirects/gcc-8
+ ln -s ccache /tmp/ccache-redirects/g++
+ ln -s ccache /tmp/ccache-redirects/g++-8
+ ln -s ccache /tmp/ccache-redirects/nvcc
+ ln -s ccache /tmp/ccache-redirects/clang++-3.9
+ ln -s ccache /tmp/ccache-redirects/clang-3.9
+ ln -s ccache /tmp/ccache-redirects/clang++-5.0
+ ln -s ccache /tmp/ccache-redirects/clang-5.0
+ ln -s ccache /tmp/ccache-redirects/clang++-6.0
+ ln -s ccache /tmp/ccache-redirects/clang-6.0
+ ln -s ccache /usr/local/bin/gcc
+ ln -s ccache /usr/local/bin/gcc-8
+ ln -s ccache /usr/local/bin/g++
+ ln -s ccache /usr/local/bin/g++-8
+ ln -s ccache /usr/local/bin/nvcc
+ ln -s ccache /usr/local/bin/clang++-3.9
+ ln -s ccache /usr/local/bin/clang-3.9
+ ln -s ccache /usr/local/bin/clang++-5.0
+ ln -s ccache /usr/local/bin/clang-5.0
+ ln -s ccache /usr/local/bin/clang++-6.0
+ ln -s ccache /usr/local/bin/clang-6.0
+ export NVCC=ccache
+ NVCC=ccache
jenkins_slave@9c9d63f274c2:/work/mxnet$ ls -l /tmp/ccache-redirects/
total 0
lrwxrwxrwx 1 jenkins_slave jenkins_slave 6 Apr 19 01:40 clang++-3.9 -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave 6 Apr 19 01:40 clang++-5.0 -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave 6 Apr 19 01:40 clang++-6.0 -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave 6 Apr 19 01:40 clang-3.9 -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave 6 Apr 19 01:40 clang-5.0 -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave 6 Apr 19 01:40 clang-6.0 -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave 6 Apr 19 01:40 g++ -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave 6 Apr 19 01:40 g++-8 -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave 6 Apr 19 01:40 gcc -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave 6 Apr 19 01:40 gcc-8 -> ccache
lrwxrwxrwx 1 jenkins_slave jenkins_slave 6 Apr 19 01:40 nvcc -> ccache
jenkins_slave@9c9d63f274c2:/work/mxnet$ /work/runtime_functions.sh build_ccache_wrappers2^C
jenkins_slave@9c9d63f274c2:/work/mxnet$ test -e /tmp/ccache-redirects/g++
jenkins_slave@9c9d63f274c2:/work/mxnet$ test -e /tmp/ccache-redirects/g++ ^C
jenkins_slave@9c9d63f274c2:/work/mxnet$ man test
jenkins_slave@9c9d63f274c2:/work/mxnet$ test -e /tmp/ccache-redirects/g++
jenkins_slave@9c9d63f274c2:/work/mxnet$ test -e /tmp/ccache-redirects/g++ && echo "link works"
jenkins_slave@9c9d63f274c2:/work/mxnet$ rm /tmp/ccache-redirects/g++
jenkins_slave@9c9d63f274c2:/work/mxnet$ ln -s `which ccache` /tmp/ccache-redirects/g++
jenkins_slave@9c9d63f274c2:/work/mxnet$ test -e /tmp/ccache-redirects/g++ && echo "link broken"
link broken
jenkins_slave@9c9d63f274c2:/work/mxnet$ 

@larroy
Copy link
Contributor Author

larroy commented Apr 19, 2019

Btw I tested that this is calling ccache using statistics as expected. And I'm using the redirect method as in the ccache documentation, just not in /usr/local/bin path. Why are you insisting on having them in /usr/local/bin when you admit that the reason is that in /tmp/ccache-redirects it didn't work for you due to your broken links?

@larroy
Copy link
Contributor Author

larroy commented Apr 19, 2019

As you can see in your patch, ccache is disabled from one build because there's a problem. My PR fixes that problems, makes the function work without needing root privileges and more. Can you provide a reason why are you blocking this PR?

This PR doesn't affect the hit rate of ccache but increase it, based on what are you making these claims?

It's also not idempotent, the second time you call it it will fail, preventing all the build functions to be called multiple times which is essential if you are making changes or want to rebuild.

jenkins_slave@9c9d63f274c2:/work/mxnet$ /work/runtime_functions.sh build_ubuntu_cpu
+ NOSE_COVERAGE_ARGUMENTS='--with-coverage --cover-inclusive --cover-xml --cover-branches --cover-package=mxnet'
+ NOSE_TIMER_ARGUMENTS='--with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error'
+ CI_CUDA_COMPUTE_CAPABILITIES='-gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_70,code=sm_70'
+ CI_CMAKE_CUDA_ARCH_BIN=52,70
+ set +x
+ export CC=gcc
+ CC=gcc
+ export CXX=g++
+ CXX=g++
+ build_ccache_wrappers
+ set -ex
+ '[' -z x ']'
+ '[' -z x ']'
+ mkdir /tmp/ccache-redirects
mkdir: cannot create directory '/tmp/ccache-redirects': File exists

@larroy
Copy link
Contributor Author

larroy commented Apr 19, 2019

As written in the description, addresses and (fully) fixes the two linked issues.

@larroy
Copy link
Contributor Author

larroy commented Apr 19, 2019

My PR increases ccache hit rate because ccache is disabled for "build_ubuntu_gpu_cuda91_cudnn7" as you can see in the PR. This is slowing down development if you need to use this build, as it doesn't use ccache.

@larroy
Copy link
Contributor Author

larroy commented Apr 19, 2019

jenkins_slave@1a383a8c60d8:/work/mxnet$ ccache -s
cache directory /work/ccache
primary config /work/ccache/ccache.conf
secondary config (readonly) /usr/local/etc/ccache.conf
stats zero time Fri Apr 19 03:06:17 2019
cache hit (direct) 4
cache hit (preprocessed) 0
cache miss 0
cache hit rate 100.00 %
called for link 21
unsupported compiler option 21
no input file 16
cleanups performed 0
files in cache 5953
cache size 11.1 GB
max cache size 500.0 GB

@larroy
Copy link
Contributor Author

larroy commented Apr 22, 2019

@mxnet-label-bot add [pr-awaiting-review] remove [pr-awaiting-response]

@marcoabreu marcoabreu added the pr-awaiting-review PR is waiting for code review label Apr 22, 2019
@marcoabreu
Copy link
Contributor

Our Docker setup is running under the expectation to fully run as root.

Usually these scripts are not intended to be called multiple times within the same session, so idempotency was never an issue. But great that you fixed, I'm happy to merge that part.

For the broken symlink, it now makes sense - thanks for pointing that out.

Since the /usr/local/bin method is recommended by the ccache authors, I'd just keep it in to not divert from the standard usage. It doesn't hurt to stay in, so I don't see much reason to have a discussion about it.

Back when I worked on ccache, I had no issue with NVCC and it was officially supported (although the hit-rate wasn't that high). And as we can see, the setup we have works since otherwise our CI would break. There might be some use-cases in which it doesn't work, but until ccache/ccache#373 (comment) is confirmed and contains proper steps to reproduce the error, I would like to keep it as-is á la "never change a running system".

As a summary, please split your change into two atomic PRs that address a single concern:

  1. Make mkdir and symlinking idempotent
  2. Disable CCache-NVCC

I won't merge removing /usr/local/bin, so there's no need to make a PR for that.

I'm happy to merge PR 1 right away and would wait with merging PR 2 until detailed instructions to reproduce are available and have been confirmed. In my tests it worked and thus I'd like to have confirmation first. Right now, CI works and thus I'd keep it until it breaks.

@larroy
Copy link
Contributor Author

larroy commented Apr 24, 2019

Thanks for your response. I don't think NVCC through ccache worked, otherwise why did you leave the build that this PR enables disabled?

Having the ccache links in /usr/local/bin makes absolutely no difference versus having them in any other path is which prepended to the PATH variable. The result of insisting on keeping the paths in /usr/local/bin results in our build functions not working from a local environment to reproduce exact builds (since they require root) and because they are not idempotent.

I can split the PR as you request. Have you seen that I verified that the hit rate is 100% while using the redirects in the /tmp path? Isn't this enough validation that the links in /usr/local/bin are not needed?

@anirudhacharya
Copy link
Member

@larroy can you please update the PR as per your comments above

@larroy larroy force-pushed the ccache_wrappers branch 2 times, most recently from 74816e8 to a5c630a Compare May 14, 2019 21:40
@piyushghai
Copy link
Contributor

piyushghai commented Jun 7, 2019

@marcoabreu Can this PR be merged ?

@mxnet-label-bot Update [pr-awaiting-merge, CI]

@marcoabreu marcoabreu added pr-awaiting-merge Review and CI is complete. Ready to Merge and removed pr-awaiting-response PR is reviewed and waiting for contributor to respond pr-awaiting-review PR is waiting for code review labels Jun 7, 2019
* Fix broken links
* Make it idempotent
fixes apache#13456
fixes apache#14117
fixes apache#11516
@marcoabreu marcoabreu merged commit 7fe478a into apache:master Jun 24, 2019
@larroy larroy deleted the ccache_wrappers branch June 25, 2019 21:40
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
CI pr-awaiting-merge Review and CI is complete. Ready to Merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Would like to be able to execute build function several times, building ccache wrapers prevents this
5 participants